One of the most far-reaching benefits of fulltime (or frequent) Internet connectivity might be the ability to transmit near-realtime reports from users to application developers. Most users of Microsoft Office Outlook 2003 have seen this functionality at work in Office Watson (aka Windows Error Reporting), an error feedback mechanism that sends Microsoft information about program crashes. These reports have helped Microsoft pinpoint and prioritize fixes for the most serious errors: those that halt the application.
Office 2003 introduced the Customer Experience Improvement Program (CEIP), a richer reporting mechanism than Office Watson. You can opt in to the CEIP during Office installation, or you can use the Help, Customer Feedback Options command to opt in to or out of the CEIP (you can opt out at any time). In general, the CEIP reports back to Microsoft once a day, through a small binary file. The CEIP uses a randomly generated number to identify the reporting machine, and all information that the mechanism collects is totally anonymous.
The CEIP tells Microsoft about what's going right in the application, as well as revealing more about what's going wrong. For starters, according to Ben Canning, a lead program manager in the Microsoft Office team, the CEIP reports information about all errors, not just crashes. This behavior lets Microsoft zero in on problems that occur more frequently than expected (both in terms of how many customers receive the error and how often individual users see it). Depending on the nature and severity of the problem, Canning said, Microsoft might fix code to avoid the problem or to suppress the error message or might push down a new, more informative error message that will help users avoid the problem in the future. Microsoft also can use the CEIP to deliver new Help information to participants' systems, completing the feedback loop.
Canning added that this level of error reporting, coupled with data about how long each application session lasts and whether it ends normally or in a crash, gives Microsoft metrics (e.g., mean time-to-crash, average number of nonfatal errors) that help the company measure the stability of each release. Canning confirmed that the CEIP can also help answer one of my burning questions: Do Outlook users really live in Outlook all day? Canning reported that apparently the answer is yes. Among the largest segment of CEIP users, Outlook runs for more than 8 hours at a time (although a large spread in actual times exists).
The CEIP holds huge implications for future versions of Microsoft applications. Microsoft has the potential to gather information from millions of real-world users, and that information could reveal whether customers are actually using a particular feature and whether Exchange Server users outnumber POP and IMAP users, as well as help Microsoft identify performance and reliability concerns.
Because the reporting is by machine, not by user, Microsoft doesn't know exactly how many Office 2003 customers have opted into the CEIP, but Canning said the best estimate so far is 20 percent. He stated that Microsoft is talking up the CEIP with its largest corporate customers as they contemplate their Office 2003 rollouts, touting the programs long-term contribution to product reliability. Organizations might want to enable CEIP participation by users so that they can "cast their votes" for which errors Microsoft fixes first and which features it continues to enhance. Office 2003 includes a Group Policy setting that you can use to opt systems into the CEIP; look for the setting in the Microsoft Office 2003 administrative template under Help\Help|Privacy Settings\Enable Customer Experience Improvement Program. For more information about the CEIP, see the URL below.
Continuing Software Improvement at Microsoft http://www.microsoft.com/products/ceip/english/default.htm
Register
