Lessons from Survey System Testing at Microsoft: The Good, The Bad and The Ugly Presented by L. Allen Slade Microsoft People Research at the 17 th Annual Conference of the Society for Industrial and Organizational Psychology Contact: L. Allen Slade One Microsoft Way Redmond WA,

Lack of Systematic Testing: The Bad (1998) Actions First global web based survey Vendor churn Trust and hope (i.e., dont test) Impact: Failure Survey locked up Vendor replaced 6 weeks of downtime Data integrity issues? Lesson: Testing is necessary

Emerging Testing: The Ugly (1999) Actions Very limited testing was planned New research team put together a very quick test plan: Load testing Data integrity testing Full system testing Testing results 25% decrease in capacity vs. previous year. Adding encryption reduced capacity further Miscoding employee response. Impact Good user experience. Fixes delayed survey launch by 3 weeks. Lessons Testing must be planned. Always have slack time after the last test to fix your system.

Systematic Testing: The Good (2000) Actions Selected new vendor Evaluated internal testing capabilities. Agreed to full testing as part of contract. Partnered more effectively with internal IT. Did not test reporting ordering tool. Testing results Identified load issues. Tweaked a few data issues. Impact Fixes delayed survey launch 10 days. Virtually flawless experience for employees/managers. HR users: Report ordering tool had slow response time during last two days. Lesson Testing is a partnership. Always have slack time after the last test to fix your system. Test all components.

Systematic Testing v. 2.0: The Better (2001) Actions/context: Refined the test plan – including load testing report tool. Outsourced tasks. Nimda virus hit day after survey launch Testing results Outsourced testers not very effective No load issues identified. Test environment had only MS survey. Data integrity testing revealed bugs – in test data Impact System not penetrated by Nimda virus. Downtime as precaution. Survey locked up on a few users. DLL conflicts between different surveys The tilde problem. Vision impaired employees unable to access survey. Lessons Simplify where few bugs are identified Build test cases that approximate reality. E.g., 70% favorable responses Test survey in production IT environment Accessibility testing (or accommodation)

The Impact of Survey System Testing 1998 The Bad 1999 The Ugly 2000 The Good 2001 The Better Response57.7% (15,824/ 27,444) 66.4% (21,323/ 32,089) 80.8% (32,123/ 39,753) 86.4% (41,252/ 47,721) Downtime6 weeks10 hours2 hours 24 hours Report Ordering Few reports1,550 reports ordered offline through Excel 2,748 reports ordered online 2,226 reports ordered online 4,774 reports pushed thru manager portal

There Never is A Final Version A flawless survey would still need to be tested the next year: Technology changes Systems are complex