May 29 th, 2003 Curtis Anderson Sivaprasad Padisetty

Microsoft Confidential Overview of Presentation Part I – Software Development and Automation The Project Cycle Benefits of Automation Effective Automation Part II – Test Distribution and Development Test Distribution Systems Test Development Summary

Software Development and Automation Curtis Anderson

Microsoft Confidential The Project Cycle Test Investment in Automation The investment in test automation starts early Focus on adding testability to spec/design phase Investment usually peaks as Development slows feature coding Planning (M0) Major Milestone Phase M1,2… Release Phase Time Testing Investment Testing Phase

Microsoft Confidential The Project Cycle Automation Payback Planning (M0) Major Milestone Phase M1,2… Release Phase Time Automation Payback Testing Phase Return on Investment is near zero early Largest gains at project end game and sustaining engineering

Microsoft Confidential Benefits of Automation What does Automation Bring to the Project? Consistent Test Execution Stability Guarantees Reliability Metrics Reproduce-able Quality Assessment

Microsoft Confidential Advantages of Automation Why Automate ? Machines instead of People Run existing tests automatically on a new build or version Exact same test happens every time Do More Testing Simplistic, repetitive (BORING) tests are run for free. People spend time looking only at results People are free to ad hoc and search for bugs

Microsoft Confidential Advantages (continued) Automation can perform costly, hard, or impossible to execute tests Perf, Stress, Scalability, Config, OM/API, Matrix Consistency and Reproducibility Performance, Regression Reusability How many File Open drivers are there? Logging, etc. Ensure a level of stability before a human spends time Keeps the ‘patient healthy’ Easy to hand off (external teams & sustaining engineering)

Microsoft Confidential Risks and Problems Over-reliance on automation results Its easy to generate lots of results that creates too much noise Maintenance Exact same test happens every time False failures and false positives Skill set to produce Re-inventing the wheel Time Management is Key!

Microsoft Confidential Test Categorization Unit Test Interoperability Localizability Interoperability Performance Reliability Scalability Security Globalization Checkin Test Build Verification Acceptance Test Pass/Comprehensive Regression DogFood/Deployment System Integration Code Coverage What does it do vs. When does it run Categorizing tests helps target testing and keep things focused.

Microsoft Confidential Make Your Automation Effective Time Management is key Cost of authoring Re-use everything Write solid automation Document well Base all automation on test cases Be willing to punt if it’s not working Run the right automation at the right time Roll up the right metrics to your project

Microsoft Confidential Test Distribution System Overview Why do we need a system? Standard way to execute tests Automate the execution Standard reporting and history What does it do? Repository for managing automation Test distribution Report Results Track & manage failures

Test Distribution and Development Sivaprasad Padisetty

Microsoft Confidential Test Distribution System Components User Interface Middle Tier Database Machine Management/Controller Client Agent Test Harness Machine Imaging and other common tasks Results Viewer

Microsoft Confidential Test Distribution System Features/Requirements of a good system User Friendly, Modular, Flexible Integrated Automated/Manual test support Distributes execution – lab management Multi stage, multi machine Offline execution and Private network support Test Version Control Object Model Queuing and Resource management Hardware and Software tracking Configuration management & Matrix support Deployment/Image management Results reporting

Microsoft Confidential Test Development Abstract from code as much as possible – the less code that changes to add a test case the better Separate infrastructure code from test code Re-use as much code as possible Modeling and Data Driven tests help scale

Microsoft Confidential Test Development Good Authoring is: Focused Consistent Accurate & Reproducible Portable, Easily Leveraged Impervious to Feature Changes Feature churn Language, even application, user, etc. Cost / Benefit is the ONLY measurement (Don’t forget sustain engineering)

Microsoft Confidential Test Development Writing Automation is Software Development Log-Log-Log-Log-Log everything Plan & Design Develop coding standards Write maintainable code Document you code and your test cases Peer-reviews before check in Use a version control system (Source Depot or VSS) Test your test. Log-Log-Log-Log-Log everything This code will last as long as the product code!

Microsoft Confidential Test Development Develop Configuration Agnostic tests No hard coded anything! links like files in the definition No dependency on a specific machine name No dependency on a specific lab Should be package-able. Configuration changes & dependencies should be clearly defined.

Microsoft Confidential Reporting and Metrics Good logging saves time Individual Log files from Test Execution Roll up summary reports for all tests Available through system UI and Project roll up Metrics help with project status

Microsoft Confidential Summary Test Automation is a competitive advantage for MS Lack of Automation hurts innovation in the product Test Development is Software Development Many resources and experts exist across the company

Microsoft Confidential Test Automation Resources Automated Testing Handbook – Linda G. Hayes Software Test Automation, Efffective Use of Test Execution Tools, Mark Fewster and Dorothy Graham Internal Windows (WTT & WCTI) and Test Categoization doc link: Join Partner DL wttpart, to participate in Shared Development community. Office and Automation Strategy Doc Office automation contacts – mailto:offautomailto:offauto DevDiv and MadDog forums - MadDog General Docs -

Questions?