1. Software Testers: The Next Generation Bj Rollison, Test Architect Engineering Excellence Group Microsoft, Inc. (willro@microsoft.com)

2. Microsoft ® Copyright © Bj Rollison, 2005 Overview  Today’s testers  State of tester’s knowledge  Tester performance  The proof (Lies, damn lies and statistics)  Empirical evidence  Why the current way isn’t good enough anymore  Influences causing change  Tomorrow’s testers  Here's what’s happening at MS  Here's what you should/could do personally to help yourself

3. Microsoft ® Copyright © Bj Rollison, 2005 Tester training (or lack thereof)  “Less than 10% of testers have formal training in test techniques” – Dorothy Graham  “…few testers or developers have received any training in formal methods, especially test techniques.” – Marne Hutcheson (Software Testing Fundamentals)  The average software developer reads less than one professional book per year and subscribes to no professional magazines. - Steve McConnell, DeMarco and Lister, Peopleware, 2d Ed, 1999.

4. Microsoft ® Copyright © Bj Rollison, 2005 Tester’s Body of Knowledge  The average tester has read one book on software testing  Anecdotal evidence suggests the majority of testers have NOT read more than one book on testing  97% read Testing Computer Software (Kaner et. al)  4% read How to Break Software (Whittaker)  Less than 25% have a degree in computer science or other engineering field  More than 75% unable to effectively code using a modern programming language

5. Microsoft ® Copyright © Bj Rollison, 2005 The problem is... Universe of functionality & bugs Actual testing effort Under-testing Over-testing Average 30%

6. Microsoft ® Copyright © Bj Rollison, 2005 The venerable triangle case study “A program reads three (3) integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program displays a message that states whether the triangle is scalene, isosceles, or equilateral.” – G. Myers Suggested answers G. Meyers – 65 P. Jorgensen – 185 R. Binder - 65 K. Beck – 6 R. Collard – 4 All answers are right because there is no real context!

7. Microsoft ® Copyright © Bj Rollison, 2005 Establishing the baseline if (Side A) or (Side B) or (Side C) != int16 (signed 16 bit int) if input > maxint then error message for overflow condition else if input ! = integer value then error message for format exception if Side A or Side B or Side C <= 0 then error message sides must be > 0 if (Side A + Side B <= Side C) or (Side B + Side C <= Side A) or (Side A + Side C <= Side C) then input does not equate to valid triangle if (Side A = = Side B) and (Side B = = Side C) then triangle is equilateral else if (Side A = = Side B) or (Side A = = Side C) or (Side B = = Side C) then triangle is isosceles else triangle is scalene *Aside from the obvious conditions above additional tests include 1 for max val for all 3 sides, 1 for min val for all 3 sides, and 3 to test for BO of sum of 2 sides

8. Microsoft ® Copyright © Bj Rollison, 2005 Testing the testers  Generate a set of tests using error guessing and exploratory testing methods that will:  adequately* evaluate the  triangle algorithm functionality  implemented in C#  against the specification (Time limit 15 minutes)

9. Microsoft ® Copyright © Bj Rollison, 2005 Case study demographics  Total number participants = 400  0% ever had formal training in software testing  Avg. years testing experience = 2¼ years  53% – < 1 year experience (20% STE)  12% – 1 - 2 years experience (60% STE)  10% – 2 - 4 years experience (70% STE)  25% – > 4 years experience (100% STE)  Approx. 50% C/C++/C# coding skill

10. Microsoft ® Copyright © Bj Rollison, 2005 Case study synopsis  31% of tests covered baseline functionality  28% of the tests did not add significant value  Redundant coverage, results would not prove or disprove anything not covered by previous tests  14% of the tests missed the primary objective  These tests did not focus on the specific triangle functionality of the program  12% of the tests were incorrect assumptions  Wrong or incorrect results expected

11. Microsoft ® Copyright © Bj Rollison, 2005 Overall Results 31% of tests met the primary objective > 50% of test effort was redundant or not focused on primary objective

12. Microsoft ® Copyright © Bj Rollison, 2005 Results by Experience

13. Microsoft ® Copyright © Bj Rollison, 2005 STE Results by Experience

14. Microsoft ® Copyright © Bj Rollison, 2005 SDET Results by Experience

15. Microsoft ® Copyright © Bj Rollison, 2005 Technical skill comparison

16. Microsoft ® Copyright © Bj Rollison, 2005 Detailed Analysis  40% of the tests covered only 4 tests for the triangle (equilateral, scalene, isosceles, invalid)  STE 56% more likely to execute 4 or less tests  20% STEs (1-12 month) did not test for triangle  24% of STEs did not test for invalid triangle input  Only 54% tested for specific boundary conditions  Only 5% tested for overflow conditions  Less than 1% of STEs tested for overflow conditions

17. Microsoft ® Copyright © Bj Rollison, 2005 Skill Comparison  STE 4 times more likely to write an incorrect / invalid test  STE 1.5 times more likely to execute a redundant test  SDET 2 times more likely to exercise specific boundary conditions and overflow errors  SDET tests 2 times more likely to be an effective test

18. Microsoft ® Copyright © Bj Rollison, 2005 Case study summary  Black box testing is less than 35% effective  Testers who lack formal training in testing techniques are most likely to under test boundaries, exception handling routines, and critical functional areas  More than 50% of the testing effort by untrained testers results in redundant or ineffective* tests  Testers without computer/programming knowledge are less capable of improving test effectiveness efficiently

19. Microsoft ® Copyright © Bj Rollison, 2005 So, what’s changing?  Market pressures  Increasing demand for higher quality  Customer’s more aware of capabilities  Complex solutions  Increasing complexity (esp. integration)  Drive quality upstream  Unit testing, test driven development, etc  “Good Bugs harder to find  Cost cutting  Hot-fixes, service packs, shelf-life (10 years)  Less testers with greater skills & knowledge

20. Microsoft ® Copyright © Bj Rollison, 2005 Changes at Microsoft  Hiring standards  Computer science and other engineering background  Training  40 hours of hands on training for new test engineers  Study groups, focus groups, etc.  External conferences and seminars  Retention in testing discipline  Stop the “brain-drain”  Remove the “glass ceiling” for non-management roles  Provide greater challenges and scope of influence

21. Microsoft ® Copyright © Bj Rollison, 2005 What should I do?  Increase professional knowledge  Formal training, conferences, etc.  Books, magazines, industry white papers  Software Testing Techniques 2 nd Ed. – Boris Beizer  The Art of Software Testing. – Glenford Myers  Testing Object Oriented Systems – Robert Binder  A Practitioners Guide to Software Test Design – Lee Copeland  Increase technical knowledge  Modern programming language  Automation  Domain expertise  System expertise

22. Questions? Testing is our profession; Quality is our passion! ™