1. Testing & Software Quality Seminar on software quality 13.5.2005 Karipekka Kaunisto

2. Contents  Role of testing in quality assurance  Challenges of software testing  What is test automation?  Test automation: Possible benefits  Common pitfalls of test automation  Conclusions  References

3. Role of testing in quality assurance  Quality controlling  Final product meets it’s requirements  Find potential errors and flaws  Enforce standards and good design principles  Regression testing  Improving quality  Preventive testing  Find cause of an error not just symtoms

4. Role of testing...  Testing as supportive action  Data collected during testing can be used to develop various quality metrics  These can be used to some extend when evaluating system quality and maturity  However, numbers alone don’t solely assure good quality!

5. Examples of poor testing  A major U.S. retailer was hit with a large govermental fine in October of 2003 due to web site errors that enabled customers to view one anothers' online orders  In early 1999 a major computer game company recalled all copies of a popular new product due to software problems. The company made a public apology for releasing a product before it was ready  A retail store chain filed suit in August of 1997 against a transaction processing system vendor (not a credit card company) due to the software's inability to handle credit cards with year 2000 expiration dates

6. Challenges of software testing  Complexity of testing  Even in a seemingly simple program there can be potentially infinite number of possible input and output permutations to test  What about large software systems with complex control logic and numerous dependencies on other modules and entire systems?

7. Complexity of testing  => It is not feasible to get even close to testing all combinations and thus finding all possible errors!  Tester needs to carefully create test set in a way that minimises risk of fatal errors in final product  Related problem: How do you know when to stop testing?  Acceptable risk level

8. Managing large test sets  Various general techniques have been introduced for managing test sets  Partitioning to smaller subsets  Testing special cases (boundaries, special values etc.)  Testing most important functions only (focus testing)  Invalid inputs and data  Program flow and code coverage testing

9. Are we ready to ship?  Even with all the techniques available it will require tester’s personal expertise and domain knowledge to create test plan and make the final decision to approve the product  Business issues may also affect on this: Risk of errors vs. risk of delay  Plan and test effort correlate quite well to quality controlling role of testing

10. Other challenges  Testing activities require significant amount of time and resources of the project => Delays, hasty testing  Testing is often regarded as dull, monotonous and laborous part of software development => Poor effort  System architecture is often quite complex, which require special testing effort => Reliability suffers, all tests not even possible manually

11. What is test Automation?  ”The management and performance of test activities to include the development and execution of test scripts so as to verify test requirements, using an automated test tool”. – Dustin, Rashka & Paul  ”Testing supported by software tool”. – Faught, Bach

12. Automation in practice  Tester describes the test cases for tool by using special scripting language designed by tool developers  Some tools may also include graphical interface and recording options but in practise scripting has to be used  Script should also specify how tool is supposed to interpret the correct results of any given test case  Tool then takes care of executing specified tests and examining the results

13. Automation in practise (cont.)  Result validation includes text outputs, elapsed time, screen captures etc.  Can be very challenging part to do automatically and may require some human intervention in some cases!  Evaluation results are presented in clear test reports that can be used to examine results of test round  Produced reports can also be used to gather data for various quality metrics

14. Areas of test automation  Automation suits mainly on testing that requires repeated effort of similar tests cases  Regression testing  Portability testing  Performance and stress testing  Configuration testing  Smoke testing ...

15. Possible benefits  More reliable system  Improved requirements definition  Improved performance (load & stress) testing  Better co-operation with developers  Quality metrics & Test optimisation  Enchanced system development life cycle

16. Benefits (2)  More effective testing process  Improved effort in various sub-areas like regression, smoke, configuration and multi-platform compatibility testing  Ability to reproduce errors  Dull routine tests can be executed without human intervention  ”After-hours testing”

17. More effective...  Execution of tests that are not possible manually  Better focus on more advanced testing issues  Enchanced business expertise

18. Benefits (3)  Reduced test effort and schedule  Initial costs of automation are usually very high  Payback comes later on (possibly quite much later) when team has adopted the process and use of tools

19. Pitfalls of test automation  Automatic test planning and design  There are no tools that can generate these automatically!  Requires human expertise and domain knowledge  Tool just does what it is scripted to do and nothing else

20. Pitfalls (2)  Immeadiate cost and time savings  On the contrary introduction of automation and new tools will increase the need of resources!  Automation process must be planned, test architecture created, tools evaluated, people trained, scripts programmed...  = Lot’s of work

21. Immediate...  Potential savings will be archieved (much) later on when organisation has ’learned’ the process and created needed infrastructure for it  If automation is introduced poorly, savings will never be gained at all!  In the worst case automation can just mess things up

22. Pitfalls (3)  One tool does it all  Wide array of operating systems, hardware and programming languages  Very different systems and architectures are often used  Testing requirements differ depending on system and project  Result analysis differ (graphical, text, time etc.)

23. Pitfalls (4)  Automation requires no technical skill  Tools rely solely on scripts when executing tests  Maintainable and reusable script building requires good programming skills and knowledge of the tool  Testers may have to be able to use several different tools with different scripting technologies!

24. Pitfalls (5)  100% test automation  Even if automation succeeds it cannot completely replace manual testing  Some tests must be conducted manually and others require at least some human intervention  Automation is really useful only with test cases that are executed repeatedly over time (regression)

25. Other related tools  Code analyzers  Coverage analyzers  Memory analyzers (purifiers)  Web test tools  Test management tools

26. Conclusions  Testing has significant role in software quality assurance  Automation, when implemented properly can further improve test effort and thus lead to improved quality  However many automation attempts have failed because of unrealistic expectations and inproper introduction of automation tools

27. References  Dustin E., Rashka J., Paul J.: Automated Software Testing: Introduction, Management and Performance. Adison Wesley, 1999  Craig R. and Jaskiel S.: Systematic Software Testing, Artech House Publishing, 2002  Pettichord Bret, Presentations and Publications. http://www.io.com/~wazmo/papers/