1. Advanced Software Engineering: Software Testing COMP 3702 Instructor: Anneliese Andrews

2. Advanced Software Engineering: Software Testing News & Project  News  Guest lecture 10/5  Project  Next deadline 10/5: Report to supervisor and peer review group  Book time

3. Advanced Software Engineering: Software Testing Lecture  Chapter 8  Organization  Chapter 14  Tools  Test automation

4. Advanced Software Engineering: Software Testing Organisation Support for decision making Enhance teamwork Independency Balance testing – quality Assist test management Ownership of test technology Resources utilization Career path

5. Advanced Software Engineering: Software Testing Test Organisation Degrees of freedom  Tall or flat  Market or product  Centralized or decentralized  Hierarchical or diffused  Line or staff  Functional or project

6. Advanced Software Engineering: Software Testing 7 approaches to test organisation 1.Each person’s responsibility 2.Each unit’s responsibility 3.Dedicated resource 4.Test organisation in QA 5.Test organisation in development 6.Centralized test organisation 7.Test technology centre Explanation Benefits Drawbacks ( Example ) ( Change )

7. Advanced Software Engineering: Software Testing 1. Each person’s responsibility

8. Advanced Software Engineering: Software Testing 2. Each unit’s responsibility Product developers M PPPPP P T P T

9. Advanced Software Engineering: Software Testing 3a. Dedicated resource

10. Advanced Software Engineering: Software Testing 3b. Dedicated resource in a large scale

11. Advanced Software Engineering: Software Testing 4. Test organisation in QA

12. Advanced Software Engineering: Software Testing 5. Test organisation in development

13. Advanced Software Engineering: Software Testing 6. Centralized test organisation

14. Advanced Software Engineering: Software Testing 7. Test technology centre

15. Advanced Software Engineering: Software Testing  What criteria to consider? Which organisation should we choose?  Size  Maturity  Focus

16. Advanced Software Engineering: Software Testing History of software testing Defect prevention & Test process 195019601970198019902000 Fix bugs Exhaustive testing Prove it works Prove it does not work Early test design Internet (Agile) Test automation tools Advanced test automation Automated business optimization

17. Advanced Software Engineering: Software Testing Tools  Good at repeating tasks  Good at organising data  Requires training  Introduced incrementally  No “silver bullet”  Ease of use  Power  Robustness  Functionality  Ease of insertion  Quality of support  Cost  Company policies and goals

18. Advanced Software Engineering: Software Testing Test tools Test management tools Test execution and comparison tools Dynamic analysis tools Debugging tools Coverage tools Static analysis tools Test design tools Architectural design Detailed design Code Requirement specification Unit test Integration test Performance simulator tools System test Acceptance test

19. Advanced Software Engineering: Software Testing Tools categories by activity  Reviews and inspections  Test planning  Test design and development  Test execution and evaluation  Testing support

20. Advanced Software Engineering: Software Testing Reviews and inspections  Complexity analysis  Identify problem areas  Code comprehension  Show different views of the artefact  Syntax and semantic analysis  Check and warn

21. Advanced Software Engineering: Software Testing Test planning  Templates for test plan documentation  Test schedule and staffing estimates  Complexity analyser To large extent general project management tools

22. Advanced Software Engineering: Software Testing Test design and development  Test data generator  Requirements-based test design tool  Capture/replay  Coverage analysis Often integrated with test execution tools

23. Advanced Software Engineering: Software Testing Test execution and evaluation  Capture/replay  Coverage analysis  Memory testing  Test case management  Simulators and performance

24. Advanced Software Engineering: Software Testing Capture/replay  Static c/r tools (without scripting language)  Static c/r tools (with scripting language)  Variable c/r tools

25. Advanced Software Engineering: Software Testing Coverage analysis  Monitoring execution  Preprocessing code Problem: intrusive, in particular for real-time systems

26. Advanced Software Engineering: Software Testing Memory testing  Leaks (allocate but not free memory)  Array bounds  Initialization of variables Easy to use, low cost

27. Advanced Software Engineering: Software Testing Simulators and performance  Hardware emulators  Environment simulators  Load generators

28. Advanced Software Engineering: Software Testing Test case management  Keep track of testware  Manage test execution  Seamless integration with other tools  Automated test reporting Not yet available :-(

29. Advanced Software Engineering: Software Testing Testing support tools  Trouble reporting  Report, Dispatch, Follow-up  Configuration management  Manage, control and coordinate changes

30. Advanced Software Engineering: Software Testing Test automation  Effectiveness – independent of automation  Efficiency – can be improved by automation Automation takes 2-10(-30) times the time for first run!

31. Advanced Software Engineering: Software Testing Different levels of automation Select/identify test cases to run Set up test environment  Create test environment  Load test data  Repeat for each test case  Set up test prerequisites  Execute  Compare results  Log results  Analyze test failures  Report defects  Clear up after test case Clear up test environment  Delete unwanted test data  Save important data Summarize results Select/identify test cases to run Set up test environment  Create test environment  Load test data Repeat for each test case  Set up test prerequisites  Execute  Compare results  Log results  Clear up after test case Clear up test environment  Delete unwanted test data  Save important data  Summarize results  Analyze test failures  Report defects

32. Advanced Software Engineering: Software Testing Test automation levels [Bob Binder, ISSRE04] 1Test by poking around1 test/h 2Manual testing10 tests/h 3Automated test scripting100 tests/h 4Model-based testing1000 tests/h 5Full test automation*10 000 test/h * Evaluation is the bottle-neck of automated testing (Oracle)

33. Advanced Software Engineering: Software Testing Test automation promises 1.Efficient regression test 2.Run tests more often 3.Perform difficult tests (e.g. load, outcome check) 4.Better use of resources 5.Consistency and repeatability 6.Reuse of tests 7.Earlier time to market 8.Increased confidence

34. Advanced Software Engineering: Software Testing Common problems 1.Unrealistic expectations 2.Poor testing practice ”Automatic chaos just gives faster chaos” 3.Expected effectiveness 4.False sense of security 5.Maintenance of automatic tests 6.Technical problems (e.g. Interoperability) 7.Organizational issues

35. Advanced Software Engineering: Software Testing What can be automated? IntellectualPerformedonce Repeated Clerical 1. Identify 2. Design 3. Build 4. Execute 5. Check

36. Advanced Software Engineering: Software Testing Cost distribution Manual testing Same tests automated More mature automation Edit Set up Execute Analyze Clear up

37. Advanced Software Engineering: Software Testing Limitations of automating testing  Does not replace manual testing  Manual tests find more defects than automated tests  Does not improve effectiveness  Greater reliance on quality of tests  Oracle problem  Test automation may limit the software development  Costs of maintaining automated tests

38. Advanced Software Engineering: Software Testing Limitations of automating testing

39. Advanced Software Engineering: Software Testing What to automate first?  Most important tests  A set of breadth tests (sample each system area overall)  Test for the most important functions  Tests that are easiest to automate  Tests that will give the quickest payback  Test that are run the most often

40. Advanced Software Engineering: Software Testing Tests...... to Automate  Tests that are run often  Regression tests  Important but trivial  Expensive to perform manually  Multi-user tests  Endurance/reliability tests  Difficult to perform manually  Timing critical  Complex tests  Difficult comparisons... not to automate  Tests that are run rarely  Tests that are not important  will not find severe faults  Hard to recognize defects  Usability tests  Do the colours look nice?

41. Advanced Software Engineering: Software Testing There is no silver bullet! The Reckless assumptions of automation “Automated tests execute a sequence of actions without human intervention. This approach helps eliminate human error, and provides faster results. Since most products require tests to be run many times, automated testing generally leads to significant labour cost savings over time. Typically a company will pass the break-even point for labour costs after just two or three runs of an automated test.”

42. Advanced Software Engineering: Software Testing Reckless assumptions of automation  Testing is a sequence of actions that we can automate  Manual testing is a process that adapts easily to change and can cope with complexity  Humans are able to detect and evaluate problem patterns  How do you automatically interpret test results?

43. Advanced Software Engineering: Software Testing Reckless assumptions of automation  Testing means repeating the same actions over and over  Regression testing, no new defects found  Stepping in someone else’s footprints minimizes the change of being blown up by a land mine

44. Advanced Software Engineering: Software Testing Reckless assumptions of automation  An automated test is faster, because it needs no human intervention  It can be surprisingly hard to create a test suite that runs without human intervention  E.g. interpreting results, fixing/maintaining tests, problems and bugs in the test tools, changes to software under test,...

45. Advanced Software Engineering: Software Testing Reckless assumptions of automation  Automation reduces human errors  Some errors are amplified – any error that goes unnoticed when the test scripts and result files are created will go systematically unnoticed over and over again

46. Advanced Software Engineering: Software Testing Reckless assumptions of automation  We can quantify the costs and benefits of manual vs. automated testing  Two different processes that reveal different failures – a fare comparison is difficult  Test automation is one part of a test strategy – not a substitute for manual testing

47. Advanced Software Engineering: Software Testing Reckless assumptions of automation  Automation will lead to significant labour cost savings  Developing automation Writing test scripts is software development Testers are seldom competent coders The quality of test code must be good  Operating automated tests  Maintaining automated test suites  Probably does not significantly reduce the need for manual testing

48. Advanced Software Engineering: Software Testing Reckless assumptions of automation  Automation will not harm the test process  It’s dangerous to automate something we don’t understand  The result will be a large mass of test code that nobody understands

49. Advanced Software Engineering: Software Testing This week  Project  Read literature  Write report  Meeting with supervisor (if needed)  Lab  White-box testing  Report of Lab 3 to Carina