1. Introduction to Software Testing Testing Strategy Dr. Pedro Mejia Alvarez From Book: Introduction to Software Testing: P.Amman & J.Offutt

2. Software Testing 2 Testing is the process of exercising a program with the specific intent of finding errors prior to delivery to the end user.

3. 3 Software Testing Testing can only show the presence of failures but NOT their abscence.

4. 4 What Testing Shows errors requirements conformance performance an indication of quality

5. Complexity of Testing Software n No other engineering field builds products as complicated as software n The term correctness has no meaning –Is a building correct? –Is a car correct? –Is a subway system correct? n Like other engineers, we must use abstraction to manage complexity –Writing a model or a program is an abstraction Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 5

6. Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 6 Testing & Debugging n Testing : Evaluating software by observing its execution n Test Failure : Execution of a test that results in a software failure n Debugging : The process of finding a fault given a failure Not all inputs will “trigger” a fault into causing a failure

7. Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 7 Fault & Failure Model Three conditions necessary for a failure to be observed 1. Reachability : The location or locations in the program that contain the fault must be reached 2. Infection : The state of the program must be incorrect 3. Propagation : The infected state must cause some output or final state of the program to be incorrect

8. Software Testing Activities n Test Engineer : An IT professional who is in charge of one or more technical test activities –Designing test inputs –Producing test values –Running test scripts –Analyzing results –Reporting results to developers and managers n Test Manager : In charge of one or more test engineers –Sets test policies and processes –Interacts with other managers on the project –Otherwise supports the engineers Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 8

9. 9 Strategic Approach To perform effective testing, you should conduct effective technical reviews. By doing this, many errors will be eliminated before testing commences. Testing begins at the component level and works "outward" toward the integration of the entire computer-based system. Different testing techniques are appropriate for different software engineering approaches and at different points in time. Testing is conducted by the developer of the software and (for large projects) an independent test group. Testing and debugging are different activities, but debugging must be accommodated in any testing strategy.

10. 10 V & V Verification refers to the set of tasks that ensure that software correctly implements a specific function. Validation refers to a different set of tasks that ensure that the software that has been built is traceable to customer requirements. Boehm [Boe81] states this another way: –Verification: "Are we building the product right?" –Validation: "Are we building the right product?"

11. 11 Who Tests the Software? developer independent tester Understands the system but, will test "gently" and, is driven by "delivery" Must learn about the system, but, will attempt to break it and, is driven by quality

12. 12 Testing Strategy System engineering Analysis modeling Design modeling Code generationUnit test Integration test Validation test System test

13. 13 Testing Strategy n We begin by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’ n For conventional software –The module (component) is our initial focus –Integration of modules follows n For OO software –our focus when “testing in the small” changes from an individual module (the conventional view) to an OO class that encompasses attributes and operations and implies communication and collaboration

14. 14 Strategic Issues n Specify product requirements in a quantifiable manner long before testing commences. n State testing objectives explicitly. n Understand the users of the software and develop a profile for each user category. n Develop a testing plan that emphasizes “rapid cycle testing.” n Build “robust” software that is designed to test itself n Use effective technical reviews as a filter prior to testing n Conduct technical reviews to assess the test strategy and test cases themselves. n Develop a continuous improvement approach for the testing process.

15. 15 Unit Testing module to be tested test cases results softwareengineer

16. 16 Unit Testing interface local data structures boundary conditions independent paths error handling paths module to be tested test cases

17. 17 Unit Test Environment Module stub stub driver RESULTS interface local data structures boundary conditions independent paths error handling paths test cases

18. 18 Integration Testing Strategies Options: the “big bang” approach an incremental construction strategy

19. 19 Top Down Integration top module is tested with stubs stubs are replaced one at a time, "depth first" as new modules are integrated, some subset of tests is re-run A B C DE FG

20. 20 Bottom-Up Integration drivers are replaced one at a time, "depth first" worker modules are grouped into builds and integrated A B C DE FG cluster

21. 21 Regression Testing Regression testing is the re-execution of some subset of tests that have already been conducted to ensure that changes have not propagated unintended side effects Whenever software is corrected, some aspect of the software configuration (the program, its documentation, or the data that support it) is changed. Regression testing helps to ensure that changes (due to testing or for other reasons) do not introduce unintended behavior or additional errors. Regression testing may be conducted manually, by re- executing a subset of all test cases or using automated capture/playback tools.

22. Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 22 Traditional Testing Levels Class A method mA1() method mA2() Class B method mB1() method mB2() main Class P n Acceptance testing : Is the software acceptable to the user? n Integration testing : Test how modules interact with each other n System testing : Test the overall functionality of the system n Module testing (developer testing) : Test each class, file, module, component n Unit testing (developer testing) : Test each unit (method) individually This view obscures underlying similarities

23. 23 OO Testing Strategy n class testing is the equivalent of unit testing –operations within the class are tested –the state behavior of the class is examined n integration applied three different strategies –thread-based testing—integrates the set of classes required to respond to one input or event –use-based testing—integrates the set of classes required to respond to one use case –cluster testing—integrates the set of classes required to demonstrate one collaboration

24. Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 24 Object-Oriented Testing Levels Class A method mA1() method mA2() Class B method mB1() method mB2() n Intra-class testing : Test an entire class as sequences of calls n Inter-class testing : Test multiple classes together n Inter-method testing : Test pairs of methods in the same class n Intra-method testing : Test each method individually

25. 25 WebApp Testing - I The content model for the WebApp is reviewed to uncover errors. The interface model is reviewed to ensure that all use cases can be accommodated. The design model for the WebApp is reviewed to uncover navigation errors. The user interface is tested to uncover errors in presentation and/or navigation mechanics. Each functional component is unit tested.

26. 26 WebApp Testing - II Navigation throughout the architecture is tested. The WebApp is implemented in a variety of different environmental configurations and is tested for compatibility with each configuration. Security tests are conducted in an attempt to exploit vulnerabilities in the WebApp or within its environment. Performance tests are conducted. The WebApp is tested by a controlled and monitored population of end- users. The results of their interaction with the system are evaluated for content and navigation errors, usability concerns, compatibility concerns, and WebApp reliability and performance.

27. 27 High Order Testing n Validation testing –Focus is on software requirements n System testing –Focus is on system integration n Alpha/Beta testing –Focus is on customer usage n Recovery testing –forces the software to fail in a variety of ways and verifies that recovery is properly performed n Security testing –verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration n Stress testing – executes a system in a manner that demands resources in abnormal quantity, frequency, or volume n Performance Testing –test the run-time performance of software within the context of an integrated system

28. Coverage Criteria n Even small programs have too many inputs to fully test them all –private static double computeAverage (int A, int B, int C) –On a 32-bit machine, each variable has over 4 billion possible values –Over 80 octillion possible tests!! –Input space might as well be infinite n Testers search a huge input space –Trying to find the fewest inputs that will find the most problems n Coverage criteria give structured, practical ways to search the input space –Search the input space thoroughly –Not much overlap in the tests Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 28

29. Advantages of Coverage Criteria n Maximize the “bang for the buck” n Provide traceability from software artifacts to tests –Source, requirements, design models, … n Make regression testing easier n Gives testers a “stopping rule” … when testing is finished n Can be well supported with powerful tools Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 29

30. Test Requirements and Criteria n Test Criterion : A collection of rules and a process that define test requirements ̶ Cover every statement ̶ Cover every functional requirement n Test Requirements : Specific things that must be satisfied or covered during testing –Each statement is a test requirement –Each functional requirement is a test requirement Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 30 Testing researchers have defined dozens of criteria, but they are all really just a few criteria on four types of structures … 1.Graphs 2.Logic expressions 3.Input domains 4.Syntax descriptions

31. Model-Driven Test Design n Test Design is the process of designing input values that will effectively test software n Test design is one of several activities for testing software –Most mathematical –Most technically challenging Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 31

32. Types of Test Activities n Testing can be broken up into four general types of activities 1.Test Design 2.Test Automation 3.Test Execution 4.Test Evaluation n Each type of activity requires different skills, background knowledge, education and training n No reasonable software development organization uses the same people for requirements, design, implementation, integration and configuration control Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 32 Why do test organizations still use the same people for all four test activities?? This clearly wastes resources 1.a) Criteria-based 1.b) Human-based

33. 1. Test Design— (a) Criteria-Based n This is the most technical job in software testing n Requires knowledge of : –Discrete math –Programming –Testing n Requires much of a traditional CS degree n This is intellectually stimulating, rewarding, and challenging n Test design is analogous to software architecture on the development side n Using people who are not qualified to design tests is a sure way to get ineffective tests Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 33 Design test values to satisfy coverage criteria or other engineering goal

34. 1. Test Design— (b) Human-Based n This is much harder than it may seem to developers n Criteria-based approaches can be blind to special situations n Requires knowledge of : –Domain, testing, and user interfaces n Requires almost no traditional CS –A background in the domain of the software is essential –An empirical background is very helpful (biology, psychology, …) –A logic background is very helpful (law, philosophy, math, …) n This is intellectually stimulating, rewarding, and challenging –But not to typical CS majors – they want to solve problems and build things Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 34 Design test values based on domain knowledge of the program and human knowledge of testing

35. 2. Test Automation n This is slightly less technical n Requires knowledge of programming n Requires very little theory n Often requires solutions to difficult problems related to observability and controllability n Can be boring for test designers n Programming is out of reach for many domain experts n Who is responsible for determining and embedding the expected outputs ? –Test designers may not always know the expected outputs –Test evaluators need to get involved early to help with this Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 35 Embed test values into executable scripts

36. 3. Test Execution n This is easy – and trivial if the tests are well automated n Requires basic computer skills –Interns –Employees with no technical background n Asking qualified test designers to execute tests is a sure way to convince them to look for a development job n If, for example, GUI tests are not well automated, this requires a lot of manual labor n Test executors have to be very careful and meticulous with bookkeeping Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 36 Run tests on the software and record the results

37. 4. Test Evaluation n This is much harder than it may seem n Requires knowledge of : –Domain –Testing –User interfaces and psychology n Usually requires almost no traditional CS –A background in the domain of the software is essential –An empirical background is very helpful (biology, psychology, …) –A logic background is very helpful (law, philosophy, math, …) n This is intellectually stimulating, rewarding, and challenging –But not to typical CS majors – they want to solve problems and build things Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 37 Evaluate results of testing, report to developers

38. Other Activities n Test management : Sets policy, organizes team, interfaces with development, chooses criteria, decides how much automation is needed, … n Test maintenance : Save tests for reuse as software evolves –Requires cooperation of test designers and automators –Deciding when to trim the test suite is partly policy and partly technical – and in general, very hard ! –Tests should be put in configuration control n Test documentation : All parties participate –Each test must document “why” – criterion and test requirement satisfied or a rationale for human-designed tests –Ensure traceability throughout the process –Keep documentation in the automated tests Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 38

39. Organizing the Team n A mature test organization needs only one test designer to work with several test automators, executors and evaluators n Improved automation will reduce the number of test executors –Theoretically to zero … but not in practice n Putting the wrong people on the wrong tasks leads to inefficiency, low job satisfaction and low job performance –A qualified test designer will be bored with other tasks and look for a job in development –A qualified test evaluator will not understand the benefits of test criteria n Test evaluators have the domain knowledge, so they must be free to add tests that “blind” engineering processes will not think of n The four test activities are quite different Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 39 Many test teams use the same people for ALL FOUR activities !!

40. Applying Test Activities Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 40 To use our people effectively and to test efficiently we need a process that lets test designers raise their level of abstraction

41. Using MDTD in Practice n This approach lets one test designer do the math n Then traditional testers and programmers can do their parts –Find values –Automate the tests –Run the tests –Evaluate the tests n Just like in traditional engineering … an engineer constructs models with calculus, then gives direction to carpenters, electricians, technicians, … Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 41 Test designers become the technical experts

42. Model-Driven Test Design Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 42 software artifact model / structure test requirements refined requirements / test specs input values test cases test scripts test results pass / fail IMPLEMENTATION ABSTRACTION LEVEL DESIGN ABSTRACTION LEVEL test requirements

43. Model-Driven Test Design – Steps Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 43 software artifact model / structure test requirements refined requirements / test specs input values test cases test scripts test results pass / fail IMPLEMENTATION ABSTRACTION LEVEL DESIGN ABSTRACTION LEVEL analysis criterionrefine generate prefix postfix expected automate execute evaluate test requirements domain analysis feedback

44. Model-Driven Test Design –Activities Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 44 software artifact model / structure test requirements refined requirements / test specs input values test cases test scripts test results pass / fail IMPLEMENTATION ABSTRACTION LEVEL DESIGN ABSTRACTION LEVEL Test Design Test Execution Test Evaluation Raising our abstraction level makes test design MUCH easier

45. Small Illustrative Example Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 45 Software Artifact : Java Method /** * Return index of node n at the * first position it appears, * -1 if it is not present */ public int indexOf (Node n) { for (int i=0; i < path.size(); i++) if (path.get(i).equals(n)) return i; return -1; } 45 3 2 1 i = 0 i < path.size() if return i return -1 Control Flow Graph

46. Example (2) Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 46 Support tool for graph coverage http://www.cs.gmu.edu/~offutt/softwaretest/ 45 3 2 1 Graph Abstract version Edges 1 2 2 3 3 2 3 4 2 5 Initial Node: 1 Final Nodes: 4, 5 6 requirements for Edge-Pair Coverage 1. [1, 2, 3] 2. [1, 2, 5] 3. [2, 3, 4] 4. [2, 3, 2] 5. [3, 2, 3] 6. [3, 2, 5] Test Paths [1, 2, 5] [1, 2, 3, 2, 5] [1, 2, 3, 2, 3, 4] Find values …

47. 47 Debugging: A Diagnostic Process

48. 48 The Debugging Process

49. 49 Debugging Effort time required to diagnose the symptom and determine the cause time required to correct the error and conduct regression tests

50. 50 Symptoms & Causes symptom cause symptom and cause may be geographically separated symptom may disappear when another problem is fixed cause may be due to a combination of non-errors cause may be due to a system or compiler error cause may be due to assumptions that everyone believes symptom may be intermittent

51. 51 Consequences of Bugs damage mild annoying disturbing serious extreme catastrophic infectious Bug Type Bug Categories: function-related bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards violations, etc.

52. 52 Debugging Techniques brute force / testing backtracking induction deduction

53. 53 Correcting the Error Is the cause of the bug reproduced in another part of the program? In many situations, a program defect is caused by an erroneous pattern of logic that may be reproduced elsewhere. What "next bug" might be introduced by the fix I'm about to make? Before the correction is made, the source code (or, better, the design) should be evaluated to assess coupling of logic and data structures. What could we have done to prevent this bug in the first place? This question is the first step toward establishing a statistical software quality assurance approach. If you correct the process as well as the product, the bug will be removed from the current program and may be eliminated from all future programs. What if the bug persist and I cannot correct it ? Can I do Fault Tolerance ?

54. 54 Final Thoughts n Think -- before you act to correct n Use tools to gain additional insight n If you’re at an impasse, get help from someone else n Once you correct the bug, use regression testing to uncover any side effects

55. 55 Object-Oriented Testing n begins by evaluating the correctness and consistency of the analysis and design models n testing strategy changes –the concept of the ‘unit’ broadens due to encapsulation –integration focuses on classes and their execution across a ‘thread’ or in the context of a usage scenario –validation uses conventional black box methods n test case design draws on conventional methods, but also encompasses special features