1. Topic 12Summer 2003 1 ICS 52: Introduction to Software Engineering Lecture Notes for Summer Quarter, 2003 Michele Rousseau Topic 12 Partially based on lecture notes written by Sommerville, Frost, Van Der Hoek, Taylor & Tonne. Duplication of course material for any commercial purpose without the written permission of the lecturers is prohibited

2. Topic 12Summer 2003 2 Verification and Validation Informal Requirements Formal Specification Software Implementation Validation Verification Verification: is implementation consistent with requirements specification? Validation: does the system meet the customer’s/user’s needs?

3. Topic 12Summer 2003 3 Software Quality: assessment by V&V l Software process must include verification and validation to measure product qualities correctness, reliability, robustness efficiency, usability, understandability verifiability, maintainability reusability, portability, interoperability, real-time, safety, security, survivability, accuracy l Products can be improved by improving the process by which they are developed and assessed

4. Topic 12Summer 2003 4 Testing Terminology l Failure: Incorrect or unexpected output, based on specifications Symptom of a fault l Fault: Invalid execution state Symptom of an error May or may not produce a failure l Error: Defect or anomaly or “bug” in source code May or may not produce a fault

5. Topic 12Summer 2003 5 Examples: Faults, Errors, and Failures l Suppose node 6 should be X:= C*(A+2*B) Failure/Fault-less error: - Suppose the inputs are (A=2,B=1) – the executed path will be (1,2,4,5,7,8) which will not reveal this fault because 6 is not executed - Suppose the inputs are (A=-2,B=-1) – the executed path will be (1,2,3,5,6,8) which will not reveal the fault because C = 0 l Need to make sure proper test cases are selected the definitions of C at nodes 3 and 4 both affect the use of C at node 6 - executing path (1,2,4,5,6,8) will reveal the failure, but only if B != 0 - (e.g. Inputs (A=1,B=-2) )

6. Topic 12Summer 2003 6 Software Testing l Exercising a system [component] on some predetermined input data capturing the behavior and output data comparing with test oracle for the purposes of »identifying inconsistencies »verifying consistency between actual results and specification  to provide confidence in consistency with requirements and measurable qualities  to demonstrate subjective qualities »validating against user needs l Limitations only as good as the test data selected subject to capabilities of test oracle

7. Topic 12Summer 2003 7 Definition to execution (Levels of testing) l System Testing Defined at Requirements -> Run after integration testing l Integration Testing Defined at Design -> Run after Module Testing l Module Testing Defined at Design -> Run after Unit Testing l Unit Testing Defined at Implementation -> Run after Implementation of each unit l Regression Testing (testing after Change) Defined throughout the process -> Run after modifcations

8. Topic 12Summer 2003 8 Testing-oriented Life Cycle Process

9. Topic 12Summer 2003 9 How to make the most of limited resources? Fundamental Testing Questions l Test Criteria: What should we test? l Test Oracle: Is the test correct? l Test Adequacy: How much is enough? l Test Process: Is our testing effective?

10. Topic 12Summer 2003 10 Practical Issues l Purpose of testing Fault detection High assurance of reliability Performance/stress/load Regression testing of new versions l Conflicting considerations safety, liability, risk, customer satisfaction, resources, schedule, market windows and share l Test Selection is a sampling technique choose a finite set from an infinite domain

11. Topic 12Summer 2003 11 Test Criteria l Testing must select a subset of test cases that are likely to reveal failures l Test Criteria provide the guidelines, rules, strategy by which test cases are selected actual test data conditions on test data requirements on test data l Equivalence partitioning is the typical approach a test of any value in a given class is equivalent to a test of any other value in that class if a test case in a class reveals a failure, then any other test case in that class should reveal the failure some approaches limit conclusions to some chosen class of errors and/or failures

12. Topic 12Summer 2003 12 Test Adequacy l Coverage metrics when sufficient percentage of the program structure has been exercised l Empirical assurance when failures/test curve flatten out l Error seeding percentage of seeded faults found is proportional to the percentage of real faults found l Independent testing faults found in common are representative of total population of faults

13. Topic 12Summer 2003 13 Functional and Structural Testing l Functional Testing Test cases selected based on specification Views program/component as black box l Structural Testing Test cases selected based on structure of code Views program /component as white box (also called glass box testing)

14. Topic 12Summer 2003 14 Black Box vs. White Box Testing

15. Topic 12Summer 2003 15 Structural (White Box) Testing l Use source code to derive test cases l Build flow graph of program l State test requirements in terms of graph coverage l Various types of coverage identified l Choose test cases that guarantee different types of coverage All-Statements coverage All-Branches coverage All-Edges All-Paths

16. Topic 12Summer 2003 16 Graph representation of control flow and data flow relationships Flow Graphs l Control Flow The partial order of statement execution, as defined by the semantics of the language l Data Flow The flow of values from definitions of a variable to its uses

17. Topic 12Summer 2003 17 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 function P return INTEGER is begin X, Y: INTEGER; READ(X); READ(Y); while (X > 10) loop X := X – 10; exit when X = 10; end loop; if (Y < 20 and then X mod 2 = 0) then Y := Y + 20; else Y := Y – 20; end if; return 2*X + Y; end P; A Sample Program to Test

18. Topic 12Summer 2003 18 2,3,45 6 9´ 10 12 14 TT F F 9 T F 7 T F 9a9a 9b9b P’s Control Flow Graph (CFG) function P return INTEGER is begin X, Y: INTEGER; READ(X); READ(Y); while (X > 10) loop X := X – 10; exit when X = 10; end loop; 1234567812345678 9 10 11 12 13 14 15 if (Y < 20 and then X mod 2 = 0) then Y := Y + 20; else Y := Y – 20; end if; return 2*X + Y; end P;

19. Topic 12Summer 2003 19 All-Statements Coverage l Select test cases such that every node in the graph is visited Also called node coverage »Guarantees that every statement in the source code is executed at least once l Selects minimal number of test cases 137 824569 10

20. Topic 12Summer 2003 20 2,3,4 5 610 12 14 T F 9 T F 7 T F At least 2 test cases needed Example all-statements-adequate test set: (X = 20, Y = 10) (X = 20, Y = 30) All-Statements Coverage of P

21. Topic 12Summer 2003 21 All-Branches Coverage l Select test cases such that every branch in the graph is visited »Guarantees that every branch in the source code is executed at least once l More thorough than All-Statements coverage More likely to reveal logical errors 137 824569 10

22. Topic 12Summer 2003 22 2,3,45 610 12 14 T F 9 T F 7 T F At least 2 test cases needed Example all-branches-adequate test set: (X = 20, Y = 10) (X = 15, Y = 30) All-Branches Coverage of P

23. Topic 12Summer 2003 23 All-Edges Coverage l Select test cases such that every edge in the graph is visited »Takes complex statements into consideration – treats them as separate nodes l More thorough than All-Branches coverage More likely to reveal logical errors

24. Topic 12Summer 2003 24 2,3,45 6 9b9b 10 12 14 TT F F 9a9a T F 7 T F At least 3 test cases needed Example all-edges-adequate test set: (X = 20, Y = 10) (X = 5, Y = 30) (X = 21, Y = 10) All-Edges Coverage of P

25. Topic 12Summer 2003 25 All-Paths Coverage l Path coverage Select test cases such that every path in the graph is visited Loops are a problem »0, 1, average, max iterations l Most thorough… l …but is it feasible?

26. Topic 12Summer 2003 26 2,3,45 6 9b9b 10 12 14 TT F F 9a9a T F 7 T F Infinitely many test cases needed Example all-paths-adequate test set: (X = 5, Y = 10) (X = 15, Y = 10) (X = 25, Y = 10) (X = 35, Y = 10) … All-Paths Coverage of P

27. Topic 12Summer 2003 27 2,3,45 6 9b9b 10 12 14 T F 9a9a T F Y X X Y YX X Y Y X X X T F 7 T F X X P’s Control and Data Flow Graph

28. Topic 12Summer 2003 28 Subsumption of Criteria l C1 subsumes C2 if any C1-adequate testT is also C2-adequate But some T1 satisfying C1 may detect fewer faults than some T2 satisfying C2

29. Topic 12Summer 2003 29 all-statements all-edges boundary-interior loop testing min-max loop testing all-paths all-defs all-uses all-DU-paths all-p-usesall-c-uses C1C1 C2C2 subsumes all-branches Structural Subsumption Hierarchy

30. Topic 12Summer 2003 30 Equivalence partitioning l Identify the set of all inputs l Identify possible bases for subdividing the input Domain: size, order, structure correctness stated requirements & your smarts l Divide input set into (sub)domains using the basis “Equivalence partitions” Subdomains may overlap (may not be a partition) l Select representative test cases from each subdomain one test case may suffice, more is better

31. Topic 12Summer 2003 31 Example 1float homeworkAverage(float[] scores) { 2 float min = 99999; 3 float total = 0; 4 for (int i = 0 ; i < scores.length ; i++) { 5 if (scores[i] < min) 6 min = scores[i]; 7 total += scores[i]; 8 } 9 total = total – min; 10 return total / (scores.length – 1); 11}

32. Topic 12Summer 2003 32 Possible Bases l Array length empty array one element 2 or 3 elements 4 or more elements Input domain: float[] Basis: array length one small empty large } subdomains, eq. partitions

33. Topic 12Summer 2003 33 Possible Bases l Position of minimum score Smallest element first Smallest element in middle Smallest element last Input domain: float[] Basis: position of minima somewhere in middle first last

34. Topic 12Summer 2003 34 Possible Bases l Number of minima Unique minimum A few minima All minima Input domain: float[] Basis: number of minima all data equal 1 minimum 2 minima

35. Topic 12Summer 2003 35 Equivalence partitions & white-box testing The basis - equivalence partition - test case approach works for structural testing, too. A basis could be nodes, edges, or paths. Each node, edge, or path defines a partition of the input.

36. Topic 12Summer 2003 36 Challenges l Structural testing can cover all nodes or edges without revealing obvious faults No matter what input, program always returns 0 l Some nodes, edges, or loop combinations may be infeasible Unreachable/unexecutable code l “Thoroughness” A test suite that guarantees edge coverage also guarantees node coverage… …but it may not find as many faults as a different test suite that only guarantees node coverage

37. Topic 12Summer 2003 37 Limitations of Structural Testing l Interactive programs l Listeners, event-driven programs l Concurrent programs or tasks; threads l Exceptions l Self-modifying programs (assembly language) l Objects loaded over the network l Super-class constructors l Asynchronous garbage collection

38. Topic 12Summer 2003 38 Summary of Structural Testing l Conceptually simple l Relies on access to source l Not applicable in all situations l Acts in ignorance of specifications

39. Topic 12Summer 2003 39 Ensure that changes made during maintenance do not destroy existing functionality Regression Testing l Permanent suite of test cases Saves effort creating test cases Provides record of existing functionality l Add new test cases and delete obsolete ones when necessary

40. Topic 12Summer 2003 40 Cost reduction: Select minimal subset of regression test suite that tests the changes Selective Regression Testing l Analyze relationship between the test cases and the software entities they cover l Use information about changes to select test cases for new version

41. Topic 12Summer 2003 41 = modified entity System Under Test Test 2Test 1Test 3 = exercises relation = entity Simple Example

42. Topic 12Summer 2003 42 System Under Test Test 3 = exercises relation = entity = modified entity Selective Regression Testing

43. Topic 12Summer 2003 43 Cost of running necessary tests Retest All Cost-Ineffective Selective Retest Cost of running necessary tests Cost Cost-Effective Selective Retest Cost of running unnecessary tests Cost of additional analysis The Cost-Effectiveness Tradeoff

44. Topic 12Summer 2003 44 The Reality of Software Testing l One of the most widely-used, practical class of techniques for revealing faults in software l An area of software engineering showing the greatest gap between theory and practice