1. Software Testing

2. 2 CMSC 345, Version 4/12 Topics The testing process  unit testing  integration and system testing  acceptance testing Test case planning and tracking Testing types  black box vs. white box testing  coverage testing  regression testing  stress testing Test case prioritization

3. 3 CMSC 345, Version 4/12 Testing Goal is to discover defects The only validation technique for non- functional requirements Successful test: causes a program to behave in an anomalous way Testing shows the presence, not the absence, of defects.

4. 4 CMSC 345, Version 4/12 Unit Testing Integration and System Testing Acceptance Testing Testing Process Overview unit 1 unit 2 unit 3 unit n unit Subsystem 1 unit Subsystem 2 unit Subsystem m Final System

5. 5 CMSC 345, Version 4/12 unit Subsystem The Testing Process Elaborated Unit (component) testing  Testing of individual program components (e.g., functions, methods, or classes)  Usually the responsibility of the component’s developer  Must be structured  White box testing Integration and system testing  Testing of groups of components integrated to create a subsystem or the entire system  Sometimes the responsibility of an independent testing team  Tests are based on system specification (black box testing)  Is iterative (keeps building)  May include alpha and beta testing Acceptance testing  Run in the presence of customer or by customer  Used to validate all system requirements unit Final System $$$$!

6. 6 CMSC 345, Version 4/12 Unit (Component) Testing You may do this on your own, but always … have a plan keep records automate!  JUnit (for Java) or other tools  even #ifdef - #endif pairs (C/C++)  main( ) method (Java)

7. 7 CMSC 345, Version 4/12 Integration Testing Approaches – Procedural Programming Top-down testing  Start with high-level components and integrate from top down  Uses stubs Bottom-up testing  Start with low-level components and integrate from bottom up  Uses drivers Sandwich testing  Combination of top-down and bottom-up Thread testing  Tests one piece of functionality at a time

8. 8 CMSC 345, Version 4/12 n Level 0 1 Top-down Integration & Testing Being Tested Stub Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary

9. 9 CMSC 345, Version 4/12 n Level 0 1 Top-down Integration & Testing Tested Being Tested Stub Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary

10. 10 CMSC 345, Version 4/12 n Level 0 1 Top-down Integration & Testing Tested Being Tested Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary

11. 11 CMSC 345, Version 4/12 n Level 0 1 Bottom-up Integration & Testing Being Tested Driver Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any

12. 12 CMSC 345, Version 4/12 n Level 0 1 Bottom-up Integration & Testing Being Tested Tested Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any

13. 13 CMSC 345, Version 4/12 n Level 0 1 Bottom-up Integration & Testing Tested Driver Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any

14. 14 CMSC 345, Version 4/12 n Level 0 1 Sandwich Integration & Testing Uses both stubs and drivers

15. 15 CMSC 345, Version 4/12 n Level 0 1 Thread Integration & Testing May be done top-down, bottom-up, or sandwich

16. 16 CMSC 345, Version 4/12 Exercise (on your own!) What are some of the benefits associated with each of the following integration and testing methods? Top-down Bottom-up Sandwich Thread

17. 17 CMSC 345, Version 4/12 Integration Testing Approaches – Object-oriented Programming Use based  Start with classes that have no dependencies  Move “out” to next “level” of classes, and so on  Similar to bottom-up for procedural programming Cluster  Test clusters of associated classes  Integrate the clusters  May be combined with use based testing (above) Scenario  Based on functional specifications (possibly expressed as use cases)  Similar to thread testing for procedural programming

18. 18 CMSC 345, Version 4/12 System Testing Alpha  Internal to development organization, but not the developers themselves Beta  External to development organization Acceptance  Performed in the presence of or possibly by the customer  Determines final “acceptance” by the customer

19. 19 CMSC 345, Version 4/12 Test Case Planning and Tracking Natural language description ID Specific values (if possible) Pass, Fail, other notes How to determine Pass or Fail

20. 20 CMSC 345, Version 4/12 A Sample Testing Form Before Testing

21. 21 CMSC 345, Version 4/12 A Sample Testing Form After Testing

22. 22 CMSC 345, Version 4/12 A Sample Testing Form Before Testing

23. 23 CMSC 345, Version 4/12 A Sample Testing Form After Testing

24. 24 CMSC 345, Version 4/12 White Box Testing Sometimes called structural testing Can “see inside” the code unit Objectives:  Exercise unique algorithms  Exercise all program statements at least once Typically tests small program units such as functions or class methods code code code code code input output

25. 25 CMSC 345, Version 4/12 Coverage Testing A technique for white box testing Ensures that each statement is executed at least once Use a program flow graph to derive paths Find minimum paths needed to “cover” all statements (there are tools to do this) Drive the code through all of these paths All branches are executed, but not all combinations of branches. Some paths may be impossible to test.

26. #include using namespace std; void triangle (int, int, int);// parameters are sides of the triangle int main() { int s1, s2, s3; char repeat; cout << "This program will accept the sides of a triangle\n" << " and tell you the type of triangle.\n" << endl; do { cout << "Enter the 3 sides: " ; cin >> s1 >> s2 >> s3; triangle(s1, s2, s3); cout << "Want to do another? (Y or N) "; cin >> repeat; cout << endl << endl; } while (repeat =='Y' || repeat == 'y'); cout <<"Program terminated" << endl << endl; cout << "Enter any character to continue" << endl; cin >> repeat; return 0; } void triangle (int side1, int side2, int side3) { // Error checking /***********************************/ // Check sides for validity if (side1 <= 0 || side2 <= 0)// 1 cout << "Illegal value for side" << endl << endl;// 2 // Check Triangle inequality if (side1+side2 <= side3 || side2+side3 <= side1)// 3 cout << "Triangle inequality violation" << endl << endl;// 4 /***********************************/ // Pick triangle type if (side1 == side2 && side2 == side3)// 5 cout << "Triangle is equilateral" << endl;// 6 else if (side1 == side2 || side2 == side3 || side1 == side3)// 7 cout << "Triangle is isosceles" << endl;// 8 else if (side1*side1 + side2*side2 == side3*side3 ||// 9 side1*side1 + side3*side3 == side2*side2) cout << "Triangle is right" << endl;//10 else cout <<"Triangle is scalene" << endl;//11 return;//12 } D. Cheslock, Northrup Grumman, 2009 triangle.cpp CMSC 345, Version 4/12 26

27. triangle.exe Control Graph Vertices= 18 Edges = 28 Regions= 12 V(g) = 28 - 18 + 2 (e-n+2) = 11 + 1(p+1) = 12 1 7c 7b 5b 1b 3b 1a 2 3a 4 5a 67a 8 9a 10 11 12 9b 2 3 4 5 6 7 8 9 11 10 12 Baseline path D. Cheslock, Northrup Grumman, 2009 CMSC 345, Version 4/12 27

28. A Set Of Basis Paths For triangle.cpp 1)1a - 1b - 3a - 3b - 5a - 7a - 7b -7c -9a - 9b - 11 - 12 sample data: 3, 5, 7 (baseline path) 2)1a - 2 - 3a - 3b - 5a - 7a - 7b -7c -9a - 9b - 11 - 12 sample data: -3, 7, 3 3)1a - 1b - 2 - 3a - 3b - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12 sample data: can’t perform this test 4)1a - 1b - 3a - 4 - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12 sample data: 2, 3, 7 5)1a - 1b - 3a - 3b - 4 - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12 sample data: 7, 3, 2 6)1a - 1b - 3a - 3b - 5a - 5b - 6 - 12 sample data: 5, 5, 5 7)1a - 1b - 3a - 3b - 5a - 5b - 7a - 7b -7c - 9a - 9b - 11 - 12 sample data: can’t perform this test 8)1a - 1b - 3a - 3b - 5a - 5b - 7a - 8 - 12 sample data: 4, 4, 2 9)1a - 1b - 3a - 3b - 5a - 7a - 7b - 8 - 12 sample data: 2, 4, 4 10)1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 8 - 12 sample data: 4, 2, 4 11)1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 9a - 10 - 12 sample data: 3, 4, 5 12)1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 9a - 9b - 10 - 12 sample data: 3, 5, 4 D. Cheslock, Northrup Grumman, 2009 CMSC 345, Version 4/12 28

29. 29 CMSC 345, Version 4/12 Metrics – Cyclomatic Complexity The minimum number of tests needed to execute all statements CC = number_edges – number_nodes + 2 OR CC = number_decisions + 1 OR the number of regions

30. 30 CMSC 345, Version 4/12 Black Box Testing Cannot “see inside” code unit Test cases are:  based on the system specification and/or interfaces (preconditions, postconditions, invariants, and parameters) Use equivalence partitions when conducting black box testing input output

31. 31 CMSC 345, Version 4/12 Equivalence Partitions and Boundary Testing Equivalence Partition A group of input values that will produce equivalent behavior Example  Square root function the set of all numbers >= zero the set of all negative numbers Test cases should be chosen from each partition, especially at the boundaries.

32. 32 CMSC 345, Version 4/12 Equivalence Partitions and Boundary Testing Exercise A system accepts 4 to 10 inputs that are 5- digit integers >= 10,000 Partition the system inputs into groups (partitions) that should cause equivalent behavior. Include both valid and invalid inputs. Ian Somerville, Software Engineering, 6 th edition

33. 33 CMSC 345, Version 4/12 Ian Somerville, Software Engineering, 6 th edition The six partitions with corresponding boundary values

34. 34 CMSC 345, Version 4/12 Regression Testing Code A Test 1 Test 2 Code A Code B Code C Test 1 Test 2 Test 3 Test 4 Test 5 Code A Code B Code C Code D Test 2 Test 3 Test 4 Test 5 Test 1 Test 6 Test 7 Test 8 … and so on

35. 35 CMSC 345, Version 4/12 Stress Testing Example: A child’s car seat Exercises the system beyond its maximum design load. Examples:  Exceed string lengths  Store/manipulate more data than in specification  Load system with more users than in specification Often causes defects to come to light Systems should not fail catastrophically.  Stress testing checks for unacceptable loss of service or data.

36. 36 CMSC 345, Version 4/12 Test Case Prioritization Why? To locate defects quickly to:  begin debugging ASAP  debug and deliver in increments ASAP To locate the majority of errors quickly Other How? Can prioritize according to:  module or functional dependencies  module size  module complexity  module suspected to be the most error-prone  most “important” module or functionality  other

37. 37 CMSC 345, Version 4/12 References Somerville, I., Software Engineering, 6 th ed SourceForge.net, JUnit, http://junit.sourceforge.net, accessed 11/6/07 http://junit.sourceforge.net JUnit.org, http://www.junit.org, accessed 11/6/07http://www.junit.org