1. CS5103 Software Engineering Lecture 15 System Testing Testing Coverage

2. 2 Higher level testing  Integration Testing  Testing the interaction among a number of interactive components  System Testing  Testing the system as a whole, considering various environments, external exceptions  Acceptance Testing  Final user testing  Usually on GUI

3. 3 Integration Testing  Strategies  Big Bang  Top down  Bottom Up

4. 4 Big Bang!  Prepare all relevant components  Data, Global variables…  Put them together  Pray!

5. 5 Usage Scenario of Big Bang  Quite common in small projects  Requires no extra effort for integration  May work well if all interfaces are well-defined … Not likely to happen…

6. 6 Top down strategy  Feature decomposition  A hierarchical structure of all software features  May not require extra efforts: You should use it in requirement and design

7. 7 An example feature decomposition

8. 8 Top down strategy  Focus on the validation of the whole system first  Test an empty system with no components at first  All components are replaced with test doubles, e.g., stubs  Gradually add components  Until all components are added  Advantages  Easier to understand the process of the integration  May have a working system earlier (or always have a working system): important for modern development, such as XP

9. 9 Top-down strategy: Illustration

10. 10 Top down strategy  Disadvantages  Requires to write test doubles  Since DOC matters in integration, so more complex test doubles is usually required  Centralized  So the integration cannot be done parallelly

11. 11 Bottom-up strategy  Focus on the integration of lowest units at first  Start from an unit depending nothing else  When all sub-unit of a component are integrated, integrate the component  Until the whole system is built

12. 12 Bottom-up strategy: Illustration Bottom Level Subtree Middle Level Subtree Top Level Subtree

13. 13 Bottom-up strategy  Advantages  No test stub is required  Requires test driver but may re-use unit test cases (view the whole component as a unit)  Support parallel integration  Issues  No working system is available until the end  Higher risk  Require more interactions among sub-teams working on each component

14. 14 System testing  Test the system as a whole  Usually test against specifications  For each item in the specification  Work out a test case and a test oracle  Test boundary values  Test with invalid inputs  Test with environment errors

15. 15 Environment issues  Building  Compile options and configurations  Environment variables  Third party dependencies  Underlying platforms  OS, database, application server, browser  Compatibility is a huge problem  Different platforms (if cross-platform)  Different versions of dependencies and platforms  Different configurations

16. 16 Acceptance Testing  Command Line  Writing test scripts  With different options and inputs  Try invalid cases : missing option parameter, invalid paths, etc.  GUI Testing

17. 17 GUI testing  Testing Graphics User Interface is easier  But harder to automate  And harder to compare results with oracles  Manual testing is still widely performed for GUI testing  Manually explore the user interface  Record the steps in the test for future testings  Observe the GUI for errors

18. 18 GUI testing  Record and Replay  Screen record and replay  Record click on the screen and keyboard inputs  Replay all the clicks and inputs  Not robust, affected by screen size, resolution, resize, …  Event record and replay  Record all the UI events in the UI framework (swing, MFC, android etc.), and outputs (texts) in the UI  Re-trigger all the events and compare the UI texts  More robust, but requires more preparation and overhead

19. 19 Today’s class  Higher Level Testing  Software Testing Coverage  Code Coverage  Input Combination Coverage  Mutation Coverage

20. 20 Test Coverage  After we have done some testing, how do we know the testing is enough?  The most straightforward: input coverage  # of inputs tested / # of possible inputs  Unfortunately, # of possible inputs is typically infinite  Not feasible, so we need approximations…

21. 21 Test Coverage  Code Coverage  Specification Coverage  Model coverage  Error coverage

22. 22 Code Coverage  Basic idea:  Bugs in the code that has never been executed will not be exposed  So the test suite is definitely not sufficient  Definition:  Divide the code to elements  Calculate the proportion of elements that are executed by the test suite

23. 23 Code Coverage  Criteria  Statement (basic block) coverage, are they the same?  Branch coverage (cover all edges in a control flow graph), same with basic block coverage?  Data flow coverage  Class/Method coverage

24. 24 Control Flow Graph How many test cases to achieve full statement coverage?

25. 25 Statement Coverage in Practice  Microsoft reports 80-90% statement coverage  Safely-critical software must achieve 100% statement coverage  Usually about 85% coverage, 100% for large systems is usually very hard

26. 26 Statement Coverage: Example

27. 27 Branch Coverage  Cover the branches in a program  A branch is consider executed when both (All) outcomes are executed  Also called multiple-condition coveage

28. 28 Control Flow Graph How many test cases to achieve full branch coverage?

29. 29 Branch Coverage: Example

30. 30 Branch Coverage: Example An untested flow of data from an assignment to a use of the assigned value, could hide an erroneous computation Even though we have 100% statement and branch coverage

31. 31 Data Flow Coverage  Cover all def-use pairs in a software  Def: write to a variable  Use: read of a variable  Use u and Def d are paired when d is the direct precursor of u in certain execution

32. 32 Data Flow Coverage  Formula  Not easy to locate all use-def pairs  Easy for inner-procedure (inside a method)  Very difficult for inter-procedure  Consider the write to a field var in one method, and the read to it in another method

33. 33 Path coverage  The strongest code coverage criterion  Try to cover all possible execution paths in a program  Covers all previous coverage criteria?  Usually not feasible  Exponential paths in acyclic programs  Infinite paths in some programs with loops

34. 34 Path coverage  N conditions  2 N paths  Many are not feasible  e.g., L 1 L 2 L 3 L 4 L 6 X = 0 => L 1 L 2 L 3 L 4 L 5 L 6 X = -1 => L 1 L 3 L 4 L 6 X = -2 => L 1 L 3 L 4 L 5 L 6

35. 35 Control Flow Graph How many paths? How many test cases to cover?

36. 36 Path coverage, not enough 1. main() { 2. int x, y, z, w; 3. read(x); 4. read(y); 5. if (x != 0) 6. z = x + 10; 7. else 8. z = 1; 9. if (y>0) 10. w = y / z; 10. else 11. w = 0; 12.} Test Requirements: – 4 paths Test Cases – (x = 1, y = 22) – (x = 0, y = 10) – (x = 1, y = -22) – (x = 1, y = -10) We are still not exposing the fault ! Faulty if x = -10 – Structural coverage cannot reveal this error

37. 37 Method coverage  So far, all examples are inner-method  Quite useful in unit testing  It is very hard to achieve 100% statement coverage in system testing  Need higher level code element  Method coverage  Similar to statements  Node coverage : method coverage  Edge coverage : method invocation coverage  Path coverage : stack trace coverage

38. 38 Method coverage

39. 39 Code coverage: summary  Coverage of code elements and their connections  Node coverage:  Class/method/statement/predicate coverage  Edge coverage:  Branch/Dataflow/MethodInvok  Path coverage:  Path/UseDefChain/StackTrace

40. 40 Code coverage: limitations  Not enough  Some bugs can not be revealed even with full path coverage  Cannot reveal bugs due to missing code

41. 41 Code coverage: practice  Though not perfect, code coverage is the most widely used technique for test evaluation  Also used for measure progress made in testing  The criteria used in practice are mainly:  Method coverage  Statement coverage  Branch coverage  Loop coverage with heuristic (0, 1, many)

42. 42 Code coverage: practice  Far from perfect  The commonly used criteria are the weakest, recall our examples  A lot of corner (they are not so corner if just not found by statement coverage) cases can never be found  100% code coverage is rarely achieved  Some commercial software products released with around 60% code coverage  Many open source software even lower than 50%

43. 43 Input Combinatioin Coverage  Basic idea  Origins from the most straightforward idea  In theory, proof of 100% correctness when achieve 100% coverage in theory  In practice, on very trivial cases  Main problems  Combinations are exponential  Possible values are infinite

44. 44 Input Combination Coverage  An example on a simple automatic sales machine  Accept only 1$ bill once and all beverages are 1$  Coke, Sprite, Juice, Water  Icy or normal temperature  Want receipt or not  All combinations = 4*2*2 = 16 combinations  Try all 16 combinations will make sure the system works correctly

45. 45 Combination Explosion  Combinations are exponential to the number of inputs  Consider an annual report system with 100 yes/no questions to generate a customized form for you (if you are not eligible for some questions, it will be not on the form)  2 100 combinations = about 10 30 test cases

46. 46 Observation  When there are many inputs, usually only a small number of software inputs are related to each other  The previous example: Maybe only icy coke and sprite, but receipt is independent  A long term study from NIST (national institute of standardization technology)  A combination width of 4 to 6 is enough for detecting almost all errors

47. 47 N-wise coverage  Coverage on N-wise combination of the possible values of all inputs  Example: 2-wise combinations  (coke, icy), (sprite, icy), (water, icy), (juice, icy)  (coke, normal), (sprite, normal), …  (coke, receipt), (sprite, receipt), …  (coke, no-receipt), (sprite, no-receipt), …  (icy, receipt), (normal, receipt)  (icy, no-receipt), (normal, no-receipt)  20 combinations in total

48. 48 N-wise coverage  Note: One test case may cover multiple N-wise combinations  E.g., (Coke, Icy, Receipt) covers 3 2-wise combinations  (Coke, Icy), (Coke, Receipt), (Icy, Receipt)  Note: 100% N-wise coverage will fully cover N-1- wise coverage  For K boolean inputs  Full combination coverage = 2 k combinations: exponential  Full n-wise coverage = 4*k*(k-1)* … *(k-n+1)/n! combinations: polynomial, for 2-wise combination, 2*k*(k-1)

49. 49 N-wise coverage: Example  How many test cases for 100% 2-wise coverage of our example  (coke, icy, receipt), covers 3 new 2-wise combinations  (sprite, icy, no-receipt), cover 3 new …  (juice, icy, receipt), covers 2 new …  (water, icy, receipt), covers 2 new …  (coke, normal, no-receipt), covers 3 new …  (sprite, normal, receipt), cover 3 new …  (juice, normal, no-receipt), covers 2 new …  (water, normal, no-receipt), covers 2 new …  8 test cases covers all 20 2-wise combinations

50. 50 Combination Coverage in Practice  2-wise combination coverage is very widely used  Pair-wise testing  All pairs testing  Mostly used in configuration testing  Example: configuration of gcc  All lot of variables  Several options for each variable  For command line tools: add or remove an option

51. 51 Input model  What happened if an input has infinite possible values  Integer  Float  Character  String  Note: all these are actually finite, but the possible value set is too large, so that they are deemed as infinite  Idea: map infinite values to finite value baskets (ranges)

52. 52 Input model  Input partition  Partition the possible value set of a input to several value ranges  Transform numeric variables (integer, float, double, character) to enumerated variables  Example:  int exam_score => {less than -1}, {0, 59}, {60,69}, {70,79}, {80,89}, {90, 100}, {100+}  char c => {a, z}, {A,Z}, {0,9}, {other}

53. 53 Input model  Feature extraction  For string and structure inputs  Split the possible value set with a certain feature  Example: String passwd => {contains space}, {no space}  It is possible to extract multiple features from one input  Example: String name => {capitalized first letter}, {not} => {contains space}, {not} => {length >10}, {2-10}, {1}, {0} One test case may cover multiple features

54. 54 Input combination coverage  Summary:  Try to cover the combination of possible values of inputs  Exponential combinations:  N-wise coverage  2-wise coverage is most popular, all pairs testing  Infinite possible values  Input partition  Input feature extraction  Coverage is usually 100% once adopted  It is easy to achieve, compared with code coverage  Models are not easy to write

55. 55 Specification Coverage  A type of input coverage  Covers the written formal specification in the requirement document  Example  When a number smaller than 0 is fed in, the system should report error => testcase: -1  Sometimes can be a sequence of inputs  When you input correct user name, a passwd prompt is shown, after you input the correct passwd, the user profile will be shown, … => testcase: xiaoyin, xxxxx, …

56. 56 Specification Coverage  Widely used in industry  Advantages  Target at the specification  No need for writing oracles  Usually can achieve 100% coverage  Disadvantages  Very hard to automate  can only be automated with formal specifications  No guarantee to be complete  Quality highly depend on the specification

57. 57 Today’s class  Unit Testing  System Testing  GUI Testing  Software Testing Coverage  Code Coverage  Input Combination Coverage  Mutation Coverage

58. 58 Next class  Regression Testing  Test prioritization  Total strategy  Additional strategy  Test reduction

59. 59 Thanks!