1. Introduction to Software Testing Chapter 1 Introduction Paul Ammann & Jeff Offutt www.introsoftwaretesting.com

2. What is testing? 2 Software under test Test inputsTest outputs Outputs OK? Yes Debug Acceptability test

3. What is testing? 3 Test outputsOK? Bug in SUT? Bug in Acceptability test? Bug in specification? No Yes Bug in OS, compiler, libraries, hardware No Yes

4. Types of Test Activities – Summary n These four general test activities are quite different n It is a poor use of resources to use people inappropriately Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 4 1.Test designDesign test values to satisfy coverage criteria or other engineering goal Requires technical knowledge of discrete math, programming and testing 2.Test automationEmbed test values into executable scripts Requires knowledge of scripting 3.Test executionRun tests on the software and record the results Requires very little knowledge 4.Test evaluationEvaluate results of testing, report to developers Requires domain knowledge But most test organizations use the same people for ALL FOUR activities !!

5. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 5 Important Terms Validation & Verification ( IEEE ) n Validation : The process of evaluating software at the end of software development to ensure compliance with intended usage n Verification : The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase IV&V stands for “independent verification and validation”

6. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 6 Static and Dynamic Testing n Static Testing : Testing without executing the program n Dynamic Testing : Testing by executing the program with real inputs

7. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 7 Software Faults, Errors & Failures n Software Fault : A static defect in the software n Software Error : An incorrect internal state that is the manifestation of some fault n Software Failure : External, incorrect behavior with respect to the requirements or other description of the expected behavior Faults in software are design mistakes and will always exist

8. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 8 Testing & Debugging n Testing : Finding inputs that cause the software to fail n Debugging : The process of finding a fault given a failure

9. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 9 Fault & Failure Model Three conditions necessary for a failure to be observed 1.Reachability : The location or locations in the program that contain the fault must be reached 2.Infection : The state of the program must be incorrect 3.Propagation : The infected state must propagate to cause some output of the program to be incorrect

10. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 10 Test Case n Test Case Values : The values that directly satisfy one test requirement n Expected Results : The result that will be produced when executing the test if the program satisfies its intended behavior

11. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 11 White-box and Black-box Testing n Black-box testing : Deriving tests from external descriptions of the software, including specifications, requirements, and design n White-box testing : Deriving tests from the source code internals of the software, specifically including branches, individual conditions, and statements This view is really out of date. The more general question is: from what level of abstraction to we derive tests?

12. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 12 Changing Notions of Testing n Old view of testing is of testing at specific software development phases –Unit, module, integration, system … n New view is in terms of structures and criteria –Graphs, logical expressions, syntax, input space

13. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 13 Old : Testing at Different Levels Class A method mA1() method mA2() Class B method mB1() method mB2() main Class P n Acceptance testing: Is the software acceptable to the user? n Integration testing: Test how modules interact with each other n System testing: Test the overall functionality of the system n Module testing: Test each class, file, module or component n Unit testing: Test each unit (method) individually

14. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 14 New : Test Coverage Criteria  Test Requirements : Specific things that must be satisfied or covered during testing  Test Criterion : A collection of rules and a process that define test requirements A tester’s job is simple :Define a model of the software, then find ways to cover it Testing researchers have defined dozens of criteria, but they are all really just a few criteria on four types of structures …

15. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 15 New : Criteria Based on Structures 1.Graphs 2.Logical Expressions 3.Input Domain Characterization 4.Syntactic Structures (not X or not Y) and A and B if (x > y) z = x - y; else z = 2 * x; Structures : Four ways to model software A: {0, 1, >1} B: {600, 700, 800} C: {swe, cs, isa, infs}

16. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 16 1. Graph Coverage – Structural 6 5 3 2 17 4 Node (Statement) Cover every node 12567 1343567 This graph may represent statements & branches methods & calls components & signals states and transitions Edge (Branch) Cover every edge 12567 1343567 1357 Path Cover every path 12567 1257 13567 1357 1343567 134357 …

17. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 17 Defs & Uses Pairs (x, 1, (1,2)), (x, 1, (1,3)) (y, 1, 4), (y, 1, 6) (a, 2, (5,6)), (a, 2, (5,7)), (a, 3, (5,6)), (a, 3, (5,7)), (m, 2, 7), (m, 4, 7), (m, 6, 7) 1. Graph Coverage – Data Flow 6 5 3 2 17 4 This graph contains: defs: nodes & edges where variables get values uses: nodes & edges where values are accessed def = {x, y} def = {a, m} def = {a} def = {m} use = {x} use = {a} use = {y} use = {m} use = {y}

18. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 18 1. Graph - FSM Example Memory Seats in a Lexus ES 300 Driver 1 Configuration Driver 2 Configuration [Ignition = off] | Button2 [Ignition = off] | Button1 Modified Configuration sideMirrors ()[Ignition = on] | lumbar ()[Ignition = on] | seatBottom ()[Ignition = on] | seatBack () [Ignition = on] | New Configuration Driver 1 New Configuration Driver 2 [Ignition = on] | Reset AND Button1 [Ignition = on] | Reset AND Button2 Ignition = off (to Modified) Guard (safety constraint) Trigger (input)

19. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 19 2. Logical Expressions ( (a > b) or G ) and (x < y) Transitions Software Specifications Program Decision Statements Logical Expressions

20. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 20 2. Logical Expressions n Predicate Coverage : Each predicate must be true and false –( (a>b) or G ) and (x < y) = True, False n Clause Coverage : Each clause must be true and false –(a > b) = True, False –G = True, False –(x < y) = True, False n Combinatorial Coverage : Various combinations of clauses ( (a > b) or G ) and (x < y)

21. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 21 2. Logic – Active Clause Coverage ( (a > b) or G ) and (x < y) 1 T F T 2 F F T duplicate 3 F T T 4 F F T 5 T T T 6 T T F With these values for G and (x b) determines the value of the predicate

22. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 22 3. Input Domain Characterization n Describe the input domain of the software –Identify inputs, parameters, or other categorization –Partition each input into finite sets of representative values –Choose combinations of values n System level –Number of students { 0, 1, >1 } –Level of course { 600, 700, 800 } –Major { swe, cs, isa, infs } n Unit level –Parameters F (int X, int Y) –Possible values X: { 2 }, Y : { 10, 20, 30 } –Tests F (-5, 10), F (0, 20), F (1, 30), F (2, 10), F (5, 20)

23. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 23 4. Syntactic Structures n Based on a grammar, or other syntactic definition n Primary example is mutation testing 1.Induce small changes to the program: mutants 2.Find tests that cause the mutant programs to fail: killing mutants 3.Failure is defined as different output from the original program 4.Check the output of useful tests on the original program n Example program and mutants if (x > y) z = x - y; else z = 2 * x; if (x > y)  if (x >= y) z = x - y;  z = x + y;  z = x – m; else z = 2 * x;

24. Source of Structures n These structures can be extracted from lots of software artifacts –Graphs can be extracted from UML use cases, finite state machines, source code, … –Logical expressions can be extracted from decisions in program source, guards on transitions, conditionals in use cases, … Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 24

25. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 25 Coverage Overview Four Structures for Modeling Software Graphs Logic Input Space Syntax Use cases Specs Design Source Applied to DNF Specs FSMs Source Input Models Integ Source

26. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 26 Coverage n Infeasible test requirements : test requirements that cannot be satisfied –No test case values exist that meet the test requirements –Detection of infeasible test requirements is formally undecidable for most test criteria n Thus, 100% coverage is impossible in practice Given a set of test requirements TR for coverage criterion C, a test set T satisfies C coverage if and only if for every test requirement tr in TR, there is at least one test t in T such that t satisfies tr

27. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 27 Generators and Recognizers n Generator : A procedure that automatically generates values to satisfy a criterion n Recognizer : A procedure that decides whether a given set of test values satisfies a criterion n It is possible to recognize whether test cases satisfy a criterion far more often than it is possible to generate tests that satisfy the criterion n Coverage analysis tools are quite plentiful

28. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 28 Comparing Criteria with Subsumption n Criteria Subsumption : A test criterion C1 subsumes C2 if and only if every set of test cases that satisfies criterion C1 also satisfies C2 n Must be true for every set of test cases n Example : If a test set has covered every branch in a program (satisfied the branch criterion), then the test set is guaranteed to also have covered every statement

29. Introduction to Software Testing Chapter 2.1, 2.2 Overview Graph Coverage Criteria Paul Ammann & Jeff Offutt www.introsoftwaretesting.com

30. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 30 Three Example Graphs 0 21 3 N 0 = { 0 } N f = { 3 } 0 21 3 N 0 = { } N f = { 3 } 9 0 43 7 1 5 8 2 6 N 0 = { 0, 1, 2 } N f = { 7, 8, 9 } Not a validgraph

31. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 31 Paths in Graphs n Path : A sequence of nodes – [n 1, n 2, …, n M ] –Each pair of nodes is an edge n Length : The number of edges –A single node is a path of length 0 n Subpath : A subsequence of nodes in p is a subpath of p n Reach (n) : Subgraph that can be reached from n 9 7 8 0 1 2 43 5 6 Paths [ 0, 3, 7 ] [ 1, 4, 8, 5, 1 ] [ 2, 6, 9 ] Reach (0) = { 0, 3, 4, 7, 8, 5, 1, 9 } Reach ({0, 2}) = G Reach([2,6]) = {2, 6, 9}

32. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 32 Test Paths n Test Path : A path that starts at an initial node and ends at a final node n Test paths represent execution of test cases 0 2 1 63 5 4 Double-diamond graph Four test paths [ 0, 1, 3, 4, 6 ] [ 0, 1, 3, 5, 6 ] [ 0, 2, 3, 4, 6 ] [ 0, 2, 3, 5, 6 ]

33. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 33 Visiting and Touring n Visit : A test path p visits node n if n is in p A test path p visits edge e if e is in p n Tour : A test path p tours subpath q if q is a subpath of p Path [ 0, 1, 3, 4, 6 ] Visits nodes 0, 1, 3, 4, 6 Visits edges (0, 1), (1, 3), (3, 4), (4, 6) Tours subpaths (0, 1, 3), (1, 3, 4), (3, 4, 6), (0, 1, 3, 4), (1, 3, 4, 6)

34. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 34 Tests and Test Paths n path (t) : The test path executed by test t n path (T) : The set of test paths executed by the set of tests T n Each test executes one and only one test path n A location in a graph (node or edge) can be reached from another location if there is a sequence of edges from the first location to the second –Syntactic reach : A subpath exists in the graph –Semantic reach : A test exists that can execute that subpath

35. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 35 Tests and Test Paths test 1 test 2 test 3 many-to-one test 1 test 2 test 3 many-to-many Test Path 1 Test Path 2 Test Path 3 Non-deterministic software – a test can execute different test paths Test Path Deterministic software – a test always executes the same test path

36. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 36 Testing and Covering Graphs (2.2) n We use graphs in testing as follows : –Developing a model of the software as a graph –Requiring tests to visit or tour specific sets of nodes, edges or subpaths Test Requirements (TR) : Describe properties of test paths Test Criterion : Rules that define test requirements Satisfaction : Given a set TR of test requirements for a graph criterion C, a test set T satisfies C on graph G if and only if for every test requirement tr in TR, there is a test path p in path(T) such that p meets tr. Structural Coverage Criteria : Defined on a graph just in terms of nodes and edges Data Flow Coverage Criteria : Requires a graph to be annotated with references to variables

37. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 37 Node and Edge Coverage n The first (and simplest) two criteria require that each node and edge in a graph be executed Node Coverage (NC) : Test set T satisfies node coverage on graph G iff for every syntactically reachable node n in N, there is some path p in path(T) such that p visits n. Node Coverage (NC) : TR contains each reachable node in G. This statement is a bit cumbersome, so we abbreviate it in terms of the set of test requirements

38. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 38 Node and Edge Coverage n Edge coverage is slightly stronger than node coverage Edge Coverage (EC) : TR contains each reachable path of length up to 1, inclusive, in G. The “length up to 1” allows for graphs with one node and no edges NC and EC are only different when there is an edge and another subpath between a pair of nodes (as in an “ if-else ” statement) Node Coverage : TR = { 0, 1, 2 } Test Path = [ 0, 1, 2 ] Edge Coverage : TR = { (0,1), (0, 2), (1, 2) } Test Paths = [ 0, 1, 2 ] [ 0, 2 ] 1 2 0

39. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 39 Covering Multiple Edges n Edge-pair coverage requires pairs of edges, or subpaths of length 2 Edge-Pair Coverage (EPC) : TR contains each reachable path of length up to 2, inclusive, in G. The “length up to 2” is used to include graphs that have less than 2 edges Complete Path Coverage (CPC) : TR contains all paths in G. Specified Path Coverage (SPC) : TR contains a set S of test paths, where S is supplied as a parameter. The logical extension is to require all paths … Unfortunately, this is impossible if the graph has a loop, so a weak compromise is to make the tester decide which paths:

40. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 40 Structural Coverage Example Node Coverage TR = { 0, 1, 2, 3, 4, 5, 6 } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] 602134 Edge Coverage TR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ] Edge-Pair Coverage TR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6], [2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 5, 4, 6 ] Complete Path Coverage Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 5, 4, 6 ] … 5

41. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 41 Loops in Graphs n If a graph contains a loop, it has an infinite number of paths n Thus, CPC is not feasible n SPC is not satisfactory because the results are subjective and vary with the tester n Attempts to “deal with” loops: –1970s : Execute cycles once ([4, 5, 4] in previous example, informal) –1980s : Execute each loop, exactly once (formalized) –1990s : Execute loops 0 times, once, more than once (informal description) –2000s : Prime paths

42. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 42 Simple Paths and Prime Paths n Simple Path : A path from node ni to nj is simple if no node appears more than once, except possibly the first and last nodes are the same –No internal loops –Includes all other subpaths –A loop is a simple path n Prime Path : A simple path that does not appear as a proper subpath of any other simple path Simple Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ], [ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ], [ 2, 3, 0, 1 ], [ 0, 1, 3 ], [ 0, 2, 3 ], [ 1, 3, 0 ], [ 2, 3, 0 ], [ 3, 0, 1 ], [3, 0, 2 ], [ 0, 1], [ 0, 2 ], [ 1, 3 ], [ 2, 3 ], [ 3, 0 ], [0], [1], [2], [3] Prime Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ], [ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ], [ 2, 3, 0, 1 ] 12 0 3

43. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 43 Prime Path Coverage n A simple, elegant and finite criterion that requires loops to be executed as well as skipped Prime Path Coverage (PPC) : TR contains each prime path in G. Will tour all paths of length 0, 1, … That is, it subsumes node, edge, and edge-pair coverage

44. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 44 Touring, Sidetrips and Detours n Prime paths do not have internal loops … test paths might Tour : A test path p tours subpath q if q is a subpath of p Tour With Sidetrips : A test path p tours subpath q with sidetrips iff every edge in q is also in p in the same order The tour can include a sidetrip, as long as it comes back to the same node Tour With Detours : A test path p tours subpath q with detours iff every node in q is also in p in the same order The tour can include a detour from node ni, as long as it comes back to the prime path at a successor of ni

45. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 45 Sidetrips and Detours Example 0 2 1 5 3 40 2 1 5 3 4 Touring with a sidetrip 0 2 1 5 3 4 Touring with a detour 1256 34 125 3 4 1234 Touring without sidetrips or detours p = [0, 1, 2, 4, 5 ] q = [1, 2, 4 ]

46. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 46 Infeasible Test Requirements n An infeasible test requirement cannot be satisfied –Unreachable statement (dead code) –A subpath that can only be executed if a contradiction occurs (X > 0 and X < 0) Practical recommendation – Best Effort Touring – Satisfy as many test requirements as possible without sidetrips – Allow sidetrips to try to satisfy unsatisfied test requirements Most test criteria have some infeasible test requirements It is usually undecidable whether all test requirements are feasible When sidetrips are not allowed, many structural criteria have more infeasible test requirements However, always allowing sidetrips weakens the test criteria

47. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 47 Simple & Prime Path Example 5 0 2 1 34 6 Len 0 [0] [1] [2] [3] [4] [5] [6] ! ‘!’ means path terminates Len 1 [0, 1] [0, 2] [1, 2] [2, 3] [2, 4] [3, 6] ! [4, 6] ! [4, 5] [5, 4] Len 2 [0, 1, 2] [0, 2, 3] [0, 2, 4] [1, 2, 3] [1, 2, 4] [2, 3, 6] ! [2, 4, 6] ! [2, 4, 5] ! [4, 5, 4] * [5, 4, 6] ! [5, 4, 5] * ‘*’ means path cycles Len 3 [0, 1, 2, 3] [0, 1, 2, 4] [0, 2, 3, 6] ! [0, 2, 4, 6] ! [0, 2, 4, 5] ! [1, 2, 3, 6] ! [1, 2, 4, 5] ! [1, 2, 4, 6] ! Len 4 [0, 1, 2, 3, 6] ! [0, 1, 2, 4, 6] ! [0, 1, 2, 4, 5] ! Prime Paths Simple paths

48. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 48 Data Flow Criteria n Definition (def) : A location where a value for a variable is stored into memory n Use : A location where a variable’s value is accessed n def (n) or def (e) : The set of variables that are defined by node n or edge e n use (n) or use (e) : The set of variables that are used by node n or edge e Goal: Try to ensure that values are computed and used correctly 0 2 1 63 5 4 X = 42 Z = X-8 Z = X*2 Defs: def (0) = {X} def (4) = {Z} def (5) = {Z} Uses: use (4) = {X} use (5) = {X}

49. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 49 DU Pairs and DU Paths n DU pair : A pair of locations (l i, l j ) such that a variable v is defined at l i and used at l j n Def-clear : A path from l i to l j is def-clear with respect to variable v if v is not given another value on any of the nodes or edges in the path n du-path : A simple subpath that is def-clear with respect to v from a def of v to a use of v n du (n i, n j, v) – the set of du-paths from n i to n j n du (n i, v) – the set of du-paths that start at n i

50. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 50 Data Flow Test Criteria All-defs coverage (ADC) : For each set of du-paths S = du (n, v), TR contains at least one path d in S. All-uses coverage (AUC) : For each set of du-paths to uses S = du (n i, n j, v), TR contains at least one path d in S. All-du-paths coverage (ADUPC) : For each set S = du (ni, nj, v), TR contains every path d in S. Then we make sure that every def reaches all possible uses Finally, we cover all the paths between defs and uses First, we make sure every def reaches a use

51. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 51 Data Flow Testing Example 0 2 1 63 5 4 X = 42 Z = X-8 Z = X*2 All-defs for X [ 0, 1, 3, 4 ] All-uses for X [ 0, 1, 3, 4 ] [ 0, 1, 3, 5 ] All-du-paths for X [ 0, 1, 3, 4 ] [ 0, 2, 3, 4 ] [ 0, 1, 3, 5 ] [ 0, 2, 3, 5 ]

52. Introduction to Software Testing Chapter 2.3 Graph Coverage for Source Code Paul Ammann & Jeff Offutt www.introsoftwaretesting.com

53. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 53 Overview n The most common application of graph criteria is to program source n Graph : Usually the control flow graph (CFG) n Node coverage : Execute every statement n Edge coverage : Execute every branch n Loops : Looping structures such as for loops, while loops, etc. n Data flow coverage : Augment the CFG –defs are statements that assign values to variables –uses are statements that use variables

54. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 54 Control Flow Graphs n A CFG models all executions of a method by describing control structures n Nodes : Statements or sequences of statements (basic blocks) n Edges : Transfers of control n Basic Block : A sequence of statements such that if the first statement is executed, all statements will be (no branches) n CFGs are sometimes annotated with extra information –branch predicates –defs –uses n Rules for translating statements into graphs …

55. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 55 CFG : The if Statement if (x < y) { y = 0; x = x + 1; } else { x = y; } 4 1 23 x >= yx < y x = y y = 0 x = x + 1 if (x < y) { y = 0; x = x + 1; } 3 1 2 x >= y x < y y = 0 x = x + 1

56. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 56 CFG : The if-Return Statement if (x < y) { return; } print (x); return; 3 1 2 x >= y x < y return print (x) return No edge from node 2 to 3. The return nodes must be distinct.

57. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 57 Loops n Loops require “extra” nodes to be added n Nodes that do not represent statements or basic blocks

58. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 58 CFG : while and for Loops x = 0; while (x < y) { y = f (x, y); x = x + 1; } 1 x = 0 43 y =f(x,y) x = x + 1 x >= yx < y for (x = 0; x < y; x++) { y = f (x, y); } 1 x = x + 1 2 35 x >= yx < y y = f (x, y) 4 2 dummy node x = 0 implicitly initializes loop implicitly increments loop

59. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 59 Example Control Flow – Stats public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } med = numbers [ length / 2 ]; mean = sum / (double) length; varsum = 0; for (int i = 0; i < length; i++) { varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println ("length: " + length); System.out.println ("mean: " + mean); System.out.println ("median: " + med); System.out.println ("variance: " + var); System.out.println ("standard deviation: " + sd); }

60. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 60 Control Flow Graph for Stats public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } med = numbers [ length / 2 ]; mean = sum / (double) length; varsum = 0; for (int i = 0; i < length; i++) { varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println ("length: " + length); System.out.println ("mean: " + mean); System.out.println ("median: " + med); System.out.println ("variance: " + var); System.out.println ("standard deviation: " + sd); } i = 0 i >= length i < length i++ i >= length i < length i = 0 i++ 1 2 3 5 4 6 8 7

61. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 61 Control Flow TRs and Test Paths – EC 1 2 3 5 4 6 8 7 TR A. [ 1, 2 ] B. [ 2, 3 ] C. [ 3, 4 ] D. [ 3, 5 ] E. [ 4, 3 ] F. [ 5, 6 ] G. [ 6, 7 ] H. [ 6, 8 ] I. [ 7, 6 ] Test Path [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ] Edge Coverage

62. iA, B, D, E, F, G, I, JC, H Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 62 Control Flow TRs and Test Paths – EPC 1 2 3 5 4 6 8 7 TR A. [ 1, 2, 3 ] B. [ 2, 3, 4 ] C. [ 2, 3, 5 ] D. [ 3, 4, 3 ] E. [ 3, 5, 6 ] F. [ 4, 3, 5 ] G. [ 5, 6, 7 ] H. [ 5, 6, 8 ] I. [ 6, 7, 6 ] J. [ 7, 6, 8 ] K. [ 4, 3, 4 ] L. [ 7, 6, 7 ] Test Paths i. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ] ii. [ 1, 2, 3, 5, 6, 8 ] iii. [ 1, 2, 3, 4, 3, 4, 3, 5, 6, 7, 6, 7, 6, 8 ] Edge-Pair Coverage TPTRs touredsidetripsii A, C, E, H iii A, B, D, E, F, G, I, J, K, L C, H

63. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 63 Control Flow TRs and Test Paths – PPC 1 2 3 5 4 6 8 7 TR A. [ 3, 4, 3 ] B. [ 4, 3, 4 ] C. [ 7, 6, 7 ] D. [ 7, 6, 8 ] E. [ 6, 7, 6 ] F. [ 1, 2, 3, 4 ] G. [ 4, 3, 5, 6, 7 ] H. [ 4, 3, 5, 6, 8 ] I. [ 1, 2, 3, 5, 6, 7 ] J. [ 1, 2, 3, 5, 6, 8 ] Test Paths i. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ] ii. [ 1, 2, 3, 4, 3, 4, 3, 5, 6, 7, 6, 7, 6, 8 ] iii. [ 1, 2, 3, 4, 3, 5, 6, 8 ] iv. [ 1, 2, 3, 5, 6, 7, 6, 8 ] v. [ 1, 2, 3, 5, 6, 8 ] Prime Path Coverage iA, D, E, F, GH, I, J TPTRs toured sidetripsii A, B, C, D, E, F, G,H, I, J iii A, F, H J iv D, E, F, I J v J

64. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 64 Data Flow Coverage for Source n def : a location where a value is stored into memory –x appears on the left side of an assignment ( x = 44 ;) –x is an actual parameter in a call and the method changes its value –x is a formal parameter of a method (implicit def when method starts) –x is an input to a program n use : a location where variable’s value is accessed –x appears on the right side of an assignment –x appears in a conditional test –x is an actual parameter to a method –x is an output of the program –x is an output of a method in a return statement n If a def and a use appear on the same node, then it is only a DU- pair if the def occurs after the use and the node is in a loop

65. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 65 Example Data Flow – Stats public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } mean = sum / (double) length; med = numbers [ length / 2 ]; varsum = 0; for (int i = 0; i < length; i++) { varsum = varsum + ((numbers [ i ] - mean) * (numbers [ i ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println ("length: " + length); System.out.println ("mean: " + mean); System.out.println ("median: " + med); System.out.println ("variance: " + var); System.out.println ("standard deviation: " + sd); }

66. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 66 8 1 2 4 3 5 6 7 Control Flow Graph for Stats ( numbers ) sum = 0 length = numbers.length i = 0 i >= length i < length sum += numbers [ i ] i++ mean = sum / (double) length; med = numbers [ length / 2 ] varsum = 0 i = 0 i >= length i < length varsum = … i++ var = varsum / ( length - 1.0 ) sd = Math.sqrt ( var ) print (length, mean, med, var, sd)

67. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 67 8 1 2 4 3 5 6 7 CFG for Stats – With Defs & Uses def (1) = { numbers, sum, length } def (2) = { i } def (5) = { mean, med, varsum, i } use (5) = { numbers, length, sum } def (8) = { var, sd } use (8) = { varsum, length, mean, med, var, sd } use (3, 5) = { i, length } use (3, 4) = { i, length } def (4) = { sum, i } use (4) = { sum, numbers, i } use (6, 8) = { i, length } use (6, 7) = { i, length } def (7) = { varsum, i } use (7) = { varsum, numbers, i, mean }

68. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 68 Defs and Uses Tables for Stats NodeDefUse 1{ numbers, sum, length } 2{ i } 3 4{ sum, i }{ numbers, i, sum } 5{ mean, med, varsum, i } { numbers, length, sum } 6 7{ varsum, i }{ varsum, numbers, i, mean } 8{ var, sd }{ varsum, length, var, mean, med, var, sd } EdgeUse (1, 2) (2, 3) (3, 4){ i, length } (4, 3) (3, 5){ i, length } (5, 6) (6, 7){ i, length } (7, 6) (6, 8){ i, length }

69. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 69 DU Pairs for Stats variableDU Pairs numbers(1, 4) (1, 5) (1, 7) length(1, 5) (1, 8) (1, (3,4)) (1, (3,5)) (1, (6,7)) (1, (6,8)) med(5, 8) var(8, 8) sd(8, 8) mean(5, 7) (5, 8) sum(1, 4) (1, 5) (4, 4) (4, 5) varsum(5, 7) (5, 8) (7, 7) (7, 8) i(2, 4) (2, (3,4)) (2, (3,5)) (2, 7) (2, (6,7)) (2, (6,8)) (4, 4) (4, (3,4)) (4, (3,5)) (4, 7) (4, (6,7)) (4, (6,8)) (5, 7) (5, (6,7)) (5, (6,8)) (7, 7) (7, (6,7)) (7, (6,8)) No def-clear path … different scope for i defs come before uses, do not count as DU pairs defs after use in loop, these are valid DU pairs

70. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 70 DU Paths for Stats – No Duplicates There are 38 DU paths for Stats, but only 12 unique [ 1, 2, 3, 4 ] [ 1, 2, 3, 5 ] [ 1, 2, 3, 5, 6, 7 ] [ 1, 2, 3, 5, 6, 8 ] [ 2, 3, 4 ] [ 2, 3, 5 ] [ 4, 3, 4 ] [ 4, 3, 5 ] [ 5, 6, 7 ] [ 5, 6, 8 ] [ 7, 6, 7 ] [ 7, 6, 8 ] 5 expect a loop not to be “entered” 5 require at least one iteration of a loop 2 require at least two iteration of a loop

71. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 71 Test Cases and Test Paths Test Case : numbers = (44) ; length = 1 Test Path : [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ] Additional DU Paths covered (no sidetrips) [ 1, 2, 3, 4 ] [ 2, 3, 4 ] [ 4, 3, 5 ] [ 5, 6, 7 ] [ 7, 6, 8 ] The five stars that require at least one iteration of a loop Test Case : numbers = (2, 10, 15) ; length = 3 Test Path : [ 1, 2, 3, 4, 3, 4, 3, 4, 3, 5, 6, 7, 6, 7, 6, 7, 6, 8 ] DU Paths covered (no sidetrips) [ 4, 3, 4 ] [ 7, 6, 7 ] The two stars that require at least two iterations of a loop Other DU paths require arrays with length 0 to skip loops But the method fails with divide by zero on the statement … mean = sum / (double) length; A fault was found

72. Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 72 Graph Coverage Criteria Subsumption Node Coverage NC Edge Coverage EC Edge-Pair Coverage EPC Prime Path Coverage PPC Complete Path Coverage CPC All-DU-Paths Coverage ADUP All-uses Coverage AUC All-defs Coverage ADC

73. Introduction to Software Testing Chapter 2.4 Graph Coverage for Design Elements Paul Ammann & Jeff Offutt www.introsoftwaretesting.com

74. © Ammann & Offutt 74 Structural Graph Coverage for DE n Call Graph: The most common graph for structural design testing n Nodes : Units (in Java – methods) n Edges : Calls to units Example call graph A BCD FE Node coverage : call every unit at least once (method coverage) Edge coverage : execute every call at least once (call coverage) Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com

75. © Ammann & Offutt 75 Data Flow at the Design Level n Data flow couplings among units and classes are more complicated than control flow couplings –When values are passed, they “change names” –Many different ways to share data –Finding defs and uses can be difficult – finding which uses a def can reach is very difficult (in unit test defs and uses were in the same unit) Caller : A unit that invokes another unit Callee : The unit that is called Callsite : Statement or node where the call appears Actual parameter : Variable in the caller Formal parameter : Variable in the callee Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com

76. © Ammann & Offutt 76 Example Call Site A  B (x)  end A B (Y)  end B Caller Actual Parameter Formal Parameter Callee interface n Applying data flow criteria to def-use pairs between units is too expensive n Too many possibilities n But this is integration testing, and we really only care about the interface … Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com

77. © Ammann & Offutt 77 Inter-procedural DU Pairs n If we focus on the interface, then we just need to consider the last definitions of variables before calls and returns and first uses inside units and after calls n Last-def : The set of nodes that define a variable x and has a def- clear path from the node through a callsite to a use in the other unit –Can be from caller to callee (parameter or shared variable) or from callee to caller as a return value n First-use : The set of nodes that have uses of a variable y and for which there is a def-clear and use-clear path from the callsite to the nodes Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com

78. Coupling DU-path Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 78

79. © Ammann & Offutt 79 Example Inter-procedural DU Pairs FX = 14  y = G (x)  print (y) G (a) print (a)  b = 42  return (b) Caller Callee DU pair 11 B (int y) Z = yT = y print (y) 10 12 13 Last Defs 2, 3 First Uses 11, 12 callsite first-use last-def x = 5 x = 4 x = 3 B (x) 1 2 3 4 Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com

80. © Ammann & Offutt 80 1 // Program to compute the quadratic root for two numbers 2 import java.lang.Math; 3 4 class Quadratic 5 { 6 private static float Root1, Root2; 7 8 public static void main (String[] argv) 9 { 10 int X, Y, Z; 11 boolean ok; 12 int controlFlag = Integer.parseInt (argv[0]); 13 if (controlFlag == 1) 14 { 15 X = Integer.parseInt (argv[1]); 16 Y = Integer.parseInt (argv[2]); 17 Z = Integer.parseInt (argv[3]); 18 } 19 else 20 { 21 X = 10; 22 Y = 9; 23 Z = 12; 24 } 25 ok = Root (X, Y, Z); 26 if (ok) 27 System.out.println 28 (“Quadratic: ” + Root1 + Root2); 29 else 30 System.out.println (“No Solution.”); 31 } 32 33 // Three positive integers, finds quadratic root 34 private static boolean Root (int A, int B, int C) 35 { 36 float D; 37 boolean Result; 38 D = (float) Math.pow ((double)B, (double2-4.0)*A*C ); 39 if (D < 0.0) 40 { 41 Result = false; 42 return (Result); 43 } 44 Root1 = ( float ) ((-B + Math.sqr t(D))/(2.0*A)); 45 Root2 = ( float ) ((-B – Math.sqrt (D))/(2.0*A)); 46 Result = true; 47 return (Result); 48 } / /End method Root 49 50 } // End class Quadratic Example – Quadratic Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com

81. © Ammann & Offutt 81 1 // Program to compute the quadratic root for two numbers 2 import java.lang.Math; 3 4 class Quadratic 5 { 6 private static float Root1, Root2; 7 8 public static void main (String[] argv) 9 { 10 int X, Y, Z; 11 boolean ok; 12 int controlFlag = Integer.parseInt (argv[0]); 13 if (controlFlag == 1) 14 { 15 X = Integer.parseInt (argv[1]); 16 Y = Integer.parseInt (argv[2]); 17 Z = Integer.parseInt (argv[3]); 18 } 19 else 20 { 21 X = 10; 22 Y = 9; 23 Z = 12; 24 } last-defs shared variables Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com

82. © Ammann & Offutt 82 25 ok = Root (X, Y, Z); 26 if (ok) 27 System.out.println 28 (“Quadratic: ” + Root1 + Root2); 29 else 30 System.out.println (“No Solution.”); 31 } 32 33 // Three positive integers, finds the quadratic root 34 private static boolean Root (int A, int B, int C) 35 { 36 float D; 37 boolean Result; 38 D = (float) Math.pow ((double)B, (double2-4.0)*A*C); 39 if (D < 0.0) 40 { 41 Result = false; 42 return (Result); 43 } 44 Root1 = (float) ((-B + Math.sqrt(D)) / (2.0*A)); 45 Root2 = (float) ((-B – Math.sqrt(D)) / (2.0*A)); 46 Result = true; 47 return (Result); 48 } / /End method Root 49 50 } // End class Quadratic last-deffirst-use last-defs Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com

83. © Ammann & Offutt 83 Quadratic – Coupling DU-pairs Pairs of locations: method name, variable name, statement (main (), X, 15) – (Root (), A, 38) (main (), Y, 16) – (Root (), B, 38) (main (), Z, 17) – (Root (), C, 38) (main (), X, 21) – (Root (), A, 38) (main (), Y, 22) – (Root (), B, 38) (main (), Z, 23) – (Root (), C, 38) (Root (), Root1, 44) – (main (), Root1, 28) (Root (), Root2, 45) – (main (), Root2, 28) (Root (), Result, 41) – ( main (), ok, 26 ) (Root (), Result, 46) – ( main (), ok, 26 ) Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com

84. Introduction to Software Testing Chapter 5.1 Syntax-based Testing Paul Ammann & Jeff Offutt www.introsoftwaretesting.com

85. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 85 Mutation testing n The original and most widely known application of syntax-based testing is to modify programs n Operators modify a ground string (program under test) to create mutant programs n Mutant programs must compile correctly (valid strings) n Mutants are not tests, but used to find tests n Once mutants are defined, tests must be found to cause mutants to fail when executed n This is called “killing mutants”

86. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 86 Killing Mutants n If mutation operators are designed well, the resulting tests will be very powerful n Different operators must be defined for different programming languages and goals n Testers can keep adding tests until all mutants have been killed –Dead mutant : A test case has killed it –Stillborn mutant : Syntactically illegal –Trivial mutant : Almost every test can kill it –Equivalent mutant : No test can kill it (equivalent to original program) Given a mutant m  M for a ground string program P and a test t, t is said to kill m if and only if the output of t on P is different from the output of t on m.

87. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 87 Program-based Grammars Original Method int Min (int A, int B) { int minVal; minVal = A; if (B < A) { minVal = B; } return (minVal); } // end Min With Embedded Mutants int Min (int A, int B) { int minVal; minVal = A; ∆ 1 minVal = B; if (B < A) ∆ 2 if (B > A) ∆ 3 if (B < minVal) { minVal = B; ∆ 4 Bomb (); ∆ 5 minVal = A; ∆ 6 minVal = failOnZero (B); } return (minVal); } // end Min 6 mutants Each represents a separate program Replace one variable with another Changes operator Immediate runtime failure … if reached Immediate runtime failure if B==0 else does nothing

88. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 88 Syntax-Based Coverage Criteria Mutation Coverage (MC) : For each m M, TR contains exactly one requirement, to kill m. Mutation Coverage (MC) : For each m  M, TR contains exactly one requirement, to kill m. The RIP model from chapter 1: Reachability : The test causes the faulty statement to be reached (in mutation – the mutated statement) Infection : The test causes the faulty statement to result in an incorrect state Propagation : The incorrect state propagates to incorrect output The RIP model leads to two variants of mutation coverage …

89. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 89 Syntax-Based Coverage Criteria 1) Strongly Killing Mutants: Given a mutant m  M for a program P and a test t, t is said to strongly kill m if and only if the output of t on P is different from the output of t on m 2) Weakly Killing Mutants: Given a mutant m  M that modifies a location l in a program P, and a test t, t is said to weakly kill m if and only if the state of the execution of P on t is different from the state of the execution of m immediately on t after l Weakly killing satisfies reachability and infection, but not propagation

90. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 90 Weak Mutation Weak Mutation Coverage (WMC) : For each m M, TR contains exactly one requirement, to weakly kill m. Weak Mutation Coverage (WMC) : For each m  M, TR contains exactly one requirement, to weakly kill m. Some mutants can be killed under weak mutation but not under strong mutation (no propagation)

91. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 91 Mutation Example The complete test specification to kill mutant 1: Reachability : true // Always get to that statement Infection : A ≠ B Propagation: (B < A) = false // Skip the next assignment Full Test Specification : true  (A ≠ B)  ((B < A) = false) ≡ (A ≠ B)  (B ≥ A) ≡ (B > A) (A = 5, B = 7) will kill mutant 1. minVal = A; ∆ 1 minVal = B; if (B < A) minVal = B; n Mutant 1 in the Min( ) example is:

92. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 92 Equivalent Mutation Example n Mutant 3 in the Min() example is equivalent: minVal = A; if (B < A) ∆ 3 if (B < minVal) The infection condition is “(B < A) != (B < minVal)” However, the previous statement was “minVal = A” –Substituing, we get: “(B < A) != (B < A)” Thus no input can kill this mutant

93. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 93 1 boolean isEven (int X) 2 { 3 if (X < 0) 4 X = 0 - X; ∆ 4 X = 0; 5 if (float) (X/2) == ((float) X) / 2.0 6 return (true); 7 else 8 return (false); 9 } Strong Versus Weak Mutation Reachability : X < 0 Infection : X != 0 (X = -6) will kill mutant 4 under weak mutation Propagation : ((float) ((0-X)/2) == ((float) 0-X) / 2.0) != ((float) (0/2) == ((float) 0) / 2.0) That is, X is not even … Thus (X = -6) does not kill the mutant under strong mutation

94. Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com © Ammann & Offutt 94 Designing Mutation Operators n At the method level, mutation operators for different programming languages are similar n Mutation operators do one of two things: –Mimic typical programmer mistakes ( incorrect variable name ) –Encourage common test heuristics ( cause expressions to be 0 ) n Researchers design lots of operators, then experimentally select the most useful Effective Mutation Operators If tests that are created specifically to kill mutants created by a collection of mutation operators O = {o1, o2, …} also kill mutants created by all remaining mutation operators with very high probability, then O defines an effective set of mutation operators

95. Software security Michael Hicks University of Maryland Online course (www.coursera.org)www.coursera.org

96. What is computer security? n Most developers and operators are concerned with correctness: achieving desired behavior –A working banking web site, word processor, blog, … n Security is concerned with preventing undesired behavior –Considers an enemy/opponent/hacker/adversary who is actively and maliciously trying to circumvent any protective measures you put in place –Kinds of undesired behavior: Stealing information (confidentiality) Modifying information or functionality (integrity) Denying access (availability) 96

97. Defects and Vulnerabilities n Many breaches begin by exploiting a vulnerability –This is a security-relevant software defect that can be exploited to effect an undesired behavior n A software defect is present when the software behaves incorrectly, i.e., it fails to meet its requirements n Defects occur in the software’s design and its implementation –A flaw is a defect in the design –A bug is a defect in the implementation 97

98. Considering Correctness n All software is buggy. So what? n A normal user never sees most bugs, or works around them –Most (post-deployment) bugs due to rare feature interactions or failure to handle edge cases n Assessment: Would be too expensive to fix every bug before deploying –So companies only fix the ones most likely to affect normal users 98

99. Considering Security n Key difference: An adversary is not a normal user n The adversary will actively attempt to find defects in rare feature interactions and edge cases –For a typical user, (accidentally) finding a bug will result in a crash, which he will now try to avoid –An adversary will work to find a bug and exploit it to achieve his goals 99

100. 100 To ensure security, we must eliminate bugs and design flaws, and/or make them harder to exploit