COVERAGE CRITERIA FOR TESTING 1

ill-defined terms in Testing  complete testing  exhaustive testing  full coverage Are poorly defined terms because of theoretical limitation of software.  The number of potential inputs for most programs is so large as to be effectively infinite.  The only limitation is the size of the file that can be read by the parser.  the number of inputs is effectively infinite 2

Formal Coverage Criteria  We cannot test with all input  Coverage criteria are used to decide which test inputs to use.  Effective use of coverage criteria makes it more likely  Test engineers will find faults in a program This provides informal assurance Result : High quality and Reliability of the Software 3

Advantage of Coverage Criteria According to the practical perspective Coverage criteria provide rules for when to stop testing. 4

5 Traditional Testing Levels Class A method mA1() method mA2() Class B method mB1() method mB2() main Class P n Acceptance testing : Is the software acceptable to the user? n Integration testing : Test how modules interact with each other n System testing : Test the overall functionality of the system n Module testing (developer testing) : Test each class, file, module or component n Unit testing (developer testing) : Test each unit (method) individually This view obscures underlying similarities

6 Object-Oriented Testing Levels Class A method mA1() method mA2() Class B method mB1() method mB2() n Intra-class testing : Test an entire class as sequences of calls n Inter-class testing : Test multiple classes together n Inter-method testing : Test pairs of methods in the same class n Intra-method testing : Test each method individually

Coverage Criteria Even small programs have too many inputs to fully test them all – private static computeAverage (int A, int B, int C) – On a 32-bit machine, each variable has over 4 billion possible values – Over 80 octillion possible tests!! – Input space might as well be infinite Testers search a huge input space – Trying to find the fewest inputs that will find the most problems Coverage criteria give structured, practical ways to search the input space – Search the input space thoroughly – Not much overlap in the tests 7

Advantages of Coverage Criteria Provide traceability from software artifacts to tests – Source, requirements, design models, … Make regression testing easier Gives testers a “stopping rule” … when testing is finished Can be well supported with powerful tools 8

Test Requirements and Criteria  Test Criterion : A collection of rules and a process that define test requirements ̶Cover every statement ̶Cover every functional requirement  Test Requirements : Specific things that must be satisfied or covered during testing – Each statement is a test requirement – Each functional requirement is a test requirement 9 Testing researchers have defined dozens of criteria, but they are all really just a few criteria on four types of structures … 1.Graphs 2.Logic expressions 3.Input domains 4.Syntax descriptions

Graph Coverage Criteria Structural Coverage Criteria : Defined on a graph just in terms of nodes and edges Data Flow Coverage Criteria : Requires a graph to be annotated with references to variables 10

Coverage Criterion A coverage criterion is simply a recipe for generating test requirements in a systematic way: Coverage Criterion: A coverage criterion is a rule or collection of rules that impose test requirements on a test set. 11

Coverage Criterion A coverage criterion is a rule or collection of rules that impose test requirements on a test set. That is, the criterion describes the test requirements in a complete and unambiguous manner. The “flavor criterion” yields a simple strategy for selecting jelly beans. In this case, the set of test requirements, TR, can be formally written out as TR = {flavor = Lemon, flavor = Pistachio, flavor = Cantaloupe, flavor = Pear, flavor = Tangerine, flavor = Apricot} 12

Coverage Given a set of test requirements TRfor a coveragecriterion C, a test set T satisfies C if and only if for every test requirement tr in TR, at least one test t in T exists such that t satisfies tr. 13

Coverage Level Given a set of test requirements TR and a test set T, the coverage level is simply the ratio of the number of test requirements satisfied by T to the size of TR. 14

Graph Coverage Four Structures for Modeling Software Graphs LogicInput SpaceSyntax Use cases Specs Design Source Applied to DNF Specs FSMs Source Applied to Input Models Integ Source Applied to 15

Node Coverage (Formal Definition): For each node n ∈ reach G (N 0 ), TR contains the predicate “visit n.” Node Coverage (NC): TR contains each reachable node in G. Structural Coverage Criteria -I Node Coverage (NC) (Standard Definition) Test set T satisfies node coverage on graph G if and only if for every syntactically reachable node n in N, there is some path p in path(T) such that p visits n. TR = { 0, 1, 2 } path(t 1 ) = [0, 1, 2] T 1 ={t 1 } T 1 satisfies node coverage on the graph

Structural Coverage Criteria - II Edge coverage is slightly stronger than node coverage  Edge Coverage (EC) : TR contains each reachable path of length up to 1, inclusive, in G.  The “length up to 1” allows for graphs with one node and no edges Edge Coverage : TR = { (0,1), (0, 2), (1, 2) } path (t 1 ) = [ 0,1,2] path (t 2 ) = [ 0, 2 ] T 2 ={t 1, t 2 } Test Paths = [ 0, 1, 2 ] [ 0, 2 ] T 2 satisfies edge coverage on the graph

Paths of Length 1 and A graph with only one node will not have any edges It may be boring, but formally, Edge Coverage needs to require Node Coverage on this graph Edge Coverage needs to require Node Coverage on this graph Otherwise, Edge Coverage will not subsume Node Coverage So we define “length up to 1” instead of simply “length 1” We have the same issue with graphs that only have one edge for Edge Pair Coverage … 1 0

Structural Coverage Example Node Coverage TR = { 0, 1, 2, 3, 4, 5, 6 } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] Edge Coverage TR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ] path (t 1 )= [ 0, 1, 2, 3, 6 ] path (t 2 )=[ 0, 2, 4, 5, 4, 6 ] T={t 1, t 2 }