Mahadevan Subramaniam and Bo Guo University of Nebraska at Omaha An Approach for Selecting Tests with Provable Guarantees

Multi-version QA 11/30/ UNO Requirements Development Team Testing Team codes bugs R1 No bug claimed Testing is very costly and time consuming [NIST] 1) in critical path for release 2) development deadline slips can reduce test time 3) development team needs time to fix bugs Changes Development Team Testing Team Modified codes bugs R2 No bug claimed Two or Three weeks

Multi-version QA Scope 11/30/2012 Predictive Changes: new tests to validate new features Corrective Changes: Regression testing of existing features Problems Regression Test Minimization – remove redundant tests Regression Test Prioritization – fault-finding tests first Regression Test Selection (RTS) – select relevant tests 3 UNO

Regression Test Selection -- Motivation 11/30/2012 Regression testing is most frequent [NIST] and has significant impact on product quality. System behavior re-validated in practice using large test suites. Re-running the entire test suite each time is impractical. Regression test selection identifies tests from a test suite to validate a change in an evolution step. 4 UNO

Current State of Art – Code based 11/30/ UNO

Current State of Art – Model-based 11/30/ UNO

Current State of Art 11/30/2012 Store all test traces on original version Static analysis Compare new and modified Control Flow Graphs If a trace includes the differences, select test including the trace. Use data flow graphs and program dependency graphs 7 UNO

Problems 11/30/2012 Storing all traces is expensive Control flow graphs comparing may involve untested parts of programs No guarantee on the regression test suite Relevant tests may be skipped (Incomplete) Irrelevant tests may be selected 8 UNO

Goal 11/30/2012 Develop an automatic approach for regression test selection to create a regression test suite with provable guarantees and without using any history information. Main Challenges Provably predict test execution behavior without actually executing the test ? Keep regression test selection economical Selection cost + regression test cost <= brute-force run cost. 9 UNO

Model-based Proposal 11/30/2012 Key ideas Analyze test descriptions to provably predict their behavior. Test descriptions based on EFSMs and GUI event diagrams have sufficient information in terms of input and output events. Exploit overlap among descriptions to keep regression economical [See tech report] 10 UNO

EFSM – extended finite state machine 11/30/2012 UNO 11 s0 s s Initially a = 0; state is s0 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a).  List of states The machine is in only one state at a time  Triggering condition for each transition

Overview of the Approach Analyze a given test description to determine if the test execution will exercise a given change and select the test.  Identify descriptions having sufficient information about their execution (fully-observable tests)  Analyze such descriptions to determine if the change will be executed.  Patch other descriptions to obtain information about their execution. Changes add / delete / replace transitions. Test description is a sequence of input and output messages with parameters with values over booleans, integers, arrays, queues, and records /30/ UNO

Overview of the Approach (Contd.) From a given test description and a change find sets of matching transitions that can process inputs in the description descriptions having a matching change transition are candidates determine feasibility of an execution path formed by matching transitions automatically attempt to patch infeasible paths to obtain an execution path Test selected if the change transition appears in the feasible execution path /30/ UNO

Tests λ1: open(100)/ack(100), deposit(50)/ack(150), close/ack(150) λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) A Simple Example 11/30/2012 Initially a = 0; state is s0 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). Not Selected 14 UNO s0 s s

λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) A Simple Example – λ 2 11/30/2012 Find transitions matching each input of description λ2 { 1 }{ 2 }{ 3, 7 }{ 6 } Φ = [] 15 UNO 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). s0 s s

A Simple Example – λ 2 (conti…) 11/30/ UNO λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) Φ = [ {1} {2} {3, 7} {6} ] s0 s s Transition 2 can immediately follow 1 to process next input of λ2 First test input of λ2 must be processed by transition 1 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).

A Simple Example – λ 2 (conti…) 11/30/ UNO λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) Φ = [ {1} {2} {3, 7} {6} ] s0 s s Transition 7 can immediately follow 2 since (160 > 0) ˄ (160 > 150) satisfiable. Selected 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).

11/30/2012 A Simple Example – λ 3 18 UNO s0 s s λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) Φ = [ {1} {2} {3, 7} {3, 7} {6} ] First two inputs processed by transition 1 followed by transition 2. Transition 7 cannot immediately follow 2 since (100 > 0) ^ (100 > 150) is unsatisfiable. 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).

11/30/2012 A Simple Example – λ 3 (conti..) 19 UNO s0 s s λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) Φ = [ {1} {2} {3, 7} {3, 7} {6} ] But 3 can immediately follow 2 since (100 > 0) ˄ (100 <= 150) is satisfiable. 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).

11/30/2012 A Simple Example – λ 3 (conti..) 20 UNO s0 s s λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) Φ = [ {1} {2} {3, 7} {3, 7} {6} ] Neither 7 nor 3 can immediately follow transition 3. 7 can follow 3 after some transitions not using test inputs. ? Can we patch λ3 to obtain a path with transition 7? 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).

11/30/2012 A Simple Example – λ 3 (conti..) 21 UNO s0 s s λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) Φ = [ {1} {2} {3, 7} {3, 7} {6} ] 4 cannot patch since (50 > 0) ^ (50 < 50) is unsatisfiable. 5 can patch since (50 >= 50) is satisfiable and 7 can immediately follow 5. Selected ? 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).

Salient Aspects 11/30/2012 Regression test selection for test descriptions with rich data types. Fully-observable tests: descriptions with sufficient information. A simple structural invariant to identify fully-observable tests. Automatically analyze failure of invariant to patch test descriptions. Procedures to select tests guaranteed to exercise changes to EFSMs. Experiments with 10 web services and protocols Study costs of running the full test case and selective test cases. Proposed approach reduces test running times in all examples. 22 UNO

Outline 11/30/2012 Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Experiments Conclusion 23 UNO

EFSMs 11/30/2012 E = (I i, O i, S i, V i, T i ) I i, O i : input and output messages S i : local states V i : Data and queue variables T i : Deterministic transition relation Transition t: m j, P t, s t  q t, m l, A t m i m l : parameterized input and output messages P t : a predicate over variables from V i s t, q t : states from S i A t : ordered sequence of assignments to variables from V i Explicit transition is transition having both input and output messages 24 UNO

Test Descriptions and Changes 11/30/2012 Test description λ = g 0 is concrete initial global state Finite sequence of input/output elements i k /o k Test run r λ = g 0 t 0 …t m g m …g 0 is an EFSM run produced by applying λ to the EFSM in state g 0. All the global states in r λ are concrete global states. Changes to the EFSM Performed at the transition level δ =, sign  { +, - } (addition, deletion) δ = (replacement) 25 UNO

Outline Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Experiments Conclusion 26 11/30/ UNO

11/30/2012 Transitions Matching a Test Description Transition matches a test if an input-output element of test is an instance of input-output message of the transition and satisfies the input condition of the transition. Transition 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a) matches withdraw(160)/ack(150). Φ : sequence of sets of transitions point-wise matching the sequence of input-output elements in a test description. Φ = [ {1} {2} {3, 7} {6} ] for test λ2  : sequence of transitions selected point-wise from the sets.  = [ ] 27 UNO

11/30/2012 Fully-Observable Tests  has a disjunct for each sequence of transitions ρ in Φ. kth conjunct states that execution path with k transitions can be extended using k+1 st transition. Test is fully-observable if its invariant  is a satisfiable formula A disjunct with value true denotes execution path for a satisfiable  At most one disjunct can evaluate to true since EFSMs are deterministic 28 UNO ( n-1  = ˅ ˄ Pos([t 1,…, t k ], g 0 )  Pre(t k+1 ) ρ Φ k=0

Initially a = 0; state is s0 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a) /30/2012 Identifying Fully-Observable Tests 29 UNO 12 λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) Φ = [ {1} {2} {3, 7} {6} ] 3 7 ρ 1 : conjunct 0: (a 0 == 0) ˄ (100 > 0) ρ 1 : conjunct 1: (a 0 == 0) ˄ (100 > 0) ˄ (a 1 == a ) ˄ (50 > 0) ρ 1 : conjunct 2: (a 0 == 0) ˄ (100 > 0) ˄ (a 1 == a ) ˄ (50 > 0) ˄ (a 2 == a 1 +50) ˄ (160 > 0) ˄ (160 <= a 2 ) ρ 2 : conjunct 2: (a 0 == 0) ˄ (100 > 0) ˄ (a 1 == a ) ˄ (50 > 0) ˄ (a 2 == a 1 +50) ˄ (160 > 0) ˄ (160 > a 2 )

Outline Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Preliminary Experiments Conclusion 30 11/30/ UNO

Tests with Non-Observable Regions 31 11/30/ UNO Identify a non-observable region (t k, t k+1 ) for a conjunct that fails to extend to transition t k+1 EFSM paths from output state of t k to input state of t k+1. All intermediate transitions in all paths have no test inputs. Eliminating regions without loops Merge the intermediate transitions with t k to t merge Patch conjunct C = Pos([t 1,…, t merge ], g 0 )  Pre(t k+1 ) and check if satisfiable. Fails if patched conjuncts for all paths in region unsatisfiable.

Patching Failures – Example 32 11/30/ UNO λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) s0 s s ? open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). ? ignore

Regions with Terminating Loops 33 11/30/ UNO Regions having loops cannot be eliminated by simple merging. Each loop iteration requires additional test input. Merge intermediate transitions in the path until the loop end- points and attempt to construct a path by executing loop transitions. Process terminates since all loops are terminating over concrete global states.

7 Patching Involving Loop – Example 34 11/30/ UNO λ4: open(100)/ack(100), deposit(50)/ack(150), withdraw(50)/ack(100), withdraw(60)/ack(40), withdraw(60)/ack(50), close/ack(50) s0 s s ? 7?? ignore Patch fails ? 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a) Φ = [ {1} {2} {3, 7} {3, 7} {3, 7} {6} ]

Outline Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Experiments Conclusion 35 11/30/ UNO

Experiments Regression Cost Model [Leung and White 91]  C1: Cost of running the full test suite  C2: Cost of running the selected tests  C3: Cost of analyzing the regression test selection Atm: automatic teller machine; Thp: third party call; Bnk: back web services; Ven: a vending machine (Cmp, Tcp, Cnf) : completion, two-phase commit, and conference protocol 36 - >

Time Savings across Multiple Changes 37 11/30/2012 UNO

Test Selected across Multiple Changes 38 11/30/2012 UNO

SPG with Code-based approach 11/30/ UNO P1: incr1 (x) { p1 while(++x <= 0) { } s1 return x; } P1 E s1 X T F s0 s1 t2: while() [x+1<=0] /{x=x+1} t1: incr1(p) /{x = p} t3: jump() [x+1>0] /{x=x+1; return(x)}

SPG with Code-based approach 11/30/ UNO P1: incr1 (x) { p1 while(++x <= 0) { p2 while(++x <= 0) {} } s1 return x; } P1 E s1 X F P2 T T F x = 0 : (E, P1) (P1, s1) P1 E s1 X T F x = 0 : (E, P1) (P1, s1)

SPG with Code-based approach 11/30/ UNO P1: incr1 (x) { p1 while(++x <= 0) { p2 while(++x <= 0) {} } s1 return x; } Incr1(0)/null, jump()/return() : t1,t3 t3: jump() [x+1>0] /{x=x+1; return(x)} t1: incr1(p) /{x = p} s0 s1 t4: while(), [x+1<=0] /{x=x+1} s2 t5: while() [x+1<=0] /{x=x+1} t6: jump(), [x+1>0] /{x=x+1} s0 s1 t2: while() [x+1<=0] /{x=x+1} t1: incr1(p) /{x = p} t3: jump() [x+1>0] /{x=x+1; return(x)} Incr1(0)/null, jump()/return() : t1,t3

SPG with Code-based approach 11/30/ UNO P3: incr1 (x) { p1 while(++x <= 0) { p2 while(++x <= 0) {} s2 return x; } s1 return x; } P1 E s1 X T F P1 E s1 X F P2 T T F s2 x = -1 : (E, P1) (P1, P1) (P1, s1) (s1, X) x = -1 : (E, P1) (P1, P2) (P2, s2) (s2, X)

SPG with Code-based approach 11/30/ UNO P3: incr1 (x) { p1 while(++x <= 0) { p2 while(++x <= 0) {} s2 return x; } s1 return x; } incr1(-1)/null, while()/null, jump()/return(0) : t1, t2, t3 s0 s1 t2: while() [x+1<=0] /{x=x+1} t1: incr1(p) /{x = p} t3: jump() [x+1>0] /{x=x+1; return(x)} t3: jump() [x+1>0] /{x=x+1; return(x)} t1: incr1(p) /{x = p} s0 s1 t4: while(), [x+1<=0] /{x=x+1} s2 t5: while() [x+1<=0] /{x=x+1} t6: jump(), [x+1>0] /{x=x+1; return(x)} incr1(-1)/null, while()/null, jump()/return(0) : t1, t4, t6

Conclusions 11/30/2012 Proposed an approach for regression test selection for high- level EFSM test descriptions supporting common data types. Test descriptions are automatically analyzed to identify tests that exercise a given change. Procedures to identify fully-observable tests, patching non- observable regions, and reducing the test suites are described. Initial results of our experiments are promising. Comparison with Code-based approach. 44 UNO

11/30/2012 Pre- and Post-Images of Transitions Pre-image of t: a symbolic global state that triggers the transition t. 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a) Pre(3) = 0 ^ v <= a 0 ^ IQ 0.hd == withdraw(v) ) Post-image of t: a symbolic global state that is obtained after executing t. Pos(3) = 0 ^ v ^ a 1 == a 0 – v ) > Executable path: a transition sequence [t 1,…, t n ] from a given global state g if Pos([t 1,…t n ], g) is satisfiable. n-1 Pos([t 1,..,t n ], g) = pred ˄ ( ˄ Pos(t i ) ˄ Pre(t i+1 )) ˄ Pos(t n ) i=1 45