AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in

Publication Trend in SBSE

Spread of Activities in SBSE

Yet another tool?  IGUANA (Phil Univ. of Sheffield) for C  not officially available  eTOC (Paolo FBK) for JAVA  Other SBT tools for C language? Challenges – no reflection Pointers and the ability to cast arbitrarily between types

Yet Another Search Based Testing Tool? IGUANA - for C (Phil McMinn) eTOC – for JAVA (Paolo Tonella) Others?

Yet Another Search Based Testing Tool? IGUANA - for C (Phil McMinn) eTOC – for JAVA (Paolo Tonella) Others? AUSTIN

Outline AUSTIN Introduction to Austin ETF Overview of ETF Empirical Study Hypotheses, Test Subjects Results and Analysis Conclusion ETF or AUSTIN?

AUgmented Search based TestINg CIL AST, CFG instrumentgcc -o sut.exe run AUSTIN gcc -P -E c source files preprocessed c source files instrumented c source

AUgmented Search based TestINg CIL AST, CFG instrumentgcc -o sut.exe run AUSTIN gcc -P -E c source files preprocessed c source files instrumented c source

AUgmented Search based TestINg CIL AST, CFG instrumentgcc -o sut.exe run AUSTIN gcc -P -E c source files preprocessed c source files instrumented c source

AUgmented Search based TestINg CIL AST, CFG instrumentgcc -o sut.exe run AUSTIN gcc -P -E c source files preprocessed c source files instrumented c source

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target }

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 87, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 88, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 86, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 84, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 80, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 80, key2 = 33, p = 0, q = 0 Fitness Function: approach_level + norm(branch_distance) Search Method: Alternating Variable Method

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0 Solve: p ≠ 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0 Solve: p ≠ 0 p0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0x..., q = 0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0x..., q = 0 Symbolic Path Condition: key1 == key2 && p ≠ 0 && p ≠ q Simplified Path Condition: p ≠ 0 && p ≠ q Solve: p ≠ 0 && p == q p0 q p,q0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0x..., q = 0 Symbolic Path Condition: key1 == key2 && p ≠ 0 && p ≠ q Simplified Path Condition: p ≠ 0 && p ≠ q Solve: p ≠ 0 && p == q p,q0

AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0x..., q = 0x...

Evolutionary Testing Framework (ETF) rZSlJ3hb  ETF was developed as part of the EvoTest project  Integrated into the Eclipse IDE  Supports white & black box testing  Uses Evolutionary Algorithms (through the GUIDE, EO evolutionary library interface) to generate test data ETF Selection Recombination Mutation Reinsertion GUIDE + EO Send Fitness Values Send Individuals

ETF cont. rZSlJ3hb Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009

ETF cont. rZSlJ3hb Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009

Fitness Function and Algorithm  Both ETF and AUSTIN use the standard fitness function for branch coverage in Search Based Testing  apporach_level + norm(branch_distance) normalization function: branch_distance  ETF uses a Genetic Algorithm  AUSTIN uses the Alternating Variable Method

Pointers in ETF and AUSTIN  AUSTIN  Construct an equivalence of memory locations based on = and ≠ operators in symbolic path condition  Initialize (pointer) input variables based on the graph Assign new memory location via malloc or use existing memory location On-demand pointer handling  ETF  Generate an index and auxiliary variable for each pointer input  Assign all input variables of the same type (including globals) to a pool Optimize indices and values of auxiliary variables along with the inputs. Indices are used to select variables from pools and assign their address to the pointer input

Pointers in ETF struct cell{ int value; struct cell* next; }; int globX; double globDbl; struct cell* pCell; void testme(int* p, int* q) {... } p = &globX; q = &val_p;

Hypothesis Null HypothesesAlternate Hypotheses AUSTIN is as effective as the ETF in achieving branch coverage. AUSTIN is more effective than the ETF in achieving branch coverage. AUSTIN is equally as efficient as the ETF in achieving branch coverage of a function. AUSTIN is more efficient than the ETF in achieving branch coverage of a function.

Hypothesis  H 01 : AUSTIN is as effective as the ETF in achieving branch coverage.  H A1 : AUSTIN is more effective than the ETF in achieving branch coverage.  H 02 : AUSTIN is equally as efficient as the ETF in achieving branch coverage of a function.  H A2 : AUSTIN is more efficient than the ETF in achieving branch coverage of a function.

Empirical Study  3 case studies (comprising 8 functions)  Embedded Software Modules from Automotive Industry Case StudyDescriptionLOC - TestedTotal BranchesNr. Inputs BAdaptive Headlight Control Software 1, CDoor-Lock Control Software DElectric Windows Control Software

Coverage ETF vs AUSTIN

Efficiency ETF vs AUSTIN

Coverage Random vs AUSTIN

Efficiency Random vs AUSTIN

ETF or AUSTIN?  Coverage is about the same  AUSTIN is overall a little bit more efficient than the ETF in terms of fitness evaluations. In terms of wall clock time the difference is greater (due to the overhead of the ETF-GUIDE setup)  AUSTIN does not construct random memory graphs  ETF is not able generate inputs such as struct cell{ int mem; }; void testme(int* p, struct cell* c) { if( p == &c->mem ) //target }

Summary