1. AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in

2. Publication Trend in SBSE

3. Spread of Activities in SBSE

4. Yet another tool?  IGUANA (Phil McMinn @ Univ. of Sheffield) for C  not officially available  eTOC (Paolo Tonella @ FBK) for JAVA  Other SBT tools for C language? Challenges – no reflection Pointers and the ability to cast arbitrarily between types

5. Yet Another Search Based Testing Tool? IGUANA - for C (Phil McMinn) eTOC – for JAVA (Paolo Tonella) Others?

6. Yet Another Search Based Testing Tool? IGUANA - for C (Phil McMinn) eTOC – for JAVA (Paolo Tonella) Others? AUSTIN

7. Outline AUSTIN Introduction to Austin ETF Overview of ETF Empirical Study Hypotheses, Test Subjects Results and Analysis Conclusion ETF or AUSTIN?

8. AUgmented Search based TestINg CIL AST, CFG instrumentgcc -o sut.exe run AUSTIN gcc -P -E c source files preprocessed c source files instrumented c source

9. AUgmented Search based TestINg CIL AST, CFG instrumentgcc -o sut.exe run AUSTIN gcc -P -E c source files preprocessed c source files instrumented c source

10. AUgmented Search based TestINg CIL AST, CFG instrumentgcc -o sut.exe run AUSTIN gcc -P -E c source files preprocessed c source files instrumented c source

11. AUgmented Search based TestINg CIL AST, CFG instrumentgcc -o sut.exe run AUSTIN gcc -P -E c source files preprocessed c source files instrumented c source

12. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target }

13. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 87, key2 = 33, p = 0, q = 0

14. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 88, key2 = 33, p = 0, q = 0

15. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 86, key2 = 33, p = 0, q = 0

16. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 84, key2 = 33, p = 0, q = 0

17. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 80, key2 = 33, p = 0, q = 0

18. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 80, key2 = 33, p = 0, q = 0 Fitness Function: approach_level + norm(branch_distance) Search Method: Alternating Variable Method

19. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0

20. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0

21. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0

22. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0 Solve: p ≠ 0

23. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0 Solve: p ≠ 0 p0

24. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0x..., q = 0

25. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0x..., q = 0 Symbolic Path Condition: key1 == key2 && p ≠ 0 && p ≠ q Simplified Path Condition: p ≠ 0 && p ≠ q Solve: p ≠ 0 && p == q p0 q p,q0

26. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0x..., q = 0 Symbolic Path Condition: key1 == key2 && p ≠ 0 && p ≠ q Simplified Path Condition: p ≠ 0 && p ≠ q Solve: p ≠ 0 && p == q p,q0

27. AUgmented Search based TestINg void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target } Inputs: key1 = 33, key2 = 33, p = 0x..., q = 0x...

28. Evolutionary Testing Framework (ETF) rZSlJ3hb  ETF was developed as part of the EvoTest project  Integrated into the Eclipse IDE  Supports white & black box testing  Uses Evolutionary Algorithms (through the GUIDE, EO evolutionary library interface) to generate test data ETF Selection Recombination Mutation Reinsertion GUIDE + EO Send Fitness Values Send Individuals

29. ETF cont. rZSlJ3hb Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009

30. ETF cont. rZSlJ3hb Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009

31. Fitness Function and Algorithm  Both ETF and AUSTIN use the standard fitness function for branch coverage in Search Based Testing  apporach_level + norm(branch_distance) normalization function: 1 - 1.001 -branch_distance  ETF uses a Genetic Algorithm  AUSTIN uses the Alternating Variable Method

32. Pointers in ETF and AUSTIN  AUSTIN  Construct an equivalence of memory locations based on = and ≠ operators in symbolic path condition  Initialize (pointer) input variables based on the graph Assign new memory location via malloc or use existing memory location On-demand pointer handling  ETF  Generate an index and auxiliary variable for each pointer input  Assign all input variables of the same type (including globals) to a pool Optimize indices and values of auxiliary variables along with the inputs. Indices are used to select variables from pools and assign their address to the pointer input

33. Pointers in ETF struct cell{ int value; struct cell* next; }; int globX; double globDbl; struct cell* pCell; void testme(int* p, int* q) {... } p = &globX; q = &val_p;

34. Hypothesis Null HypothesesAlternate Hypotheses AUSTIN is as effective as the ETF in achieving branch coverage. AUSTIN is more effective than the ETF in achieving branch coverage. AUSTIN is equally as efficient as the ETF in achieving branch coverage of a function. AUSTIN is more efficient than the ETF in achieving branch coverage of a function.

35. Hypothesis  H 01 : AUSTIN is as effective as the ETF in achieving branch coverage.  H A1 : AUSTIN is more effective than the ETF in achieving branch coverage.  H 02 : AUSTIN is equally as efficient as the ETF in achieving branch coverage of a function.  H A2 : AUSTIN is more efficient than the ETF in achieving branch coverage of a function.

36. Empirical Study  3 case studies (comprising 8 functions)  Embedded Software Modules from Automotive Industry Case StudyDescriptionLOC - TestedTotal BranchesNr. Inputs BAdaptive Headlight Control Software 1,236598132 CDoor-Lock Control Software 38331571 DElectric Windows Control Software 33924831

37. Coverage ETF vs AUSTIN

39. Efficiency ETF vs AUSTIN

41. Coverage Random vs AUSTIN

42. Efficiency Random vs AUSTIN

43. ETF or AUSTIN?  Coverage is about the same  AUSTIN is overall a little bit more efficient than the ETF in terms of fitness evaluations. In terms of wall clock time the difference is greater (due to the overhead of the ETF-GUIDE setup)  AUSTIN does not construct random memory graphs  ETF is not able generate inputs such as struct cell{ int mem; }; void testme(int* p, struct cell* c) { if( p == &c->mem ) //target }

44. Summary

45. http://www.cs.ucl.ac.uk/staff/K.Lakhotia/software/austin.htm