1 Evolutionary Testing Metaheuristic search techniques applied to test problems Stella Levin Advanced Software Tools Seminar Tel-Aviv University

2 Contents 1. Introduction 2. Metaheuristic Search Techniques 3. White-Box Testing 4. Black-Box Testing 5. Object-Oriented Testing 6. Non-Functional Testing 7. Search Based Software Engineering

3 Introduction - Why testing? “The biggest part of software cost is the cost of bugs: the cost of detecting them, the cost of correcting them, the cost of designing tests and the cost of running those tests” - Beizer “In embedded systems errors could result in high risk,endanger human life, big cost” - Wegener

4 Successful EA applications NASA evolvable antenna Equal to 12 years working of experienced designer There is no guarantee that a design would be as good [8]

5 1. Metaheuristic Search Problem characteristics Large solution space No precise algorithm, no “best” solution Classification of “better” solution “Due to non-linearity of software (if, loops…) test problems are converted to complex, discontinuous, non-linear search spaces”- Baresel

6 Transform a problem to optimization problem Candidate solution representation – individual Fitness function for individual Movement from one individual to another

7 Hill Climbing - “local” search 1. Select a point in the search space 2. Investigate neighboring points 3. If there is a better neighbor solution (with fitness function), jump to it 4. Repeat steps 2-3 until current position has no better neighbors Require definition of neighboring points

8 Hill Climbing - Problem Is there a bigger hill?

9 Simulated Annealing Analogy of the chemical process of cooling of a material in a heat bath If F(X2)<F(X1) then move to neighbor X2 Else move to X2 with probability P=e^(-ΔF/T) Initially T is high; T decreases more like hill climbing Require definition of neighbor and cooling function

10 Simulated Annealing

11 Evolutionary Algorithms Genetic algorithms Developed by J. Holland in the 70s Evolution strategies Developed in Germany at about the same time Analogy with Darwin’s evolution theory, survival of the fittest

12

13 Evolutionary Algorithms Selection: roulette wheel with fitness Crossover: Cross at random point Mutation: Random bit change

14 Genetic Programming Program is an individual Crossover and mutation of program’s abstract syntax tree Particular use – to find functions which describe data

15 Genetic Programming - Crossover

16 2. White-Box Testing Statement coverage Branch coverage Specific path(statement) selection

17 White-Box Testing Input variables (x1, x2, … Xk) Program Domain D1D2…Dk Individuals decode input of the program Goal: to find input data that satisfies coverage criteria (statement / branch / path)

18 Fitness Function = AL + D AL: Approximation Level acc. McMinn [3] Critical branch: branch missing the Target AL =(Number of critical branches between Target and diverging point) - 1

19 Fitness Function = AL + D D: branch distance If (x==y) then … if x!=y then D=abs(x-y) else D=0 D normalize to [0,1] Goal: min fitness (min D) if (x<y) then … If x>=y then D=x-y else D=0 If (flag) then … If flag==false then D=K else D=0

20 Example: Triangle Classification Input: int a,b,c from [0,15] 1. Sort a,b,c that a<=b<=c 2. If a+b<=c then NOT A TRIANGLE 3. If a==b or b==c then EQUILATERAL 4. If a==b and b==c then ISOSCELES 5. Else REGULAR

21 Optimization problem Program domain: III Individual string Goal: branch coverage Sub-goals: NOT A TRIANGLE, EQUILATERAL, ISOSCELES, REGULAR

22 Simulation by hand Goal: EQUILATERAL Generation: REGULAR NOT A TRIANGLE EQUILATERAL

23 GA vs. Random Testing Schatz[11] Comp(x,y) = x nesting “if” complexity; y condition complexity

24 Real Example Schatz[11] Autopilot system 2046 LOC 75 conditions Branch cov. Performance GA vs. random

25 Real Example Performance of gradient descent algorithm

26 White Box Testing Summary Variety of problem mapping and fitness functions Flag and state problems

27 3. Black Box Testing - Tracey [4] Specification with pre/post-conditions Search for test input data that satisfy pre-condition && ! post-condition Good fitness for data that is near to satisfy boolIf TRUE then 0 else K a==bIf abs(a-b)==0 then 0 else abs(a-b)+K a<bIf a-b<0 then 0 else (a-b)+K a&&bfit(a) + fit(b) a||bmin(fit(a), fit(b))

28 int wrap_counter(int n) {//pre (n>=0 && n<=10) if (n>=10) i=0; else i=n+1; return i; //post (n i=n+1) //post (n=10 -> i=0)} Goal1: n>=0 && n<=10 && (n<10 && i!=n+1) Goal2: n>=0 && n<=10 && (n=10 && i!=0)

29 Example Insert Error: if (n>10) i=0 Goal2: n>=0 && n<=10 && (n=10 && i!=0) n=2 i=3 : 0+0+(8+K)+0=8+K n=7 i=8 : 0+0+(3+K)+0=3+K n=10 i=11 : =0 FOUND!!!

30 Application Applied to safety-critical nuclear protection system Use simulated annealing and GA for search Use mutation testing to insert errors About 2000 lines of executable code 733 different disjunctive goals 100% error detection The code was simple

31 3. Black-Box Testing Automated Parking System [10] Individual: geometry data of parking space and vehicle – 6 parameters Fitness: minimum distance to collision area Results: 880 scenarios-25 incorrect

32 Parking System - Results

33 Parking System - Results

34 4. Object-Oriented Testing Tonella [5]: Unit Testing of Classes 1. Create object of class under test 2. Put the object to proper state Repeat 1 and 2 for all required objects 3. Invoke method under test 4. Examine final state

35 Individual String and Fitness That means A a = new A(); B b = new B(); a.m(3,b); Goal: branch coverage Fitness: proportion of exercised decision nodes that lead to the target

36 Crossover Crossover at random point after constructor and before tested method Repair the individual string

37 Mutations Mutation of input value Constructor change Insertion of method call Removal of method call

38 ClassLOCExecs Time (sec) String Tokenizer / BitSet / h HashMap / HashMap / LinkedList / Stack118510/ TreeSet / Public methods Branch Coverage Test cases

39 5. Non-Functional Testing Execution Time Testing Real-time systems: WC/BC exe time Fitness: execution time for specified input Problem: no sufficient guidance for search Wegener[6] on real systems: GA is better than random and hand-made Problem with low probability branches No guarantee to find WC/BC execution time

40 6. SBSE Search Based Software Engineering Module Clustering using simulated annealing, genetic algorithms Cost/Time Estimation using genetic programming Re-engineering using Program Transformation Ryan[7]: Automatic parallelization using genetic programming

41 1. Goldberg “Genetic Algorithms” 2. McMinn “SBS Test Data Generation: A Survey” 3. McMinn “Hybridizing ET with the Chaining Approach” 4. Tracey “A Search Based Automated Test-Data Generation Framework for Safety-Critical Systems” 5. Tonella “Evolutionary Testing of Classes” 6. Wegener,Pitschinetz,Sthamer “Automated testing of real- time tasks” 7. Ryan “Automatic re-engineering of software using genetic programming” 8. Nasa “Intelligence report” 9. “Reformulating Software Engineering as a Search Problem” Clarke,Jones…(11 authors) 10. Buehler,Wegener “Evolutionary Functional Testing of an Automated Parking System” 11. McGraw, G., Michael, C., Schatz, M. “Generating Software Test Data by Evolution” References