Introduction to Search Based Software Engineering SSS SEBASE Summer School, Birmingham University, July 2007 Mark Harman King’s College London

Introduction to Search Based Software Engineering SSSS Second SEBASE Summer School, Birmingham University, July 2008 Mark Harman King’s College London

Introduction to Search Based Software Engineering How on earth can software engineers do what John says they claim do Mark Harman King’s College London

How on earth can software engineers do what John says they claim do - To engineer is to optimize Mark Harman King’s College London

Introduction to Search Based Software Engineering Thanks John Clark, University of York, Sebastian Elbaum, University of Nebraska Lincoln Rob Hierons, Brunel University Zheng Li, King’s College London Kiarash Mahdavi, King’s College London Spiros Mancoridis, Drexel University Afshin Mansouri, King’s College London Jian Ren, King’s College London Joachim Wegener, DaimlerChrysler Shin Yoo, King’s College London YuanYuan Zhang, King’s College London

Mark Harman Introduction to SBSE 6 Where is King’s College London? 20 minutes’ walk

Mark Harman Introduction to SBSE 7 Search Based Software Engineering SBSE what and why SBSE case studies SBSE how Current and future trends Advantages Multi Objective Search Insight through search Co Evolution

Mark Harman Introduction to SBSE 8 Search Based Software Engineering SBSE what and why SBSE case studies SBSE how Current and future trends – personal views Advantages Multi Objective Search Insight through search Co Evolution

Mark Harman Introduction to SBSE 9 Search Based Software Engineering SBSE what and why SBSE case studies SBSE how Current and future trends – IMHO Advantages Multi Objective Search Insight through search Co Evolution

Mark Harman Introduction to SBSE 10 Search Based Software Engineering SBSE what and why SBSE case studies SBSE how Current and future trends – In Mark Harman’s Opinion Advantages Multi Objective Search Insight through search Co Evolution

Mark Harman Introduction to SBSE 11 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek.

Mark Harman Introduction to SBSE 12 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. Genetic Algorithms, Hill climbing, Simulated Annealing, Random, Tabu Search, Estimation of Distribution Algorithms, Particle Swarm Optimization, Ant Colonies, LP, Genetic programming, Greedy

Mark Harman Introduction to SBSE 13 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. Genetic Algorithms, Hill climbing, Simulated Annealing, Random, Tabu Search, Estimation of Distribution Algorithms, Particle Swarm Optimization, Ant Colonies, LP, Genetic programming, Greedy

Mark Harman Introduction to SBSE 14 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek.

Mark Harman Introduction to SBSE 15 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. So it is merely another application of Optimization Techniques?

Mark Harman Introduction to SBSE 16 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. So it is merely another set of applications of Optimization Techniques?

Mark Harman Introduction to SBSE 17 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. So it is merely another rich set of applications of Optimization Techniques?

Mark Harman Introduction to SBSE 18 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. So it is merely another rich set of applications of Optimization Techniques?

Mark Harman Introduction to SBSE 19 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. Have nails … seek hammer

Mark Harman Introduction to SBSE 20 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. Have nails … seek hammers

Mark Harman Introduction to SBSE 21 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. Have nails … seek hammers Generalise, compare, develop

Mark Harman Introduction to SBSE 22 What is SBSE In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek. Have nails … seek hammers Generalise, compare, develop in an unbiased manner

Mark Harman Introduction to SBSE 23 What is Research? In research we apply social search techniques to search large search spaces, guided by a natural evolution of memes Have nails … seek hammers Generalise, compare, develop in an unbiased manner

Mark Harman Introduction to SBSE 24 What is Research? In research we apply social search techniques to search large search spaces, guided by a natural evolution of memes and all the usual human frailties Have nails … seek hammers Generalise, compare, develop in an unbiased manner

Mark Harman Introduction to SBSE 25 Why?

Mark Harman Introduction to SBSE 26 The Eight Queens Problem

Mark Harman Introduction to SBSE 27 The Eight Queens Problem

Mark Harman Introduction to SBSE 28 The Eight Queens Problem

Mark Harman Introduction to SBSE 29 The Eight Queens Problem Perfect

Mark Harman Introduction to SBSE 30 The Eight Queens Problem PerfectScore 0

Mark Harman Introduction to SBSE 31 The Eight Queens Problem

Mark Harman Introduction to SBSE 32 The Eight Queens Problem

Mark Harman Introduction to SBSE 33 The Eight Queens Problem Two Attacks

Mark Harman Introduction to SBSE 34 The Eight Queens Problem Two Attacks Score -2

Mark Harman Introduction to SBSE 35 The Eight Queens Problem

Mark Harman Introduction to SBSE 36 The Eight Queens Problem

Mark Harman Introduction to SBSE 37 The Eight Queens Problem Three Attacks

Mark Harman Introduction to SBSE 38 The Eight Queens Problem Three Attacks Score -3

Mark Harman Introduction to SBSE 39 That was easy

Mark Harman Introduction to SBSE 40 Generate a solution Place 8 queens on the board so that there are no attacks

Mark Harman Introduction to SBSE 41 Scale up: Generate a solution Place 44 queens on the board so that there are no attacks

Mark Harman Introduction to SBSE 42 Scale up: Generate a solution Place 400 queens on the board so that there are no attacks

Mark Harman Introduction to SBSE 43 Checking vs Generating Task One: Write a method to determine which is the better of two placements of N queens Task Two: Write a method to construct a board placement with N non attacking queens

Mark Harman Introduction to SBSE 44 Checking vs Generating Task One: Write a method to determine which is the better of two placements of N queens Task Two: Write a method to construct a board placement with N non attacking queens

Mark Harman Introduction to SBSE 45 Checking vs Generating Search Based Software Engineering Write a method to determine which is the better of two solutions Conventional Software Engineering Write a method to construct a perfect solution

Mark Harman Introduction to SBSE 46 Checking vs Generating Search Based Software Engineering Write a method to determine which is the better of two solutions Conventional Software Engineering Write a method to construct a perfect solution

Mark Harman Introduction to SBSE 47 Checking vs Generating Search Based Software Engineering Write a method to determine which is the better of two solutions Conventional Software Engineering Write a method to construct a perfect solution

Mark Harman Introduction to SBSE 48 Checking vs Generating Search Based Software Engineering Write a fitness function to determine which is the better of two solutions Conventional Software Engineering Write a method to construct a perfect solution

Mark Harman Introduction to SBSE 49 Checking vs Generating Search Based Software Engineering Write a fitness function to guide a search aaaa Conventional Software Engineering Write a method to construct a perfect solution

Mark Harman Introduction to SBSE 50 Checking vs Generating Search Based Software Engineering Write a fitness function to guide automated search Conventional Software Engineering Write a method to construct a perfect solution

Mark Harman Introduction to SBSE 51 Examples

Mark Harman Introduction to SBSE 52 Case Studies

Mark Harman Introduction to SBSE 53 Evolutionary Algorithms Selection Insertion Recombination Mutation Fitness evaluation Test execution End?

Mark Harman Introduction to SBSE 54 Evolutionary Testing Selection Insertion Recombination Mutation Fitness evaluation Test cases Test execution End?

Mark Harman Introduction to SBSE 55 Evolutionary Testing Selection Insertion Recombination Mutation Fitness evaluation Test cases Test execution End? Execution

Mark Harman Introduction to SBSE 56 Evolutionary Testing Selection Insertion Recombination Mutation Fitness evaluation Test cases Test execution End? Execution Monitoring

Mark Harman Introduction to SBSE 57 Temporal Testing deceleration t min t max t optimal point of time for triggering the airbag igniter Selection Insertion Recombination Mutation Fitness evaluation Individuals Test data Monitoring Fitness values Test execution End?

Mark Harman Introduction to SBSE 58 Temporal Testing deceleration t min t max t optimal point of time for triggering the airbag igniter Selection Insertion Recombination Mutation Duration Individuals Test data Monitoring Fitness values Test execution End?

Mark Harman Introduction to SBSE 59 Evolution vs Random for Temporal Testing

Mark Harman Introduction to SBSE 60 Search Based Testing Applications Temporal testing Coverage based testing Functional testing Regression testing Finite State Machine testing Interaction testing Exception testing Stress testing Robustness testing

Mark Harman Introduction to SBSE 61 Search Based Testing Applications Temporal testing Coverage based testing Functional testing Regression testing Finite State Machine testing Interaction testing Exception testing Stress testing Robustness testing Phil will talk about some of these in more detail shortly

Mark Harman Introduction to SBSE 62 Search Based Testing Applications Temporal testing Coverage based testing Functional testing Regression testing Finite State Machine testing Interaction testing Exception testing Stress testing Robustness testing

Mark Harman Introduction to SBSE 63

Mark Harman Introduction to SBSE 64

Mark Harman Introduction to SBSE 65

Mark Harman Introduction to SBSE 66

Mark Harman Introduction to SBSE 67

Mark Harman Introduction to SBSE 68

Mark Harman Introduction to SBSE 69 Finding UIOs Hierons et al s1s1 s2s2 s3s3 a/0 a/1 b/1 b/0 a/1s3s3 b/0s2s2 b/1,a/1s1s1 UIOState

Mark Harman Introduction to SBSE 70 Target Level 4 Level 3 Level 2 Level 1 Fitness = Approximation_Level + Local_Distance Evaluation of predicate in a branching condition if A = B Local_Distance = | A - B | Identify relevant branching statements using control dependence Target 1. Approximation level Structural Evolutionary Testing

Mark Harman Introduction to SBSE 71 Evolutionary Testing vs Random EffectivenessCost

Mark Harman Introduction to SBSE 72 Autonomous Parking System

Mark Harman Introduction to SBSE 73 Autonomous Parking System

Mark Harman Introduction to SBSE 74 Autonomous Parking System Stop

Mark Harman Introduction to SBSE 75 Autonomous Parking System - Input psi gap dist2space space length space width

Mark Harman Introduction to SBSE 76 Autonomous Parking System

Mark Harman Introduction to SBSE 77 Regression Testing Test case prioritzation Test suite reduction Test case selection

Mark Harman Introduction to SBSE 78 Regression Testing Test case prioritzation Li, Harman, Hierons, TSE 2007 Test suite reduction Test case selection

Mark Harman Introduction to SBSE 79 Test Case Prioritization Order test cases higher priority => earlier execution

Mark Harman Introduction to SBSE 80 Greedy Optimal orderings: ABC BCA CBA

Mark Harman Introduction to SBSE 81 Regression Testing Test case prioritzation Test suite reduction Test case selection

Mark Harman Introduction to SBSE 82 Regression Testing Test case prioritzation Test suite reduction Yoo and Harman, ISSTA 07 Test case selection

Mark Harman Introduction to SBSE 83 Regression Test Selection

Mark Harman Introduction to SBSE 84 Regression Testing Test case prioritzation Test suite reduction Test case selection you and someone else, TSE 2008/9 ?

Mark Harman Introduction to SBSE 85 Regression Testing Test case prioritzation Test suite reduction Test case selection yoo and me, TSE 2008/9 ?

Mark Harman Introduction to SBSE 86 Does this look familiar?

Mark Harman Introduction to SBSE 87 Does this look familiar?

Mark Harman Introduction to SBSE 88 MDG M1 M2M3 M4 M5M6 Good Partition! M1 M2 M3 M4 M5 M6 Bad Partition! M1 M2 M4 M3 M5 M6 MQ(Good Partition) > MQ(Bad Partition) The Bunch Approach Mancoridis et al

Mark Harman Introduction to SBSE 89 Hill Climbing

Mark Harman Introduction to SBSE 90 Hill Climbing

Mark Harman Introduction to SBSE 91 Hill Climbing Maybe this is no longer the optimal approach …

Mark Harman Introduction to SBSE 92 Hill Climbing Kata Praditwong et al.

Mark Harman Introduction to SBSE 93 SBSE is so generic Testing Fitness function: execution time Representation: input vector

Mark Harman Introduction to SBSE 94 SBSE is so generic Testing Fitness function: execution time Representation: input vector Restructuring Fitness function: cohesion and coupling Representation: mapping from modules to clusters

Mark Harman Introduction to SBSE 95 SBSE is so generic Testing Fitness function: execution time Representation: input vector Requirements Fitness function: customer satisfaction, cost Representation: bitset of requirements

Mark Harman Introduction to SBSE 96 SBSE is so generic Testing Fitness function: execution time Representation: input vector Program Comprehension Fitness function: (in)direct aspect of cognition Representation: code, architecture, visualization, model

Mark Harman Introduction to SBSE 97 SBSE is so generic Testing Fitness function: execution time Representation: input vector Project Management Fitness function: project duration Representation: mapping from teams to WPs

Mark Harman Introduction to SBSE 98 SBSE is so generic Time for two more examples Program Comprehension Mahdavi et al. ICSM 05 Requirements Analysis Zhang et al. GECCO 07

Mark Harman Introduction to SBSE 99 SBSE is so generic Time for two more examples Program Comprehension Mahdavi et al. ICSM 05 Requirements Analysis

Mark Harman Introduction to SBSE 100 Concept Assignment “The process of assigning descriptive terms usually relating to computational intent.” MOVE ‘EXAMPLE’ TO PRINT-LL MOVE ‘13’ TO PRINT-CC. CALL ‘PRINT’ USING P-PRINTLINE. MOVE POLICY-NUM TO OUT-PNUM. MOVE SCHEME-REF TO OUT-SREF. CALL ‘WRITE’ USING OUT-REC. Write Print

Mark Harman Introduction to SBSE 101 Allowing Overlapping Concept Binding Fitness: Signal to noise ratio Representation: Space of possible overlapping bindings MOVE ‘EXAMPLE’ TO PRINT-LL MOVE POLICY-NUM TO OUT-PNUM. MOVE ‘13’ TO PRINT-CC. MOVE SCHEME-REF TO OUT-SREF. CALL ‘PRINT’ USING P-PRINTLINE. CALL ‘WRITE’ USING OUT-REC. Print Write Call

Mark Harman Introduction to SBSE 102 SBSE is so generic Time for two more examples Program Comprehension Requirements Analysis Zhang et al. GECCO 07

Mark Harman Introduction to SBSE 103 The Next Release Problem

Mark Harman Introduction to SBSE 104 Motorola Cell Phone Requirements

Mark Harman Introduction to SBSE 105 SBSE Applications TransformationCooper, Ryan, Schielke, Subramanian, Fatiregun, Williams Requirements Bagnall, Mansouri, Zhang Effort prediction Aguilar-Ruiz, Burgess, Dolado, Lefley, Shepperd ManagementAlba, Antoniol, Chicano, Di Pentam Greer, Ruhe Heap allocationCohen, Kooi, Srisa-an Regression testLi, Yoo, Elbaum, Rothermel, Walcott, Soffa, Kampfhamer SOA Canfora, Di Penta, Esposito, Villani RefactoringAntoniol, Briand, Cinneide, O’Keeffe, Merlo, Seng, Tratt Test GenerationAlba, Binkley, Bottaci, Briand, Chicano, Clark, Cohen, Gutjahr, Harrold, Holcombe, Jones, Korel, Pargass, Reformat, Roper, McMinn, Michael, Sthamer, Tracy, Tonella,Xanthakis, Xiao, Wegener, Wilkins MaintenanceAntoniol, Lutz, Di Penta, Madhavi, Mancoridis, Mitchell, Swift Model checkingAlba, Chicano, Godefroid Probe dist’ionCohen, Elbaum UIOsDerderian, Guo, Hierons ComprehensionGold, Li, Mahdavi ProtocolsAlba, Clark, Jacob, Troya Component selBaker, Skaliotis, Steinhofel, Yoo Agent OrientedHaas, Peysakhov, Sinclair, Shami, Mancoridis

Mark Harman Introduction to SBSE 106 SBSE Applications TransformationCooper, Ryan, Schielke, Subramanian, Fatiregun, Williams Requirements Bagnall, Mansouri, Zhang Effort prediction Aguilar-Ruiz, Burgess, Dolado, Lefley, Shepperd ManagementAlba, Antoniol, Chicano, Di Pentam Greer, Ruhe Heap allocationCohen, Kooi, Srisa-an Regression testLi, Yoo, Elbaum, Rothermel, Walcott, Soffa, Kampfhamer SOA Canfora, Di Penta, Esposito, Villani RefactoringAntoniol, Briand, Cinneide, O’Keeffe, Merlo, Seng, Tratt Test GenerationAlba, Binkley, Bottaci, Briand, Chicano, Clark, Cohen, Gutjahr, Harrold, Holcombe, Jones, Korel, Pargass, Reformat, Roper, McMinn, Michael, Sthamer, Tracy, Tonella,Xanthakis, Xiao, Wegener, Wilkins MaintenanceAntoniol, Lutz, Di Penta, Madhavi, Mancoridis, Mitchell, Swift Model checkingAlba, Chicano, Godefroid Probe dist’ionCohen, Elbaum UIOsDerderian, Guo, Hierons ComprehensionGold, Li, Mahdavi ProtocolsAlba, Clark, Jacob, Troya Component selBaker, Skaliotis, Steinhofel, Yoo Agent OrientedHaas, Peysakhov, Sinclair, Shami, Mancoridis

Mark Harman Introduction to SBSE 107 SBSE Applications TransformationCooper, Ryan, Schielke, Subramanian, Fatiregun, Williams Requirements Bagnall, Mansouri, Zhang Effort prediction Aguilar-Ruiz, Burgess, Dolado, Lefley, Shepperd ManagementAlba, Antoniol, Chicano, Di Pentam Greer, Ruhe Heap allocationCohen, Kooi, Srisa-an Regression testLi, Yoo, Elbaum, Rothermel, Walcott, Soffa, Kampfhamer SOA Canfora, Di Penta, Esposito, Villani RefactoringAntoniol, Briand, Cinneide, O’Keeffe, Merlo, Seng, Tratt Test GenerationAlba, Binkley, Bottaci, Briand, Chicano, Clark, Cohen, Gutjahr, Harrold, Holcombe, Jones, Korel, Pargass, Reformat, Roper, McMinn, Michael, Sthamer, Tracy, Tonella,Xanthakis, Xiao, Wegener, Wilkins MaintenanceAntoniol, Lutz, Di Penta, Madhavi, Mancoridis, Mitchell, Swift Model checkingAlba, Chicano, Godefroid Probe dist’ionCohen, Elbaum UIOsDerderian, Guo, Hierons ComprehensionGold, Li, Mahdavi ProtocolsAlba, Clark, Jacob, Troya Component selBaker, Skaliotis, Steinhofel, Yoo Agent OrientedHaas, Peysakhov, Sinclair, Shami, Mancoridis

Mark Harman Introduction to SBSE 108 The two SBSE ingredients Representation Fitness function

Mark Harman Introduction to SBSE 109 The two SBSE ingredients Representation Fitness function Should be easy We always represent Software Engineering problems in data structures

Mark Harman Introduction to SBSE 110 The two SBSE ingredients Representation Fitness function Often easy We often define metrics

Mark Harman Introduction to SBSE 111 So far SBSE is useful SBSE is generic SBSE is easy

Mark Harman Introduction to SBSE 112 Future Multiple objective search Co evolution Sensitivity analysis Metric validation Landscape analysis Interactive evolution

Mark Harman Introduction to SBSE 113 Future Multiple objective search Co evolution Sensitivity analysis Metric validation Landscape analysis Interactive evolution

Mark Harman Introduction to SBSE 114 Co Evolution Two populations; Two fitness functions Collaboration, and competition

Mark Harman Introduction to SBSE 115 Co Evolution Two populations; Two fitness functions Collaboration, and competition applied to mutation testing Adamopoulos, Harman and Hierons GECCO 2004

Mark Harman Introduction to SBSE 116 MutantsTest Cases

Mark Harman Introduction to SBSE 117 MutantsTest Cases Fitness?

Mark Harman Introduction to SBSE 118 MutantsTest Cases Fitness for test cases How many mutants do you kill ?

Mark Harman Introduction to SBSE 119 MutantsTest Cases Fitness for mutants How many test cases do you survive?

Mark Harman Introduction to SBSE 120 Sensitivity analysis Inputs to search are typically estimates Estimated Software Engineering attributes Which estimates matter most?

Mark Harman Introduction to SBSE 121 Sensitivity analysis Inputs to search are typically estimates Estimated Software Engineering attributes Which estimates matter most? applied to component selection Yoo, Ren, Harman, Not yet published

Mark Harman Introduction to SBSE 122 Motorola Cell Phone Requirements

Mark Harman Introduction to SBSE 123 Metric Robustness Metric performs under … change, noise, fuzziness Not just good solutions; robust solutions

Mark Harman Introduction to SBSE 124 Metric Robustness Metric performs under … change, noise, fuzziness Not just good solutions; robust solutions Metrics are fitness functions too! Harman and Clark, Metrics Symposium 2004

Mark Harman Introduction to SBSE 125 Landscape Analysis What landscape is denoted by the metric? Landscape properties smoothness, plateaux, …

Mark Harman Introduction to SBSE 126 Landscape Analysis and Improvement Improve the fitness landscape by transformation SBSE allows transformation of program under test A Testability Transformation Harman et al – 2008

Mark Harman Introduction to SBSE 127 Landscape Analysis and Improvement Improve the fitness landscape by transformation SBSE allows transformation of program under test A Testability Transformation Harman et al – 2008 An example for Structural Testing McMinn et al. ISSTA 2007

Mark Harman Introduction to SBSE 128 Hitchcock Landscape void alfred(double a, double b) { if (a == b) { // target 1 double c = b + 1; if (c == 0) { // target 2...

Mark Harman Introduction to SBSE 129 Hitchcock Landscape

Mark Harman Introduction to SBSE 130 Hitchcock Landscape

Mark Harman Introduction to SBSE 131 Interactive evolution Harman ICPC 2007 Human in the loop Caters for intangibles Can flush out implicit assumptions Co evolution of cognitive models Lots of fun things to do here …

Mark Harman Introduction to SBSE 132 Summary Software Engineering problems can be optimized Software Engineering problems are often search problems This should not be a surprise The SBSE approach gives us a whole new set of Tools Concepts Techniques Insights To optimize

Mark Harman Introduction to SBSE 133 Summary Software Engineering problems can be optimized Software Engineering problems are often search problems This should not be a surprise The SBSE approach gives us a whole new set of Tools Concepts Techniques Insights To optimize is to engineer

Mark Harman Introduction to SBSE 134 Search Based Software Engineering is the focus of major initiatives SEBASE $5m 5 year project Search based software engineering Led by King’s Birmingham, York, Brunel Daimler, Motorola, IBM EvoTest $4m 3 year project Evolutionary testing Led by Valencia King’s, FIRST, INRIA, Daimler, Motorola, RILA, Xenon

Mark Harman Introduction to SBSE 135 Summary

Mark Harman Introduction to SBSE 136 Summary

Mark Harman Introduction to SBSE 137 Conclusions Search is what you seek

Mark Harman Introduction to SBSE 138 Conclusions Search is what you seek

Mark Harman Introduction to SBSE 139 Conclusions Search is what you seek Questions?