Software Testing: A Research Travelogue Alessandro Orso, Gregg Rothermel

Automated Test Input Generation Symbolic Execution Search-based testing Random and fuzzing testing Combinations

Symbolic Execution Static Symbolic Execution General Formula 1970 by King General Formula …

Symbolic Execution Reason of success Increase in computational power Z3 , Yices , SAT4J Different variants of symbolic execution KLEE DART CUTE Pex

Symbolic Execution Dynamic Symbolic Execution 2000 DART/CUTE/Pex Idea of performing symbolic execution that follows a specific path

Symbolic Execution Unclear for lasting practical impact Inherent limitations in the presence of highly structured inputs Interact with external libraries Large complex programs Need for an oracle

Symbolic Execution White-box fuzzing Sage

Search-based Testing Surveys on use of SBST focusing on Software engineering Comprehensive analysis and review of trends Trends, techniques and applications

Search-based Testing Search-based techniques target optimization problems finding the smallest set of test cases that cover all the branches Using meta-heuristic search-based optimization techniques

Search-based Testing Large range to which it has been applied Structural testing, model-based testing, mutation testing, temporal testing, exception testing, configuration and interaction testing, stress testing, integration testing etc. Comprehensive analysis and review of trends

Search-based Testing Come in many forms Genetic algorithms Simulated annealing Hill climbing Scatter search Partical swarm optimization Tabu search

Search-based Testing Challenges and opportunities for SBST Challenges An orchestrated survey of methodologies for automated software test case generation Challenges Oracle Combining SBST with SE Opportunities Co-evolutionary computation, involving multiple populations, possibly under different fitness functions. Predator-prey relationships Hyper-heuristic software testing, unite different software activities that utilize SBST

Search-based Testing Empirical studies Shortcoming Search-Based Test Case Generation Shortcoming Focusing on unit testing and structural coverage Limited evidence of applicability for other testing phases and types of coverage

Random Testing Improve the random input generation process Proprotional sampling strategy JCrasher Manage the often overwhelmingly large number of test inputs generated Taming Compiler fuzzers

Random Testing One example ART Studies to show the advantages High overhead and not yet been extended to handle complex input formats To address problem, techniques are proposed based on mirroring, forgetting, and Voronoi tessellation

Random Testing Representative and well-known examples Jcrasher Randoop Follow up that combines test input generation and static analysis[1][2] Randoop Incorporating feedback into the process De-facto random-testing tool for Java

Random Testing Additional approaches Swarm testing Swarm Attempts to increase the diversity of randomly-generated test inputs

Random Testing Combination with DSE to generate input that can be used to seed symbolic analysis

Combined Techniques Combination of static verification and dynamic verification(i.e., testing) Yogi project Testing, under-approximates program behaviors Static verification, complete but over-approximates program behavior

Combined Techniques Combining SE and SBST Complementary nature of SE and SBST Using SE as an additional genetic operator[1][2] Alternating between SE and SBST Using fitness to select which path to explore in SE Using symbolic execution to compute fitness values in SBST