A Comparison of Online and Dynamic Impact Analysis Algorithms Ben Breech Mike Tegtmeyer Lori Pollock University of Delaware

Impact Analysis If I change function C, what other functions could be affected (impacted)? Results useful for regression testing and maintenance A main B C D E GH F

Our Contributions New online impact analysis algorithm Experimentally compare two dynamic and two online algorithms Space Time Scalability Effectiveness

Static Impact Analysis Perform analysis on source code Call graph traversals Slicing Obtain conservative results + Accounts for all possible inputs and behaviors - Can give very large impact sets Source Code Static Analyzer Impacts

Dynamic Impact Analysis -Not conservative -- results depend on input +Give impacts related to program use Algorithms: CoverageImpact (Orso et al. FSE 03) PathImpact (Law & Rothermel, ICSE 03) Execute Instrumented Program Instrument Program Post Exec Analysis Dynamic Data Impacts Input

Dynamic: CoverageImpact Calculate Function coverage by executing program Static forward slice from changed function Intersection gives impacts A main B C D E GH F A C D F E Slice functions: {Main, A, B, C, D, E, F} Function Coverage: {Main, A, C, D, E, F, G} Find Impact of E: (Orso et al. FSE 03)

Dynamic: PathImpact Calculate function execution trace Compress using SEQUITUR ( can be done at execution ) Traverse compressed trace to find impacts Once called, function impacts all future functions Returns cause impacts A main B C D E GH F Trace: Main G r A C F C r r D r E r D r r r x main A C D E Find Impact of E: ( Law & Rothermel, ICSE 03)

Our Approach: Online Impact Analysis Dynamic Compiler Executing Program Impact Analysis Impacts Input Use dynamic compiler to perform impact analysis as program runs ( Breech et al., ICSM 2004 )

Dynamic Compilers Allows for … modifying program instructions during execution analysis during execution Executable code DynamoRIO Analysis and/or Optimization (user module) Instruction Execution Program Results (Bruening et al., CGO 2003)

Online: Optimistic Optimistic  impacts propagate only along call paths Impacts of f are functions on call stack at the same time as f + May be useful for programs with infrequent global variable use + Calculates all impacts during one execution - May miss some impacts ( Breech et al., ICSM 2004 ) A main B C D E GH F Trace: Main G r A C F C r r D r E r D r r r x (not actually stored, only for demonstration) Find Impact of E: main A CE Call stack:main GACFDEC

Online: PathImpact_AllIn1 Same heuristics as PathImpact Execution impacts all future functions Returns cause impacts Calculates all impacts during one execution Uses matrix of bits to keep track of impacts A main B C D E GH F Trace: Main G r A C F C r r D r E r D r r r x (not actually stored, only for demonstration) Find Impact of E: main A C D E

Experimental Study Gain insight into tradeoffs of each alg Used several small to medium sized applications from SPEC + ESA space 1, ,000 LOC ,015 functions Research Questions How do the algs scale in time + space? How do resulting impact sets compare? Is it useful to apply dynamic compilers?

Experimental Setup: CoverageImpact DynamoRIO P Module to Compute coverage Function Coverage CodeSurfer ( ICFG Post Execution Analyzer (Approx slicing with reachability) Impacts

Experimental Setup: PathImpact P Instrument using SUIF P’ Call Trace SEQUITUR Post Execution Analyzer Impacts execute

Experimental Setup: Optimistic P DynamoRIO Module for Optimistic Impacts

Experimental Setup: PathImpact_AllIn1 Created 2 versions to examine usefulness of using dynamic compilers P’ Execute + Pipe Trace Analyze During Exec. Impacts P DynamoRIO Module for PathImpact_AllIn1 P Instrument using SUIF

Results: Timing CoverageImpact Least run time overhead (~ 3 minutes max) Fast post execution analysis (~ few minutes) PathImpact Very high overhead (> 2 hours, in some cases) Optimistic Medium overhead (usually ~ minutes) > 2 hours for heavily recursive program PathImpact_AllIn1 Version implemented with dynamic compiler Low to medium overhead (15 minutes max) Ran successfully on all programs Overhead of pipe version too high in some cases

Results: Space CoverageImpact ICFG took ~100K to ~8 MB PathImpact Uncompressed trace ~2KB to > 15GB Compressed trace ~400B to 1.2MB Could not be completed for all traces Optimistic, PathImpact_AllIn1 Negligible storage required

Results: Impact Sets Examined 008.espresso results Largest app that all algs ran (186 exec funcs) PathImpact and PathImpact_AllIn1 gave same results CoverageImpact reported ~30 more functions per impact set than PathImpact for 172 funcs CoverageImpact reported smaller sets for 10 functions -- one function called through function pointer Optimistic reported much smaller sets (but may miss impacts)

Qualitative Summary Coverage Impact Path Impact OptimisticPI_AllIn1 Infrastructure Instr. code ICFG Instr. code Compress Dyn. Comp Static Analysis ICFG slicing (none) Analysis Timelowhighmed to highmed Spatial Overhead low to med low (if compress) low

Summary Presented new online impact analysis algorithm (PathImpact_AllIn1) Experimentally compared online against dynamic algs PathImpact_Allin1 Scaled better than PathImpact Gave better impact sets than CoverageImpact Optimistic Scaled well (not as good as PathImpact_Allin1) Gave different impact sets (may be useful in OOP)

Future Work Improve precision of online impact analysis Other uses of dynamic compilers for software engineering