Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos

Background: Static Compilers source code Lexical, syntactic, Semantic analysis Inter. Rep. (basic blocks) Static Analysis Optimization Code generation Executable Code Static Compiler

Background: Software Testing  Assesses quality of software Correctness, performance, security Correctness, performance, security  Most testing performed by executing code Executable Code Test Input Execute Program Results Expected Results Check Results (oracle)

Motivating Example if ((sptr = malloc (size + 1)) == NULL) { findmem (); findmem (); if ((sptr = malloc (size + 1)) == NULL) xlfail (“insufficient string space”); } How do I test this callsite?  Make the machine run out of memory?  Flip the conditional, recompile, flip back?  Pretend it doesn’t exist during testing?

Generalizing the Problem  Code to handle uncommon situations Difficult to test Difficult to test May need external environment event to trigger May need external environment event to trigger  Examples: Error handling code Error handling code Testing program security mechanisms Testing program security mechanisms

Observation  Hard to reach code executes when program thinks something uncommon has occurred if ((sptr = malloc (size + 1)) == NULL) { findmem (); findmem (); xlfail (“insufficient string space”); } if ((sptr = malloc (size + 1)) == NULL)  Could test findmem() by simulating error E.g., could add instructions to program so program believes malloc failed E.g., could add instructions to program so program believes malloc failed

RUGRAT Approach Use Dynamic Compilers to generate test cases for hard to reach code. Automatically add instructions to program during execution to simulate uncommon situation.

Dynamic Compilers  Dynamic compilers perform compilation tasks during program execution code Analysis & optimization Create basic block translate Basic block Mod. Basic block Execute on CPU Dynamic Compiler

RUGRAT Architecture code Analysis & optimization Create basic block translate Basic block Mod. Basic block Execute on CPU Dynamic Compiler Create basic block Dynatest Generator Test spec Test Oracle Test Report

Dynatest Generator  Decides when/where/how to add test Decision based on test specification Decision based on test specification  Test spec examples “test all mallocs in program” “test all mallocs in program” “test 2nd malloc call in function foo” “test 2nd malloc call in function foo”

Example if ((sptr = malloc (size + 1)) == NULL) { findmem (); findmem (); xlfail (“insufficient string space”); } if ((sptr = malloc (size + 1)) == NULL) call malloc (code for malloc) movl sptr cmpl sptr, 0 jnz L1 call findmem …. L1: … Dynatest Generator call malloc (code for malloc) movl 0, movl ENOMEM, errno movl sptr cmpl sptr, 0 jnz L1 call findmem …. L1: …

Good Times, Bad Times The Bad:  Not a perfect simulation The Good:  Adequate simulation  Can target system or appl calls  Saves quite a lot of tester effort

Security Mechanism Testing: Encrypting Function Pointers  Protects progs against func pointer attacks  Difficult to test (need vulnerable program and attack)  RUGRAT can simulate attack by adding instructions Very different from error handling code case Very different from error handling code case RUGRAT can be used for variety of testing tasks.

Experiments Summary  Tested variety programs with RUGRAT  120+ error code handling callsites covered Both application and system calls Both application and system calls  Increased error code coverage ~ 50% over regular test cases Not all error code statements could be covered Not all error code statements could be covered Different options, etcDifferent options, etc  Reasonable time overhead

Some related work  Holodeck [1], FIG [2] Require tester provide alternative “stub” functions to do testing Require tester provide alternative “stub” functions to do testing Miss application calls Miss application calls  Dynamic branch switching [3] Not originally intended for testing error code Not originally intended for testing error code Need to know which branch to change Need to know which branch to change Far less accurate simulation Far less accurate simulation [1] Thompson et al., SAC 2002 [2] Broadwell et al., SHAMAN 2002 [3] Zhang et al., ICSE 2006

Conclusions and Summary  Presented RUGRAT architecture Can test hard to reach (and seldom tested) code by using dynamic compilers Can test hard to reach (and seldom tested) code by using dynamic compilers Saves tester effort Saves tester effort  RUGRAT is a general tool

RUGRAT Architecture code Basic block Mod. Basic block Execute on CPU Dynamic Compiler Create basic block Dynatest Generator Test spec Test Oracle Test Report