Evaluation of DART DART: Directed Automated Random Testing Godefroid, Klarlund, Sen

Experimental Goals  Efficiency of DART directed search approach vs purely random search directed search approach vs purely random search AC-controller program AC-controller program Needham-Schroeder Protocol Needham-Schroeder Protocol  Effectiveness with a large program Open-source oSIP library, 30K LOC of C code Open-source oSIP library, 30K LOC of C code

Efficiency Experiment  AC-Controller Program: DART: DART: Explores all exec paths uptoExplores all exec paths upto depth=1 in 6 iterations and less than 1 seconddepth=1 in 6 iterations and less than 1 second Depth=2, find assertaion violation, 7 iterations, <1 secDepth=2, find assertaion violation, 7 iterations, <1 sec Random: Random: Does not find assertion violation after hoursDoes not find assertion violation after hours Probability to find inputs leading assertion = 2**64Probability to find inputs leading assertion = 2**64 Gets stuck in input-filtering codeGets stuck in input-filtering code

Another Efficiency Point  Needham-Schroeder security protocol program 406 lines of C code 406 lines of C code  DART: Took < 26 minutes on 2GHz machine to detect middle man attack  VeriSoft (model checker): Hours to detect

Effectiveness with Large App  oSIP (open-source) 30K LOC, 600 externally visible functions  DART: Found a way to crach 65% of oSIP functions within 1000 attempts of each function Found a way to crach 65% of oSIP functions within 1000 attempts of each function Most were deferencing a null pointer sent as an argument to a function Most were deferencing a null pointer sent as an argument to a function

Putting this work into Context  Colby, Godefroid, Jagadeesan 1998: automatically make program self-executable and systematically explore all behaviors Close program is simplified version Close program is simplified version  Considerable work on test-vector generation with symbolic exec Imprecise static analysis Imprecise static analysis  Dynamic Test generation only generate for specific paths only generate for specific paths Do not deal with function calls or library funcs Do not deal with function calls or library funcs