Tao Xie North Carolina State University Supported by CACC/NSA Related projects supported in part by ARO, NSF, SOSI

 Software testing is important  Software errors cost the U.S. economy about $59.5 billion each year (0.6% of the GDP) [NIST 02]  Improving testing infrastructure could save 1/3 cost [NIST 02]  Software testing is costly  Account for even half the total cost of software development [Beizer 90]  Automated testing reduces manual testing effort  Test execution: JUnit, NUnit, xUnit, etc.  Test generation: Pex, AgitarOne, Parasoft Jtest, etc.  Test-behavior checking: Pex, AgitarOne, Parasoft Jtest, etc.

LUHN Formula (Mod 10)

MASTERCARD = 0; VISA = 1; DISCOVER = 2; AMEX = 3;

Version i Version i + 1  Security  Functionality  Performance  … Testing/retesting is expensive fixing faults improving performance/design adding new features…

Version i Version i + 1 Our Tool Test inputs/conditions causing different outputs

Our Tool Test inputs/conditions causing different outputs Instructor solution Student solution == ?

 Validation of security patch  Does the patch fix ALL attacks of the same type (more types) beyond given specific attacks?  Attack generation based on patch [Song et al. S&P08]  Can we exploit patches to other systems?  Revalidation of medical device software versions at FDA  There no complete software package available  Generation of tests/regression tests DURING code development  …

Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]== void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!= a!=null && a.Length>0 && a[0]== Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition

 Loops  Fitnex [Xie et al. DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [Li et al. ASE 09-sp]  Method sequences  MSeqGen [Thummalapenta et al. ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [Marri et al. AST 09] Opportunities  Regression testing [Taneja et al. ICSE 09-nier]  Developer guidance (cooperative developer testing)

 Loops  Fitnex [Xie et al. DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [Li et al. ASE 09-sp]  Method sequences  MSeqGen [Thummalapenta et al. ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [Marri et al. AST 09] Applications  Test network app at Army Hood, Texas  Test DB app of hand-held medical assistant device at FDA  Army Telemedicine and Advanced Technology Research Center (TATRC)

Download counts (20 months) (Feb Oct ) Academic: 17,366 Devlabs: 13,022 Total: 30,388

 Developers make changes to a software system  Bug fixes, refactorings, addition of new features…  Developers need to make sure that the changes  Introduce intended effect  not introduce unintended side effect  Developed upon Dynamic Symbolic Execution, a practical, powerful recent technique in academia/industry  Good impact to software industry and agencies/defense mission

Supported by CACC/NSA Related projects supported in part by ARO, NSF, SOSI