1. Tao Xie xie@csc.ncsu.edu North Carolina State University Supported by CACC/NSA Related projects supported in part by ARO, NSF, SOSI

2.  Software testing is important  Software errors cost the U.S. economy about $59.5 billion each year (0.6% of the GDP) [NIST 02]  Improving testing infrastructure could save 1/3 cost [NIST 02]  Software testing is costly  Account for even half the total cost of software development [Beizer 90]  Automated testing reduces manual testing effort  Test execution: JUnit, NUnit, xUnit, etc.  Test generation: Pex, AgitarOne, Parasoft Jtest, etc.  Test-behavior checking: Pex, AgitarOne, Parasoft Jtest, etc.

5. http://www.beachnet.com/~hstiles/cardtype.html LUHN Formula (Mod 10)

6. MASTERCARD = 0; VISA = 1; DISCOVER = 2; AMEX = 3; http://research.microsoft.com/pex

7. Version i Version i + 1  Security  Functionality  Performance  … Testing/retesting is expensive fixing faults improving performance/design adding new features…

8. Version i Version i + 1 Our Tool Test inputs/conditions causing different outputs

9. Our Tool Test inputs/conditions causing different outputs Instructor solution Student solution == ?

10.  Validation of security patch  Does the patch fix ALL attacks of the same type (more types) beyond given specific attacks?  Attack generation based on patch [Song et al. S&P08]  Can we exploit patches to other systems?  Revalidation of medical device software versions at FDA  There no complete software package available  Generation of tests/regression tests DURING code development  …

11. Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition

12.  Loops  Fitnex [Xie et al. DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [Li et al. ASE 09-sp]  Method sequences  MSeqGen [Thummalapenta et al. ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [Marri et al. AST 09] Opportunities  Regression testing [Taneja et al. ICSE 09-nier]  Developer guidance (cooperative developer testing)

13.  Loops  Fitnex [Xie et al. DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [Li et al. ASE 09-sp]  Method sequences  MSeqGen [Thummalapenta et al. ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [Marri et al. AST 09] Applications  Test network app at Army division@Fort Hood, Texas  Test DB app of hand-held medical assistant device at FDA  Army Telemedicine and Advanced Technology Research Center (TATRC)

14. Download counts (20 months) (Feb. 2008 - Oct. 2009 ) Academic: 17,366 Devlabs: 13,022 Total: 30,388

15.  Developers make changes to a software system  Bug fixes, refactorings, addition of new features…  Developers need to make sure that the changes  Introduce intended effect  not introduce unintended side effect  Developed upon Dynamic Symbolic Execution, a practical, powerful recent technique in academia/industry  Good impact to software industry and agencies/defense mission

16. http://pexase.codeplex.com/ https://sites.google.com/site/asergrp/ http://research.microsoft.com/pex Supported by CACC/NSA Related projects supported in part by ARO, NSF, SOSI