Tao Xie Automated Software Engineering Group Department of Computer Science North Carolina State University

 Static Verification  Problem: API properties are not available ▪ E.g., fopen’s return needs to be NULL-CHECK  Solution: mining API properties from client code  Dynamic Verification (a.k.a. Software Testing)

3 Supported by NSF CSR and ARO

4 PARSEWeb [ASE 07] PARSEWeb Source object type & Destination object type Method-invocation sequence MAPO [ECOOP 09] API method  Frequent subsequences of API methods SpotWeb [ASE 08] Framework hotspots/coldspots

5 Detect deviant behavior as bugs in programs Neglected-condition bugs [ASE 09] Exception-handling bugs [ICSE 09] Error-handling bugs [FASE 09] API-sequencing bugs [ESEC/FSE 07]

6 Detect duplicate bug reports [ICSE 08] Identify security bug reports [MSR 10] Mine resource specifications from Javadoc [ASE 09, Best Paper Award, SIGSOFT Distinguished Paper] javax.resource.cci.Connection createInteraction():“Creates an interaction associated with this connection.” getMetaData():“Gets the information on the underlying EIS instance represented through an active connection.” close():“Initiates close of the connection handle at the application level.” Supported by IBM Jazz Award

= ? Outputs Expected Outputs Program + Test inputs Test Oracles  Test Generation  Generating high-quality test inputs (e.g., achieving high code coverage)  Test Oracles  Specifying high-quality test oracles (e.g., guarding against various faults)

 Human  Expensive, incomplete, …  Brute Force  Pairwise, predefined data, etc…  Random:  Cheap, Fast  “It passed a thousand tests” feeling  Dynamic Symbolic Execution: Pex, CUTE,EXE  Automated white-box  Not random – Constraint Solving

Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]== void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!= a!=null && a.Length>0 && a[0]== Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition

 Loops  Fitnex [DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [ASE 09-sp]  Method sequences  MSeqGen [ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [AST 09, ASE 10-sp] Opportunities  Regression testing [ICSE 09-nier]  Developer guidance (cooperative developer testing) Supported by NSF SoD, NSF SHF, NSF CAREER, Microsoft Research Award

 Loops  Fitnex [DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [ASE 09-sp]  Method sequences  MSeqGen [ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [AST 09, ASE 10-sp] Applications  Test network division, Fort Hood, Texas  Test DB app of hand-held medical assistant  Test.NET base

Download counts (20 months) (Feb Oct ) Academic: 17,366 Devlabs: 13,022 Total: 30,388

 Various countries/regions  Software internationalization ▪ Locating constant strings to translate [ICSE 09, FSE 10] ▪ E.g., translating Megamek (a realtime strategy game)  Various programming languages  PL translation ▪ E.g., translating Java to C# [ICSE 10]

 Various types of software  Database applications [ASE 10-sp]  Network/file-system applications [AST 09]  Game applications [ICSE 09]  Cyber-physical systems (power grid, medical device software, …)  Could applications  Social network applications  … Supported by NSF SHF, NSF CAREER

 Various types of quality attributes  Functional correctness  Security (NIST/Fermi Lab collaboration) ▪ Testing/verification of access control policies [WWW 07, ACSAC 08, SIGMETRICS 08, IEEE TC 10] ▪ Testing/verification of firewall policies [SRDS 08/09, LISA 10 Best Student Paper ] ▪ Identification of security bug reports [MSR 10] ▪ Attack generation  Performance ▪ IBM RTP collaboration on Rational Performance Tester  … Supported by NSF CyberTrust, NIST, IBM Faculty Awards