1. Tao Xie Automated Software Engineering Group Department of Computer Science North Carolina State University https://sites.google.com/site/asergrp/

2.  Static Verification  Problem: API properties are not available ▪ E.g., fopen’s return needs to be NULL-CHECK  Solution: mining API properties from client code  Dynamic Verification (a.k.a. Software Testing)

3. 3 Supported by NSF CSR and ARO

4. 4 PARSEWeb [ASE 07] PARSEWeb Source object type & Destination object type Method-invocation sequence MAPO [ECOOP 09] API method  Frequent subsequences of API methods SpotWeb [ASE 08] Framework hotspots/coldspots

5. 5 Detect deviant behavior as bugs in programs Neglected-condition bugs [ASE 09] Exception-handling bugs [ICSE 09] Error-handling bugs [FASE 09] API-sequencing bugs [ESEC/FSE 07]

6. 6 Detect duplicate bug reports [ICSE 08] Identify security bug reports [MSR 10] Mine resource specifications from Javadoc [ASE 09, Best Paper Award, SIGSOFT Distinguished Paper] javax.resource.cci.Connection createInteraction():“Creates an interaction associated with this connection.” getMetaData():“Gets the information on the underlying EIS instance represented through an active connection.” close():“Initiates close of the connection handle at the application level.” Supported by IBM Jazz Award

7. = ? Outputs Expected Outputs Program + Test inputs Test Oracles  Test Generation  Generating high-quality test inputs (e.g., achieving high code coverage)  Test Oracles  Specifying high-quality test oracles (e.g., guarding against various faults)

8.  Human  Expensive, incomplete, …  Brute Force  Pairwise, predefined data, etc…  Random:  Cheap, Fast  “It passed a thousand tests” feeling  Dynamic Symbolic Execution: Pex, CUTE,EXE  Automated white-box  Not random – Constraint Solving

9. Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition

10.  Loops  Fitnex [DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [ASE 09-sp]  Method sequences  MSeqGen [ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [AST 09, ASE 10-sp] Opportunities  Regression testing [ICSE 09-nier]  Developer guidance (cooperative developer testing) Supported by NSF SoD, NSF SHF, NSF CAREER, Microsoft Research Award

11.  Loops  Fitnex [DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [ASE 09-sp]  Method sequences  MSeqGen [ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [AST 09, ASE 10-sp] Applications  Test network app @Army division, Fort Hood, Texas  Test DB app of hand-held medical assistant device@FDA  Test.NET base libraries @Microsoft

12. Download counts (20 months) (Feb. 2008 - Oct. 2009 ) Academic: 17,366 Devlabs: 13,022 Total: 30,388

14.  Various countries/regions  Software internationalization ▪ Locating constant strings to translate [ICSE 09, FSE 10] ▪ E.g., translating Megamek (a realtime strategy game)  Various programming languages  PL translation ▪ E.g., translating Java to C# [ICSE 10]

15.  Various types of software  Database applications [ASE 10-sp]  Network/file-system applications [AST 09]  Game applications [ICSE 09]  Cyber-physical systems (power grid, medical device software, …)  Could applications  Social network applications  … Supported by NSF SHF, NSF CAREER

16.  Various types of quality attributes  Functional correctness  Security (NIST/Fermi Lab collaboration) ▪ Testing/verification of access control policies [WWW 07, ACSAC 08, SIGMETRICS 08, IEEE TC 10] ▪ Testing/verification of firewall policies [SRDS 08/09, LISA 10 Best Student Paper ] ▪ Identification of security bug reports [MSR 10] ▪ Attack generation  Performance ▪ IBM RTP collaboration on Rational Performance Tester  … Supported by NSF CyberTrust, NIST, IBM Faculty Awards

17. http://people.engr.ncsu.edu/txie/ https://sites.google.com/site/asergrp/