Nikolai Tillmann, Jonathan de Halleux Tao Xie Microsoft Research Univ. Illinois at Urbana-Champaign

 Human  Expensive, incomplete, …  Brute Force  Pairwise, predefined data, etc…  Tool Automation!!

 Pex (released on May 2008)  30,388 download# (20 months, Feb 08-Oct 09)  Active user community: 1,436 forum posts during ~3 years (Oct 08- Nov 11)  Moles (released on Sept 2009)  Shipped with VS 12 as Fakes  “Provide Microsoft Fakes w/ all Visual Studio editions” got 1,457 community votes  Code Digger (released on Oct 2008 for VS 08/10, on Apr 2013 in VS Gallery for VS 12)  22,466 download# (10 months, Apr 13-Jan 14)

 “Great tool to generate unit tests for parameter boundary tests. I like to see it integrated into Visual Studio and the testing features as far as in ReSharper! :)”  “What an awesome tool.. Help us to explore our logic by providing accurate input parameter for each logic branch.. You should try this as one of your ultimate tool :) It really saves a lot of our time to explore every logic branch in our apps..”

1,565,195 clicked 'Ask Pex!'

 NOT Random:  Cheap, Fast  “It passed a thousand tests” feeling  …  But Dynamic Symbolic Execution: e.g., Pex, CUTE,EXE  White box  Constraint Solving

Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]== void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!= a!=null && a.Length>0 && a[0]== Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition

 Pex (released on May 2008):  30,388 download# (20 months, Feb 08-Oct 09)  Active user community: 1,436 forum posts during ~3 years (Oct 08- Nov 11)  Moles (released Sept 2009)  Shipped with VS 12 as Fakes  “Provide Microsoft Fakes w/ all Visual Studio editions” got 1,457 community votes  Code Digger (released on Oct 2008 for VS 08/10, on Apr 2013 in VS Gallery for VS 12)  22,466 download# (10 months, Apr 13-Jan 14) How to make such successful case????

void TestAdd(ArrayList a, object o) { Assume.IsTrue(a!=null); int i = a.Count; a.Add(o); Assert.IsTrue(a[i] == o); } Parameterized Unit Tests Supported by Pex Moles/Fakes Code Digger Pex4Fun/Code Hunt  Surrounding (Moles/Fakes)  Simplifying (Code Digger)  Retargeting (Pex4Fun/Code Hunt)

 Developer/manager: “Who is using your tool?”  Pex team: “Do you want to be the first?”  Developer/manager: “I love your tool but no.” Tool Adoption by (Mass) Target Users Tool Shipping with Visual Studio Macro Perspective Micro Perspective

 Tackle real-world challenges  Select/demo applying Pex on real-world cases (e.g., ResourceReader) beyond textbook examples  Select/demo applying Moles to well address important scenarios (e.g., unit testing of SharePoint code)  Address technical and non-technical barriers for technology adoption in industry (e.g., tool license)  Incremental shipping (e.g., shipping Code Digger before full- fledge Pex)  Find early adopters  Provide quantitative info (reflecting tool’s importance or benefit extent)  Not all downloads are equal! (e.g., those from Fortune 500)

 Developer: “Code digger generates a lot of “\0” strings as input. I can’t find a way to create such a string via my own C# code. Could any one show me a C# snippet? I meant zero terminated string.”  Pex team: “In C#, a \0 in a string does not mean zero- termination. It’s just yet another character in the string (a very simple character where all bits are zero), and you can create as Pex shows the value: “\0”.”  Developer: “Your tool generated “\0””  Pex team: “What did you expect?”  Developer: “Marc.”

 Developer: “Your tool generated a test called Foo001. I don’t like it.”  Pex team: “What did you expect?”  Developer:“Foo_Should_Fail_When_Bar_Is_Negative.”

Object Creation messages suppressed (related to Covana by Xiao et al. [ICSE’11]) Exception Tree View Exploration Tree View Exploration Results View

public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Key observations: with respect to the coverage target  not all paths are equally promising for branch-node flipping  not all branch nodes are equally promising to flip Our solution: –Prefer to flip branch nodes on the most promising paths –Prefer to flip the most promising branch nodes on paths –Fitness function to measure “promising” extents Fitnex by Xie et al. [DSN’09] To avoid local optimal or biases, the fitness-guided strategy is integrated with Pex’s fairness search strategies

 “Simply one mouse click and then everything would work just perfectly”  Often need environment isolation w/ Moles/Fakes or factory methods, …  “One mouse click, a test generation tool would detect all or most kinds of faults in the code under test”  Developer: “Your tool only finds null references.”  Pex team: “Did you write any assertions?”  Developer: “Assertion???”  “I do not need test generation; I already practice unit testing (and/or TDD). Test generation does not fit into the TDD process”

Gathered feedback from target tool users  Directly, e.g., via  MSDN Pex forum, tech support, outreach to MS engineers and.NET user groups  Indirectly, e.g., via  interactions with MS Visual Studio team (a tool vendor to its huge user base)  Motivations of Moles  Refactoring testability issue faced resistance in practice  Observation at Agile 2008: high attention on mock objects and tool supports

 Win-win collaboration model  Win (Ind Lab): longer-term research innovation, man power, research impacts, …  Win (Univ): powerful infrastructure, relevant/important problems in practice, both research and industry impacts, …  Industry-located Collaborations  Faculty visits, e.g., Fitnex, Pex4Fun  Student internships, e.g., FloPSy, DyGen, state cov  Academia-located Collaborations

Academia-located Collaborations  Immediate indirect impacts, e.g.,  Reggae [ASE’09s]  Rex  MSeqGen [FSE’09]  DyGen  Guided Cov [ICSM’10]  state coverage  Long-term indirect impacts, e.g.,  DySy by Csallner et al. [ICSE’08]  Seeker [OOPSLA’11]  Covana [ICSE’11]

 Pex  practice impacts  Moles/Fakes, Code Digger, Pex4Fun/Code Hunt  Lessons in transferring tools  Evolving Dreams  “Chicken and Egg” Problem  Human Factors  Best vs. Worst Cases  Tool Users’ Stereotypical Mindset or Habits  Practitioners’ Voice  Collaboration w/ Academia

 Pex  practice impacts  Moles/Fakes, Code Digger, Pex4Fun/Code Hunt  Lessons in transferring tools  Evolving Dreams  “Chicken and Egg” Problem  Human Factors  Best vs. Worst Cases  Tool Users’ Stereotypical Mindset or Habits  Practitioners’ Voice  Collaboration w/ Academia