Tao Xie (North Carolina State University) Peli de Halleux, Nikolai Tillmann, Wolfram Schulte (Microsoft Research)

var list = new List(); list.Add(item); var list = new List(); list.Add(item); Assert.AreEqual(1, list.Count); } Assert.AreEqual(1, list.Count); }  Three essential ingredients (e.g., testing List.Add(int)):  Data  Method Sequence  Assertions void TestAdd() { int item = 3; void TestAdd() { int item = 3;

void TestAdd(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } void TestAdd(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }  Parameterized Unit Test = Unit Test with Parameters  Separation of concerns  Data is generated by a tool (e.g., Pex)  Developer can focus on functional (algebraic) spec [Tillmann&Schulte ESEC/FSE 05]

void TestAdd() { int item = 3; var list = new List(); list.Add(item); Assert.AreEqual(1, list.Count); } void TestAdd() { int item = 3; var list = new List(); list.Add(item); Assert.AreEqual(1, list.Count); } void TestAdd(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } void TestAdd(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } Required skill set Abstraction Generalization, Experimentation, Comprehension, …

 Graduate software testing course (CSC 712) at North Carolina State University  2008, 2009, 2010 Fall semesters (20, 18, and 27 students)  Lectures: two 75-minute lectures per week  Mons: foundation/coverage criteria instructed based on “Introduction to Software Testing” by Ammann and Offutt  Weds: practical/hands-on testing techniques/tools, with Pex, Code Contracts, and Nmodel  Assignments/projects  Test generalization + writing new PUTs

Using industrial-strength tools and technologies (e.g., Pex)  reduce the “debugging” overhead imposed on both the students and teaching staff  give students experiences that they could immediately benefit from when they took on their industrial jobs Over using academic research prototypes  lack support for dealing with various types of real-world code features  encounter many faults in prototypes  lack timely technical support or fixing of reported faults

 One/half-day tutorials (10-25 practitioners)  inside MS, e.g., internal training of MS developers  outside MS, e.g., invited tutorials at.NET user groups  Inside Microsoft (in a training lab at Microsoft)  Attendees using lab computers installed w/ Pex  Slides + demos + intensive hands-on exercises  Outside Microsoft (in a meeting room)  Attendees often not having computers installed w/ Pex  Slides + demos

 Set realistic expectations right away  Tools/technologies have limitations, not “panacea”  Need teach developers about limitations  Don’t try to change deeply ingrained beliefs  … at least not all at once  Example: Believers in TDD (Test-Driven-Development)  Emphasize how new techniques relates to existing ones, how they complement

 Trainees with different incentives of learning (inducing different emphasis on technology adoption)  Univ: earn good course grades, learning various skills  Industry: “come and watch”, learning what is going on  Training with different durations  Univ: one semester  Industry: full/half day

 Accessibility  Visual Studio vs. Browser  Active learning  Doing with feedback  Contributing back  Social aspects  Ranking  Livefeed  Competitive environment

 us/vstudio/bb aspx us/vstudio/bb aspx

 Testing real-world code (not just “toy” code examples)  Different projects for different teams  Allow learning from each other in real-world complications  Effort focused on testing, not code development  Fully exploit limited course period time  Student team selects a real open-source project to test  Equipped with existing conventional unit tests  Project milestones  Midterm: test generalization; final: new PUT writing  Drawback  Not “real” develop testing!