1. Separation of Concerns Tao Xie Peking University, China North Carolina State University, USA In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU ASE

2.  Separation of Concerns (SoC)  Aspect-Oriented Software Development (AOSD)  “Killer Apps” for SoC/AOSD?  Logging, Security Checking, …  Any Other Application Scenarios in Practice?  Tool Support for Software Testing?

3. = ? Outputs Expected Outputs Program + Test inputs Test Oracles

4. = ? Outputs Expected Outputs Program + Test inputs Test Oracles  Test Generation  Generating high-quality test inputs (e.g., achieving high code coverage)

5. = ? Outputs Expected Outputs Program + Test inputs Test Oracles  Test Generation  Generating high-quality test inputs (e.g., achieving high code coverage)  Test Oracles  Specifying high-quality test oracles (e.g., guarding against various faults)

6. var list = new ArrayList(); list.Add(item); var list = new ArrayList(); list.Add(item); Assert.AreEqual(1, list.Count); } Assert.AreEqual(1, list.Count); }  Three essential ingredients:  Data  Method Sequence  Assertions void TestAdd() { int item = 3; void TestAdd() { int item = 3;

7.  Parameterized Unit Testing  Separate Test Generation and Behavior Specification  Test Generalization  Localize Crosscutting Concerns of Behavior Under Test  Object Creation with Factory Method  Localize Crosscutting Concerns of Object Creation  Environment Isolation with Moles  Localize Crosscutting Concerns of Environment Dependency

8. void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }  Parameterized Unit Test = Unit Test with Parameters  Separation of concerns  Data is generated by a tool  Developer can focus on functional specification [Tillmann&Schulte ESEC/FSE 05]

9. Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void TestMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } void TestMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition

10. Download counts (20 months) (Feb. 2008 - Oct. 2009 ) Academic: 17,366 Devlabs: 13,022 Total: 30,388 http://research.microsoft.com/projects/pex/ Dynamic Symbolic Execution [Tillmann&de Halleux TAP 08]

11. 669,584 clicked 'Ask Pex!‘ since 2010 summer Try it at http://www.pexforfun.com/

12. Parameterized Unit Tests (PUTs) commonly supported by various test frameworks .NET: Supported by.NET test frameworks  http://www.mbunit.com/  http://www.nunit.org/  …  Java: Supported by JUnit 4.X  http://www.junit.org/ Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex  http://research.microsoft.com/Pex/  Java: Supported by Agitar AgitarOne  http://www.agitar.com/

13.  Benefits over CUTs  Help describe behaviours for all test arguments  Address two main issues with CUTs  Missing test data (that would exercise important behavior) ▪ Low fault-detection capability  Redundant test data and scenarios (that exercises the same behaviour) ▪ Redundant unit tests

14.  Parameterized Unit Testing  Separate Test Generation and Behavior Specification  Test Generalization  Localize Crosscutting Concerns of Behavior Under Test  Object Creation with Factory Method  Localize Crosscutting Concerns of Object Creation  Environment Isolation with Moles  Localize Crosscutting Concerns of Environment Dependency

15. public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } Three CUTs public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count()); }  CUT1 and CUT2 exercise push with different test data  CUT3 exercises push when stack is not empty Two main issues with CUTs:  Fault-detection capability: undetected defect where things go wrong when passing a negative value to push  Redundant tests: CUT2 is redundant with respect to CUT1

16. 16 public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } Three CUTs public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count()); }  No need to describe test data Generated automatically  Single PUT replaces multiple CUTs With reduced size of test code public void PUT(int[] elem) { Assume.IsTrue(elem != null); IntStack stk = new IntStack(); for(int i in elem) stk.push(elem); Assert.AreEqual(elem.Length, stk.count()); } An equivalent PUT [ Thummalapenta et al. FASE 11]

17.  Refactoring clone instances (crosscutting concerns) to a single copy (aspects)  Need pointcuts to specify join points, i.e., which original code locations to weave the aspects  Test generalization is different  Original code locations (old CUTs) are thrown away  PUTs + Pex generate new CUTs  No need of join points or pointcuts

18.  Parameterized Unit Testing  Separate Test Generation and Behavior Specification  Test Generalization  Localize Crosscutting Concerns of Behavior Under Test  Object Creation with Factory Method  Localize Crosscutting Concerns of Object Creation  Environment Isolation with Moles  Localize Crosscutting Concerns of Environment Dependency

19. void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

20.  Pex uses public methods to configure non- public object fields  Heuristics built-in to deal with common types  User can help if needed [PexFactoryMethod(typeof(ArrayList))] public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list; } [PexFactoryMethod(typeof(ArrayList))] public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list; }

21. [PexFactoryMethod(typeof(ArrayList))] public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list; } [PexFactoryMethod(typeof(ArrayList))] public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list; } void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

22.  Parameterized Unit Testing  Separate Test Generation and Behavior Specification  Test Generalization  Localize Crosscutting Concerns of Behavior Under Test  Object Creation with Factory Method  Localize Crosscutting Concerns of Object Creation  Environment Isolation with Moles  Localize Crosscutting Concerns of Environment Dependency

23.  Components depend on other components  Hidden Integration Tests void FileExistsTest() { File.Write(“foo.txt”, “”); var result = IsFileEmpty(“foo.txt”) Assert.IsTrue(result); } void FileExistsTest() { File.Write(“foo.txt”, “”); var result = IsFileEmpty(“foo.txt”) Assert.IsTrue(result); } bool IsFileEmpty(string file) { var content = File.ReadAllText(file); return content.Length == 0; } bool IsFileEmpty(string file) { var content = File.ReadAllText(file); return content.Length == 0; } File.ReadAllText(file); File.Write(“foo.txt”, “”);

24.  Slow, complicated setup, non-deterministic tests  Solution: Replace by Simpler Environment (“mocking”)  Testable Design: Abstraction layer + Dependency Injection + Mocks for testing  Simply use virtual methods (i.e., overridenable)  Hard-coded Design: No abstraction layer, static methods, sealed types (i.e., not inheritable).

25.  Replace any.NET method with a Delegate  similar to a C++ function pointer  Method can be overridden? Use Stubs  Interfaces  Abstract classes  Virtual methods in non-sealed types  Method cannot be overridden? Use Moles

26.  Existing external components cannot be re- factored  SharePoint, ASP.NET, …  Need mechanism to stub non-virtual methods  Static methods, methods in sealed types, constructors

27.  Method redirected to user delegate, i.e. moled  Requires code instrumentation bool result = IsFileEmpty(“foo.txt”); Assert.IsTrue(result); bool result = IsFileEmpty(“foo.txt”); Assert.IsTrue(result); MFile.ReadAllTextString = file => “”; expression lambda : (input parameters) => expression MFile.ReadAllTextString = file => “”; expression lambda : (input parameters) => expression [de Halleux&Tillmann TOOLS 10]

28. File.ReadAllText(string name) { } File.ReadAllText(string name) { } mscorlib.dll File.ReadAllText(string name) { var d = GetDetour(); if (d != null) return d(); } File.ReadAllText(string name) { var d = GetDetour(); if (d != null) return d(); } push ecx push edx push eax push ecx push edx push eax.NET Runtime Just In Time Compiler.NET Runtime Just In Time Compiler ExtendedReflection

29.  Automated White box Analysis does not ‘understand’ the environment  Isolate Code using Stubs and Moles if (DateTime.Now == new DateTime(2000,1,1)) throw new Y2KException(); if (DateTime.Now == new DateTime(2000,1,1)) throw new Y2KException(); DateTime.Now ??? public void TestY2k(DateTime dt) { MDateTime.NowGet = () => dt... } public void TestY2k(DateTime dt) { MDateTime.NowGet = () => dt... } MDateTime.NowGet = () => dt DateTime.Now == dt

30.  Parameterized Unit Testing  Separate Test Generation and Behavior Specification  Test Generalization  Localize Crosscutting Concerns of Behavior Under Test  Object Creation with Factory Method  Localize Crosscutting Concerns of Object Creation  Environment Isolation with Moles  Localize Crosscutting Concerns of Environment Dependency Application Scenarios of Separation of Concerns in Practice

31. Questions ? https://sites.google.com/site/asergrp/ http://research.microsoft.com/projects/pex/ http://www.pexforfun.com/ http://research.microsoft.com/en-us/projects/pex/community.aspx#publications http://pexase.codeplex.com/