1. Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU ASE

2.  Software testing is important  Software errors cost the U.S. economy about $59.5 billion each year (0.6% of the GDP) [NIST 02]  Improving testing infrastructure could save 1/3 cost [NIST 02]  Software testing is costly  Account for even half the total cost of software development [Beizer 90]  Automated testing reduces manual testing effort  Test execution: JUnit, NUnit, xUnit, etc.  Test generation: Pex, AgitarOne, Parasoft Jtest, etc.  Test-behavior checking: Pex, AgitarOne, Parasoft Jtest, etc.

3.  Developer testing  http://www.developertesting.com/ http://www.developertesting.com/  Kent Beck’s 2004 talk on “Future of Developer Testing” http://www.itconversations.com/shows/detail301.html http://www.itconversations.com/shows/detail301.html  This talk focuses on tool automation in developer testing (e.g., unit testing)  Not system testing etc. conducted by testers

4. = ? Outputs Expected Outputs Program + Test inputs Test Oracles

5. = ? Outputs Expected Outputs Program + Test inputs Test Oracles  Test Generation  Generating high-quality test inputs (e.g., achieving high code coverage)

6. = ? Outputs Expected Outputs Program + Test inputs Test Oracles  Test Generation  Generating high-quality test inputs (e.g., achieving high code coverage)  Test Oracles  Specifying high-quality test oracles (e.g., guarding against various faults)

7. var list = new List(); list.Add(item); var count = list.Count; var list = new List(); list.Add(item); var count = list.Count; Assert.AreEqual(1, count); } Assert.AreEqual(1, count); }  Three essential ingredients:  Data  Method Sequence  Assertions void Add() { int item = 3; void Add() { int item = 3;

8.  Which value matters?  Bad choices cause incomplete test suites.  Hard-coded values get stale when product code changes.  Why pick a value if it doesn’t matter? list.Add(3);

9. void Add(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } void Add(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }  Parameterized Unit Test = Unit Test with Parameters  Separation of concerns  Data is generated by a tool  Developer can focus on functional specification [Tillmann&Schulte ESEC/FSE 05]

10.  A Parameterized Unit Test can be read as a universally quantified, conditional axiom. void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData); } void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData); }  string name, string data: name ≠ null ⋀ data ≠ null ⇒ equals( ReadResource(name,WriteResource(name,data)), data)  string name, string data: name ≠ null ⋀ data ≠ null ⇒ equals( ReadResource(name,WriteResource(name,data)), data)

11. Parameterized Unit Tests (PUTs) commonly supported by various test frameworks .NET: Supported by.NET test frameworks  http://www.mbunit.com/  http://www.nunit.org/  …  Java: Supported by JUnit 4.X  http://www.junit.org/ Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex  http://research.microsoft.com/Pex/  Java: Supported by Agitar AgitarOne  http://www.agitar.com/

12.  Human  Expensive, incomplete, …  Brute Force  Pairwise, predefined data, etc…  Random:  Cheap, Fast  “It passed a thousand tests” feeling  Dynamic Symbolic Execution: Pex, CUTE,EXE  Automated white-box  Not random – Constraint Solving

13. Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition

14.  Loops  Fitnex [Xie et al. DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [Li et al. ASE 09-sp]  Method sequences  MSeqGen [Thummalapenta et al. ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [Marri AST 09] Opportunities  Regression testing [Taneja et al. ICSE 09-nier]  Developer guidance (cooperative developer testing)

15.  Loops  Fitnex [Xie et al. DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [Li et al. ASE 09-sp]  Method sequences  MSeqGen [Thummalapenta et al. ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [Marri AST 09] Applications  Test network app at Army division@Fort Hood, Texas  Test DB app of hand-held medical assistant device at FDA

16. Download counts (20 months) (Feb. 2008 - Oct. 2009 ) Academic: 17,366 Devlabs: 13,022 Total: 30,388

17.  Loops  Fitnex [Xie et al. DSN 09]  Generic API functions e.g., RegEx matching IsMatch(s1,regex1)  Reggae [Li et al. ASE 09-sp]  Method sequences  MSeqGen [Thummalapenta et al. ESEC/FSE 09]  Environments e.g., file systems, network, db, …  Parameterized Mock Objects [Marri AST 09] Applications  Test network app at Army division@Fort Hood, Texas  Test DB app of hand-held medical assistant device at FDA

18. There are decision procedures for individual path conditions, but…  Number of potential paths grows exponentially with number of branches  Reachable code not known initially  Without guidance, same loop might be unfolded forever Fitnex search strategy [Xie et al. DSN 09]

19. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Path condition: !(x == 90) ↓ New path condition: (x == 90) ↓ New test input: TestLoop(90, {0}) Path condition: !(x == 90) ↓ New path condition: (x == 90) ↓ New test input: TestLoop(90, {0}) Test input: TestLoop(0, {0}) Test input: TestLoop(0, {0})

20. Path condition: (x == 90) && !(y[0] == 15) ↓ New path condition: (x == 90) && (y[0] == 15) ↓ New test input: TestLoop(90, {15}) Path condition: (x == 90) && !(y[0] == 15) ↓ New path condition: (x == 90) && (y[0] == 15) ↓ New test input: TestLoop(90, {15}) Test input: TestLoop(90, {0}) Test input: TestLoop(90, {0}) public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; }

21. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Test input: TestLoop(90, {15}) Test input: TestLoop(90, {15}) Path condition: (x == 90) && (y[0] == 15) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] == 15) && (x+1 == 110) ↓ New test input: No solution!? Path condition: (x == 90) && (y[0] == 15) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] == 15) && (x+1 == 110) ↓ New test input: No solution!?

22. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Path condition: (x == 90) && (y[0] == 15) && (0 < y.Length) && !(1 < y.Length) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] == 15) && (0 < y.Length) && (1 < y.Length)  Expand array size Path condition: (x == 90) && (y[0] == 15) && (0 < y.Length) && !(1 < y.Length) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] == 15) && (0 < y.Length) && (1 < y.Length)  Expand array size Test input: TestLoop(90, {15}) Test input: TestLoop(90, {15})

23. We can have infinite paths! (both length and number) Manual analysis  need at least 20 loop iterations to cover the target branch Exploring all paths up to 20 loop iterations is practically infeasible: 2 20 paths We can have infinite paths! (both length and number) Manual analysis  need at least 20 loop iterations to cover the target branch Exploring all paths up to 20 loop iterations is practically infeasible: 2 20 paths Test input: TestLoop(90, {15}) Test input: TestLoop(90, {15}) public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; }

24. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Test input: TestLoop(90, {15, 15}) Test input: TestLoop(90, {15, 15}) Our solution:  Prefer to flip nodes on the most promising path  Prefer to flip the most promising nodes on path  Use fitness function as a proxy for promising Our solution:  Prefer to flip nodes on the most promising path  Prefer to flip the most promising nodes on path  Use fitness function as a proxy for promising Key observations: with respect to the coverage target,  not all paths are equally promising for flipping nodes  not all nodes are equally promising to flip Key observations: with respect to the coverage target,  not all paths are equally promising for flipping nodes  not all nodes are equally promising to flip

25.  FF computes fitness value (distance between the current state and the goal state)  Search tries to minimize fitness value [ Tracey et al. 98, Liu at al. 05, …]

26. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Fitness function: |110 – x |

27. (90, {0}) 20 (90, {15}) 19 (90, {15, 0}) 19 (90, {15, 15}) 18 (90, {15, 15, 0}) 18 (90, {15, 15, 15}) 17 (90, {15, 15, 15, 0}) 17 (90, {15, 15, 15, 15}) 16 (90, {15, 15, 15, 15, 0}) 16 (90, {15, 15, 15, 15, 15}) 15 … Fitness Value (x, y) Give preference to flip a node in paths with better fitness values. We still need to address which node to flip on paths … Give preference to flip a node in paths with better fitness values. We still need to address which node to flip on paths … public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Fitness function: |110 – x |

28. Fitness Value public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } (90, {0}) 20 (90, {15})  flip b4 19 (90, {15, 0})  flip b2 19 (90, {15, 15})  flip b4 18 (90, {15, 15, 0})  flip b2 18 (90, {15, 15, 15})  flip b4 17 (90, {15, 15, 15, 0})  flip b2 17 (90, {15, 15, 15, 15})  flip b4 16 (90, {15, 15, 15, 15, 0})  flip b2 16 (90, {15, 15, 15, 15, 15})  flip b4 15 … (x, y) Fitness function: |110 – x | Branch b1: i < y.Length Branch b2: i >= y.Length Branch b3: y[i] == 15 Branch b4: y[i] != 15 Flipping branch node of b4 (b3) gives us average 1 (-1) fitness gain (loss) Flipping branch node of b2 (b1) gives us average 0 (0) fitness gain (loss)

29.  Let p be an already explored path, and n a node on that path, with explored outgoing branch b.  After (successfully) flipping n, we get path p’ that goes to node n, and then continues with a different branch b’.  Define fitness gains as follows, where F(.) is the fitness value of a path.  Set FGain(b) := F(p) – F(p’)  Set FGain(b’) := F(p’) – F(p)  Compute the average fitness gain for each program branch over time

30.  Pex: Automated White-Box Test Generation tool for.NET, based on Dynamic Symbolic Execution  Pex maintains global search frontier  All discovered branch nodes are added to frontier  Frontier may choose next branch node to flip  Fully explored branch nodes are removed from frontier  Pex has a default search frontier  Tries to create diversity across different coverage criteria, e.g. statement coverage, branch coverage, stack traces, etc.  Customizable: Other frontiers can be combined in a fair round-robin scheme

31. We implemented a new search frontier “Fitnex”:  Nodes to flip are prioritized by their composite fitness value: F(p n ) – FGain(b n ), where  p n is path of node n  b n is explored outgoing branch of n  Fitnex always picks node with lowest composite fitness value to flip.  To avoid local optimal or biases, the fitness-guided strategy is combined with Pex’s search strategies

32. A collection of micro-benchmark programs routinely used by the Pex developers to evaluate Pex’s performance, extracted from real, complex C# programs Ranging from string matching like if (value.StartsWith("Hello") && value.EndsWith("World!") && value.Contains(" ")) { … } to a small parser for a Pascal-like language where the target is to create a legal program.

33.  Pex with the Fitnex strategy  Pex without the Fitnex strategy  Pex’s previous default strategy  Random  a strategy where branch nodes to flip are chosen randomly in the already explored execution tree  Iterative Deepening  a strategy where breadth-first search is performed over the execution tree

34. #runs/iterations required to cover the target Pex w/o Fitnex: avg. improvement of factor 1.9 over Random Pex w/ Fitnex: avg. improvement of factor 5.2 over Random

35.  Pex normally uses public methods to configure non-public object fields  Heuristics built-in to deal with common types  User can help if needed void (Foo foo) { if (foo.Value == 123) throw … void (Foo foo) { if (foo.Value == 123) throw … [PexFactoryMethod] Foo Create(Bar bar) { return new Foo(bar); } [PexFactoryMethod] Foo Create(Bar bar) { return new Foo(bar); }

36.  A graph example from QuickGraph library 36 interface IGraph { /* Adds given vertex to the graph */ void AddVertex(IVertex v); /* Creates a new vertex and adds it to the graph */ IVertex AddVertex(); /* Adds an edge to the graph. Both vertices should already exist in the graph */ IEdge AddEdge(IVertex v1, Ivertex v2); } interface IGraph { /* Adds given vertex to the graph */ void AddVertex(IVertex v); /* Creates a new vertex and adds it to the graph */ IVertex AddVertex(); /* Adds an edge to the graph. Both vertices should already exist in the graph */ IEdge AddEdge(IVertex v1, Ivertex v2); }

37.  Desired object state for reaching targets 1 and 2: graph object should contain vertices and edges Class SortAlgorithm { IGraph graph; public SortAlgorithm(IGraph graph) { this.graph = graph; } public void Compute (IVertex s) { foreach(IVertex u in graph.Vertices) { // Target 1 } foreach(IEdge e in graph.Edges) { //Target 2 } Class SortAlgorithm { IGraph graph; public SortAlgorithm(IGraph graph) { this.graph = graph; } public void Compute (IVertex s) { foreach(IVertex u in graph.Vertices) { // Target 1 } foreach(IEdge e in graph.Edges) { //Target 2 } method sequence

38. VertexAndEdgeProvider v0 = new VertexAndEdgeProvider(); Boolean v1 = false; BidirectionalGraph v2 = new BidirectionalGraph((IVertexAndEdgeProvider)v0, v1); IVertex v3 = v2.AddVertex(); IVertex v4 = v0.ProvideVertex(); IEdge v15 = v2.AddEdge(v3, v4); VertexAndEdgeProvider v0 = new VertexAndEdgeProvider(); Boolean v1 = false; BidirectionalGraph v2 = new BidirectionalGraph((IVertexAndEdgeProvider)v0, v1); IVertex v3 = v2.AddVertex(); IVertex v4 = v0.ProvideVertex(); IEdge v15 = v2.AddEdge(v3, v4); Achieved 31.82% (7 of 22) branch coverage Reason for low coverage: Not able to generate graph with vertices and edges Applying Randoop, a random testing approach that constructs test inputs by randomly selecting method calls Example sequence generated by Randoop v4 not in the graph, so edge cannot be added to graph.

39. VertexAndEdgeProvider v0; bool bVal; IGraph ag = new AdjacencyGraph(v0, bVal); IVertex source = ag.AddVertex(); IVertex target = ag.AddVertex(); IVertex vertex3 = ag.AdVertex(); IEdge edg1 = ag.AddEdge(source, target); IEdge edg2 = ag.AddEdge(target, vertex3); IEdge edg3 = ag.AddEdge(source, vertex3); VertexAndEdgeProvider v0; bool bVal; IGraph ag = new AdjacencyGraph(v0, bVal); IVertex source = ag.AddVertex(); IVertex target = ag.AddVertex(); IVertex vertex3 = ag.AdVertex(); IEdge edg1 = ag.AddEdge(source, target); IEdge edg2 = ag.AddEdge(target, vertex3); IEdge edg3 = ag.AddEdge(source, vertex3); Use mined sequences to assist Randoop and Pex Both Randoop and Pex achieved 86.40% (19 of 22) branch coverage with assistance from MSeqGen Mine sequences from existing code bases Reuse mined sequences for achieving desired object states A Mined sequence from an existing codebase Graph object includes both vertices and edges

40. Existing codebases are often large and complete analysis is expensive   Search and analyze only relevant portions Concrete values in mined sequences may be different from desired values   Replace concrete values with symbolic values and use dynamic symbolic execution Extracted sequences individually may not be sufficient to achieve desired object states   Combine extracted sequences to generate new sequences

41. Problem: Existing code bases are often large and complete analysis is expensive Solution: Use keyword search for identifying relevant method bodies using target classes Analyze only those relevant method bodies Target classes: System.Collections.Hashtable QuickGraph.Algorithms.TSAlgorithm Keywords: Hashtable, TSAlgorithm Target classes: System.Collections.Hashtable QuickGraph.Algorithms.TSAlgorithm Keywords: Hashtable, TSAlgorithm Shortnames of target classes are used as keywords

42. Problem: Concrete values in mined sequences are different from desired values to achieve target states Solution: Generalize sequences by replacing concrete values with symbolic values Class A { int f1 { set; get; } int f2 { set; get; } void CoverMe() { if (f1 != 10) return; if (f2 > 25) throw new Exception(“bug”); } Method Under Test A obj = new A(); obj.setF1(14); obj.setF2(-10); obj.CoverMe(); Mined Sequence for A Sequence cannot help in exposing bug since desired values are f1=10 and f2>25

43. Replace concrete values 14 and -10 with symbolic values X1 and X2 A obj = new A(); obj.setF1(14); obj.setF2(-10); obj.CoverMe(); Mined Sequence for A int x1 = *, x2 = *; A obj = new A(); obj.setF1(x1); obj.setF2(x2); obj.CoverMe(); Generalized Sequence for A Use DSE for generating desired values for X1 and X2 DSE explores CoverMe method and generates desired values (X1 = 10 and X2 = 35)

44. 44  Randoop  Without assistance from MSeqGen: achieved 32% branch coverage  achieved 86% branch coverage  In evaluation, help Randoop achieve 8.7% (maximum 20%) higher branch coverage  Pex  Without assistance from MSeqGen: achieved 45% branch coverage  achieved 86% branch coverage  In evaluation, help Pex achieve 17.4% (maximum 22.5%) higher branch coverage

45.  Write assertions and Pex will try to break them  Without assertions, Pex can only find violations of runtime contracts causing NullReferenceException, IndexOutOfRangeException, etc.  Assertions leveraged in product and test code  Pex can leverage Code Contracts

46. = ? Outputs Expected Outputs Program + Test inputs Test Oracles  Test Generation  Test inputs for PUT generated by tools (e.g., Pex)  Fitnex: guided exploration of paths [DSN 09]  MSeqGen: exploiting real-usage sequences [ESEC/FSE 09]  Test Oracles  Assertions in PUT specified by developers Division of Labors

47. http://research.microsoft.com/pex http://pexase.codeplex.com/ https://sites.google.com/site/asergrp/

48.  http://research.microsoft.com/en-us/projects/contracts/ http://research.microsoft.com/en-us/projects/contracts/  Library to state preconditions, postconditions, invariants  Supported by two tools:  Static Checker  Rewriter: turns Code Contracts into runtime checks  Pex analyses the runtime checks  Contracts act as Test Oracle  Pex may find counter examples for contracts  Missing Contracts may be suggested

49. Class invariant specification: public class ArrayList { private Object[] _items; private int _size;... [ContractInvariantMethod] // attribute comes with Contracts protected void Invariant() { Contract.Invariant(this._items != null); Contract.Invariant(this._size >= 0); Contract.Invariant(this._items.Length >= this._size); }

51.  Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small.  Integration test: exercises large portions of a system.  Observation: Integration tests are often “sold” as unit tests  White-box test generation does not scale well to integration test scenarios.  Possible solution: Introduce abstraction layers, and mock components not under test

52. AppendFormat(null, “{0} {1}!”, “Hello”, “World”);  “Hello World!”.Net Implementation: public StringBuilder AppendFormat( IFormatProvider provider, char[] chars, params object[] args) { if (chars == null || args == null) throw new ArgumentNullException(…); int pos = 0; int len = chars.Length; char ch = '\x0'; ICustomFormatter cf = null; if (provider != null) cf = (ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter)); …

53.  Introduce a mock class which implements the interface.  Write assertions over expected inputs, provide concrete outputs public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { Assert.IsTrue(formatType != null); return new MCustomFormatter(); } }  Problems:  Costly to write detailed behavior by example  How many and which mock objects do we need to write?

54.  Introduce a mock class which implements the interface.  Let an oracle provide the behavior of the mock methods. public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { … object o = call.ChooseResult (); return o; } }  Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated! 54

55.  Chosen values can be shaped by assumptions public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { … object o = call.ChooseResult (); PexAssume.IsTrue(o is ICustomFormatter); return o; } }  (Note: Assertions and assumptions are “reversed” when compared to parameterized unit tests.) 55

56.  Choices to build parameterized models class PFileSystem : IFileSystem { // cached choices PexChosenIndexedValue files; string ReadFile(string name) { var content = this.files[name]; if (content == null) throw new FileNotFoundException(); return content; }} class PFileSystem : IFileSystem { // cached choices PexChosenIndexedValue files; string ReadFile(string name) { var content = this.files[name]; if (content == null) throw new FileNotFoundException(); return content; }}

57. 57  Subjects:  QuickGraph  Facebook  Research Questions:  RQ1: Can our approach assist Randoop (random testing tool) in achieving higher code coverages?  RQ2: Can our approach assist Pex (DSE-based testing tool) in achieving higher code coverages? MSeqGen Evaluation

58. 58 RQ1: Assisting Randoop

59. 59 RQ2: Assisting Pex  Legend:  #c: number of classes  P: branch coverage achieved by Pex  P + M: branch coverage achieved by Pex and MSeqGen

60. void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); } void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); }  Assumptions and assertions induce branches  Executions which cause assumption violations are ignored, not reported as errors or test cases

61.  How to test this code? (Actual code from.NET base class libraries) 61

63. [PexClass, TestClass] [PexAllowedException(typeof(ArgumentNullException))] [PexAllowedException(typeof(ArgumentException))] [PexAllowedException(typeof(FormatException))] [PexAllowedException(typeof(BadImageFormatException))] [PexAllowedException(typeof(IOException))] [PexAllowedException(typeof(NotSupportedException))] public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } [PexClass, TestClass] [PexAllowedException(typeof(ArgumentNullException))] [PexAllowedException(typeof(ArgumentException))] [PexAllowedException(typeof(FormatException))] [PexAllowedException(typeof(BadImageFormatException))] [PexAllowedException(typeof(IOException))] [PexAllowedException(typeof(NotSupportedException))] public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } } }

64.  Exploration of constructor/mutator method sequences  Testing with class invariants

65.  Write class invariant as boolean-valued parameterless method  Refers to private fields  Must be placed in implementation code  Exploration of valid states by setting public/private fields  May include states that are not reachable

66. Class invariant specification: public class ArrayList { private Object[] _items; private int _size;... [ContractInvariantMethod] // attribute comes with Contracts protected void Invariant() { Contract.Invariant(this._items != null); Contract.Invariant(this._size >= 0); Contract.Invariant(this._items.Length >= this._size); }

67. PUT: [PexMethod] public void ArrayListTest(ArrayList al, object o) { int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o); }

68. Generated Test: [TestMethod] public void Add01() { object[] os = new object[0]; // create raw instance ArrayList arrayList = PexInvariant.CreateInstance (); // set private field via reflection PexInvariant.SetField (arrayList, "_items", os); PexInvariant.SetField (arrayList, "_size", 0); // invoke invariant method via reflection PexInvariant.CheckInvariant(arrayList); // call to PUT ArrayListTest(arrayList, null); }