1. Automated Developer Testing: Achievements and Challenges
Tao Xie
North Carolina State University
In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Research and ASE

2. Why Automate Testing? Software testing is important
Software errors cost the U.S. economy about $59.5 billion each year (0.6% of the GDP) [NIST 02]
Improving testing infrastructure could save 1/3 cost [NIST 02]
Software testing is costly
Account for even half the total cost of software development [Beizer 90]
Automated testing reduces manual testing effort
Test execution: JUnit, NUnit, xUnit, etc.
Test generation: Pex, AgitarOne, Parasoft Jtest, etc.
Test-behavior checking: Pex, AgitarOne, Parasoft Jtest, etc.

3. Automation in Developer Testing
Kent Beck’s 2004 talk on “Future of Developer Testing”
This talk focuses on tool automation in developer testing (e.g., unit testing)
Not system testing etc. conducted by testers

4. Software Testing Setup= ? +
Expected Outputs
Test inputs
Program
Outputs
Test Oracles

5. Software Testing Problems= ? +
Expected Outputs
Test inputs
Program
Outputs
Test Oracles
Test Generation
Generating high-quality test inputs (e.g., achieving high code coverage)

6. Software Testing Problems= ? +
Expected Outputs
Test inputs
Program
Outputs
Test Oracles
Test Generation
Generating high-quality test inputs (e.g., achieving high code coverage)
Test Oracles
Specifying high-quality test oracles (e.g., guarding against various faults)

7. The Recipe of Unit Testing
Three essential ingredients:
Data
Method Sequence
Assertions
void TestAdd() {
int item = 3;
var list = new List();
list.Add(item);
Assert.AreEqual(1, list.Count); }

8. Parameterized Unit Testing
[Tillmann&Schulte ESEC/FSE 05]
Parameterized Unit Test =
Unit Test with Parameters
Separation of concerns
Data is generated by a tool
Developer can focus on functional specification
void TestAdd(List list, int item) {
Assume.IsTrue(list != null);
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count); }

9. Parameterized Unit Tests are Algebraic Specifications
A Parameterized Unit Test can be read as a universally quantified, conditional axiom.
void TestReadWrite(Res r, string name, string data) { Assume.IsTrue(r!=null & name!=null && data!=null);
r.WriteResource(name, data); Assert.AreEqual(r.ReadResource(name), data); }
 string name, string data, Res r:
r ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒
equals(
ReadResource(WriteResource(r, name, data).state, name),
data)

10. Parameterized Unit Testing Getting Popular
Parameterized Unit Tests (PUTs) commonly supported by various test frameworks
.NET: Supported by .NET test frameworks …
Java: Supported by JUnit 4.X
Generating test inputs for PUTs supported by tools
.NET: Supported by Microsoft Research Pex
Java: Supported by Agitar AgitarOne

11. Test Generation Human Brute Force Random:
Expensive, incomplete, …
Brute Force
Pairwise, predefined data, etc…
Random:
Cheap, Fast
“It passed a thousand tests” feeling
Dynamic Symbolic Execution: Pex, CUTE,EXE
Automated white-box
Not random – Constraint Solving

12. Dynamic Symbolic Execution
Choose next path
Code to generate inputs for:
Solve
Execute&Monitor
void CoverMe(int[] a) {
if (a == null) return;
if (a.Length > 0)
if (a[0] == )
throw new Exception("bug"); }
Constraints to solve a!=null a!=null && a.Length>0 a.Length>0 && a[0]==
Data null {} {0} {123…}
Observed constraints a==null a!=null && !(a.Length>0) a.Length>0 && a[0]!= a[0]==
Negated condition
a==null F T
a.Length>0 F T
Done: There is no path left.
a[0]==123… F T

13. Challenges of DSE Loops
Fitnex [Xie et al. DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1)
Reggae [Li et al. ASE 09-sp]
Method sequences
MSeqGen [Thummalapenta et al. ESEC/FSE 09]
Environments e.g., file systems, network, db, …
Parameterized Mock Objects [Marri et al. AST 09]
Opportunities
Regression testing [Taneja et al. ICSE 09-nier]
Developer guidance (cooperative developer testing)

14. NCSU Industry Tech Transfer
Loops
Fitnex [Xie et al. DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1)
Reggae [Li et al. ASE 09-sp]
Method sequences
MSeqGen [Thummalapenta et al. ESEC/FSE 09]
Environments e.g., file systems, network, db, …
Parameterized Mock Objects [Marri et al. AST 09]
Applications
Test network app at Army Hood, Texas
Test DB app of hand-held medical assistant device at FDA

15. Pex on MSDN DevLabs Incubation Project for Visual Studio
11/11/2018 5:55 AM
Pex on MSDN DevLabs Incubation Project for Visual Studio
Download counts (20 months) (Feb Oct )
Academic: 17,366
Devlabs: ,022
Total: ,388
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. NCSU Industry Tech Transfer
Loops
Fitnex [Xie et al. DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1)
Reggae [Li et al. ASE 09-sp]
Method sequences
MSeqGen [Thummalapenta et al. ESEC/FSE 09]
Environments e.g., file systems, network, db, …
Parameterized Mock Objects [Marri AST 09]
Applications
Test network app at Army Hood, Texas
Test DB app of hand-held medical assistant device at FDA

17. Explosion of Search Space
There are decision procedures for individual path conditions, but…
Number of potential paths grows exponentially with number of branches
Without guidance, same loop might be unfolded forever
Fitnex search strategy
[Xie et al. DSN 09]

18. DSE Example
public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false;
Test input:
TestLoop(0, {0})
Path condition:
!(x == 90) ↓
New path condition:
(x == 90)
New test input:
TestLoop(90, {0})

19. DSE Example
public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false;
Test input:
TestLoop(90, {0})
Path condition:
(x == 90) && !(y[0] == 15) ↓
New path condition:
(x == 90) && (y[0] == 15)
New test input:
TestLoop(90, {15})

20. Challenge in DSE
public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false;
Test input:
TestLoop(90, {15})
Path condition:
(x == 90) && (y[0] == 15) && !(x+1 == 110) ↓
New path condition:
(x == 90) && (y[0] == 15) && (x+1 == 110)
New test input:
No solution!?

21. A Closer Look
public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false;
Test input:
TestLoop(90, {15})
Path condition:
(x == 90) && (y[0] == 15) && (0 < y.Length)
&& !(1 < y.Length)
&& !(x+1 == 110) ↓
New path condition:
&& (1 < y.Length)
 Expand array size

22. A Closer Look We can have infinite paths! (both length and number)
public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false;
Test input:
TestLoop(90, {15})
We can have infinite paths! (both length and number)
Manual analysis  need at least 20 loop iterations to cover the target branch
Exploring all paths up to 20 loop iterations is practically infeasible: 220 paths

23. Fitnex: Fitness-Guided Exploration
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true; }
return false;
Test input:
TestLoop(90, {15, 15})
Key observations: with respect to the coverage target,
not all paths are equally promising for flipping nodes
not all nodes are equally promising to flip
Our solution:
Prefer to flip nodes on the most promising path
Prefer to flip the most promising nodes on path
Use fitness function as a proxy for promising

24. Fitness Function
FF computes fitness value (distance between the current state and the goal state)
Search tries to minimize fitness value
[Tracey et al. 98, Liu at al. 05, …]

25. Fitness Function for (x == 110)
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true; }
return false;
Fitness function: |110 – x |

26. Compute Fitness Values for Paths
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true; }
return false;
(x, y)
(90, {0})
(90, {15})
(90, {15, 0})
(90, {15, 15})
(90, {15, 15, 0})
(90, {15, 15, 15})
(90, {15, 15, 15, 0})
(90, {15, 15, 15, 15})
(90, {15, 15, 15, 15, 0})
(90, {15, 15, 15, 15, 15}) …
Fitness function: |110 – x |
Give preference to flip a node in paths with better fitness values.
We still need to address which node to flip on paths …

27. Compute Fitness Gains for Branches
Fitness Value
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true; }
return false;
(x, y)
(90, {0})
(90, {15})  flip b
(90, {15, 0})  flip b
(90, {15, 15})  flip b
(90, {15, 15, 0})  flip b
(90, {15, 15, 15})  flip b
(90, {15, 15, 15, 0})  flip b2 17
(90, {15, 15, 15, 15})  flip b4 16
(90, {15, 15, 15, 15, 0})  flip b2 16
(90, {15, 15, 15, 15, 15})  flip b …
Fitness function: |110 – x |
Branch b1: i < y.Length
Branch b2: i >= y.Length
Branch b3: y[i] == 15
Branch b4: y[i] != 15
Flipping branch node of b4 (b3) gives us average 1 (-1) fitness gain (loss)
Flipping branch node of b2 (b1) gives us average 0 (0) fitness gain (loss)

28. Compute Fitness Gains for Branches
FGain(b) := F(p) – F(p’)
FGain(b’) := F(p’) – F(p)
Compute the average fitness gain for each program branch over time p p’ n n b b’ …. ….
F(p) is the fitness value of p
F(p’) is the fitness value of p’

29. Implementation in Pex Pex maintains global search frontier
All discovered branch nodes are added to frontier
Frontier may choose next branch node to flip
Fully explored branch nodes are removed from frontier
Pex has a default search frontier
It tries to create diversity across different coverage criteria
Frontiers can be combined in a fair round-robin scheme

30. Implementation in Pex We implemented a new search frontier “Fitnex”:
Nodes to flip are prioritized by their composite fitness value:
F(pn) – FGain(bn),
where
pn is path of node n
bn is explored outgoing branch of n
Fitnex always picks node with lowest composite fitness value to flip.
To avoid local optimal or biases, the fitness-guided strategy is combined with Pex’s search strategies

31. Evaluation Subjects
A collection of micro-benchmark programs routinely used by the Pex developers to evaluate Pex’s performance, extracted from real, complex C# programs
Ranging from string matching like
if (value.StartsWith("Hello") &&
value.EndsWith("World!") &&
value.Contains(" ")) { … }
to a small parser for a Pascal-like language where the target is to create a legal program.

32. Search Strategies Under Comparison
Pex with the Fitnex strategy
Pex without the Fitnex strategy
Pex’s previous default strategy
Random
a strategy where branch nodes to flip are chosen randomly in the already explored execution tree
Iterative Deepening
a strategy where breadth-first search is performed over the execution tree

33. Evaluation Results #runs/iterations required to cover the target
Pex w/o Fitnex: avg. improvement of factor 1.9 over Random
Pex w/ Fitnex: avg. improvement of factor 5.2 over Random

34. Object Creation
Pex normally uses public methods to configure non-public object fields
Heuristics built-in to deal with common types
User can help if needed
void (Foo foo) {
if (foo.Value == 123) throw …
[PexFactoryMethod]
Foo Create(Bar bar) {
return new Foo(bar); }

35. QuickGraph Example A graph example from QuickGraph library
interface IGraph {
/* Adds given vertex to the graph */
void AddVertex(IVertex v);
/* Creates a new vertex and adds it to the graph */
IVertex AddVertex();
/* Adds an edge to the graph. Both vertices should
already exist in the graph */
IEdge AddEdge(IVertex v1, Ivertex v2); } 35 35

36. Method Under Test
Desired object state for reaching targets 1 and 2: graph object should contain vertices and edges
Class SortAlgorithm {
IGraph graph;
public SortAlgorithm(IGraph graph) {
this.graph = graph; }
public void Compute (IVertex s) {
foreach(IVertex u in graph.Vertices)
//Target 1
foreach(IEdge e in graph.Edges)
//Target 2
method sequence

37. Example sequence generated by Randoop
Method Under Test
Applying Randoop, a random testing approach that constructs test inputs by randomly selecting method calls
Example sequence generated by Randoop
VertexAndEdgeProvider v0 = new VertexAndEdgeProvider();
Boolean v1 = false;
BidirectionalGraph v2 = new
BidirectionalGraph((IVertexAndEdgeProvider)v0, v1);
IVertex v3 = v2.AddVertex();
IVertex v4 = v0.ProvideVertex();
IEdge v15 = v2.AddEdge(v3, v4);
v4 not in the graph, so edge cannot be added to graph.
Achieved 31.82% (7 of 22) branch coverage
Reason for low coverage: Not able to generate graph with vertices and edges

38. A Mined sequence from an existing codebase
New MSeqGen Approach
Mine sequences from existing code bases
Reuse mined sequences for achieving desired object states
A Mined sequence from an existing codebase
VertexAndEdgeProvider v0;
bool bVal;
IGraph ag = new AdjacencyGraph(v0, bVal);
IVertex source = ag.AddVertex();
IVertex target = ag.AddVertex();
IVertex vertex3 = ag.AdVertex();
IEdge edg1 = ag.AddEdge(source, target);
IEdge edg2 = ag.AddEdge(target, vertex3);
IEdge edg3 = ag.AddEdge(source, vertex3);
Graph object includes both vertices and edges
Use mined sequences to assist Randoop and Pex
Both Randoop and Pex achieved 86.40% (19 of 22) branch coverage with assistance from MSeqGen

39. Challenges Addressed by MSeqGen
Existing codebases are often large and complete analysis is expensive
 Search and analyze only relevant portions
Concrete values in mined sequences may be different from desired values
 Replace concrete values with symbolic values and use dynamic symbolic execution
Extracted sequences individually may not be sufficient to achieve desired object states
 Combine extracted sequences to generate new sequences

40. MSeqGen: Code Searching
Problem: Existing code bases are often large and complete analysis is expensive
Solution:
Use keyword search for identifying relevant method bodies using target classes
Analyze only those relevant method bodies
Target classes:
System.Collections.Hashtable
QuickGraph.Algorithms.TSAlgorithm
Keywords: Hashtable, TSAlgorithm
Shortnames of target classes are used as keywords

41. MSeqGen: Sequence Generalization
Problem: Concrete values in mined sequences are different from desired values to achieve target states
Solution: Generalize sequences by replacing concrete values with symbolic values
Method Under Test
Class A {
int f1 { set; get; }
int f2 { set; get; }
void CoverMe() {
if (f1 != 10) return;
if (f2 > 25)
throw new Exception(“bug”); }
Mined Sequence for A
A obj = new A();
obj.setF1(14);
obj.setF2(-10);
obj.CoverMe();
Sequence cannot help in exposing bug since desired values are f1=10 and f2>25

42. MSeqGen: Sequence Generalization
Replace concrete values 14 and -10 with symbolic values X1 and X2
Mined Sequence for A
Generalized Sequence for A
A obj = new A();
obj.setF1(14);
obj.setF2(-10);
obj.CoverMe();
int x1 = *, x2 = *;
A obj = new A();
obj.setF1(x1);
obj.setF2(x2);
obj.CoverMe();
Use DSE for generating desired values for X1 and X2
DSE explores CoverMe method and generates desired values (X1 = 10 and X2 = 35)

43. Improvement of State-of-the-Art
Randoop
Without assistance from MSeqGen: achieved 32% branch coverage  achieved 86% branch coverage
In evaluation, help Randoop achieve 8.7% (maximum 20%) higher branch coverage
Pex
Without assistance from MSeqGen: achieved 45% branch coverage  achieved 86% branch coverage
In evaluation, help Pex achieve 17.4% (maximum 22.5%) higher branch coverage 43 43

44. Test Oracles Write assertions and Pex will try to break them
Without assertions, Pex can only find violations of runtime contracts causing NullReferenceException, IndexOutOfRangeException, etc.
Assertions leveraged in product and test code
Pex can leverage Code Contracts

45. Summary: Automated Developer Testing= ? +
Expected Outputs
Test inputs
Program
Outputs
Test Oracles
Division of Labors
Test Generation
Test inputs for PUT generated by tools (e.g., Pex)
Fitnex: guided exploration of paths [DSN 09]
MSeqGen: exploiting real-usage sequences [ESEC/FSE 09]
Test Oracles
Assertions in PUT specified by developers

46. Thank you http://research.microsoft.com/pex

47. Code Contracts
Library to state preconditions, postconditions, invariants
Supported by two tools:
Static Checker
Rewriter: turns Code Contracts into runtime checks
Pex analyses the runtime checks
Contracts act as Test Oracle
Pex may find counter examples for contracts
Missing Contracts may be suggested

48. Example: ArrayList Class invariant specification:
public class ArrayList {
private Object[] _items;
private int _size;
...
[ContractInvariantMethod] // attribute comes with Contracts
protected void Invariant() {
Contract.Invariant(this._items != null);
Contract.Invariant(this._size >= 0);
Contract.Invariant(this._items.Length >= this._size); }
How many tests do you expect to cover all reachable branches for this scenario?

49. Parameterized Models

50. Unit Testing vs. Integration Testing
Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small.
Integration test: exercises large portions of a system.
Observation: Integration tests are often “sold” as unit tests
White-box test generation does not scale well to integration test scenarios.
Possible solution: Introduce abstraction layers, and mock components not under test

51. Example Testing with Interfaces
AppendFormat(null, “{0} {1}!”, “Hello”, “World”);  “Hello World!”
.Net Implementation:
public StringBuilder AppendFormat(
IFormatProvider provider,
char[] chars, params object[] args) {
if (chars == null || args == null)
throw new ArgumentNullException(…);
int pos = 0;
int len = chars.Length;
char ch = '\x0';
ICustomFormatter cf = null;
if (provider != null)
cf = (ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter)); …

52. Stubs / Mock Objects
Introduce a mock class which implements the interface.
Write assertions over expected inputs, provide concrete outputs
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) {
Assert.IsTrue(formatType != null);
return new MCustomFormatter(); }
Problems:
Costly to write detailed behavior by example
How many and which mock objects do we need to write?

53. Parameterized Mock Objects - 1
Introduce a mock class which implements the interface.
Let an oracle provide the behavior of the mock methods.
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) { …
object o = call.ChooseResult<object>();
return o; }
Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated!

54. Parameterized Mock Objects - 2
Chosen values can be shaped by assumptions
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) { …
object o = call.ChooseResult<object>();
PexAssume.IsTrue(o is ICustomFormatter);
return o; }
(Note: Assertions and assumptions are “reversed” when compared to parameterized unit tests.)

55. Models from Choices Choices to build parameterized models
class PFileSystem : IFileSystem {
// cached choices
PexChosenIndexedValue<string,string> files; string ReadFile(string name) {
var content = this.files[name];
if (content == null)
throw new FileNotFoundException();
return content; }}

56. MSeqGen Evaluation Subjects: QuickGraph Facebook Research Questions:
RQ1: Can our approach assist Randoop (random testing tool) in achieving higher code coverages?
RQ2: Can our approach assist Pex (DSE-based testing tool) in achieving higher code coverages? 56 56 56 56 56

57. RQ1: Assisting Randoop 57 57 57 57

58. RQ2: Assisting Pex Legend: #c: number of classes
P: branch coverage achieved by Pex
P + M: branch coverage achieved by Pex and MSeqGen 58 58 58 58

59. Assumptions and Assertions
void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); } void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); }
Assumptions and assertions induce branches
Executions which cause assumption violations are ignored, not reported as errors or test cases

60. Motivation: Unit Testing Hell ResourceReader
How to test this code? (Actual code from .NET base class libraries)

61. Motivation: Unit Testing Hell ResourceReader

62. ResourceReader PUT [PexClass, TestClass]
[PexAllowedException(typeof(ArgumentNullException))]
[PexAllowedException(typeof(ArgumentException))]
[PexAllowedException(typeof(FormatException))]
[PexAllowedException(typeof(BadImageFormatException))]
[PexAllowedException(typeof(IOException))]
[PexAllowedException(typeof(NotSupportedException))]
public partial class ResourceReaderTest {
[PexMethod]
public unsafe void ReadEntries(byte[] data) {
PexAssume.IsTrue(data != null);
fixed (byte* p = data)
using (var stream = new UnmanagedMemoryStream(p, data.Length)) {
var reader = new ResourceReader(stream);
foreach (var entry in reader) { /* reading entries */ } }

63. Creating Complex Objects
Exploration of constructor/mutator method sequences
Testing with class invariants

64. Object Creation: Class Invariants
Write class invariant as boolean-valued parameterless method
Refers to private fields
Must be placed in implementation code
Exploration of valid states by setting public/private fields
May include states that are not reachable

65. Example: ArrayList Class invariant specification:
public class ArrayList {
private Object[] _items;
private int _size;
...
[ContractInvariantMethod] // attribute comes with Contracts
protected void Invariant() {
Contract.Invariant(this._items != null);
Contract.Invariant(this._size >= 0);
Contract.Invariant(this._items.Length >= this._size); }
How many tests do you expect to cover all reachable branches for this scenario?

66. Example: ArrayList PUT: [PexMethod]
public void ArrayListTest(ArrayList al, object o) {
int len = al.Count;
al.Add(o);
PexAssert.IsTrue(al[len] == o); }
How many tests do you expect to cover all reachable branches for this scenario?

67. Example: ArrayList Generated Test: [TestMethod] public void Add01() {
object[] os = new object[0];
// create raw instance
ArrayList arrayList =
PexInvariant.CreateInstance<ArrayList>();
// set private field via reflection PexInvariant.SetField<object[]>(arrayList, "_items", os); PexInvariant.SetField<int>(arrayList, "_size", 0);
// invoke invariant method via reflection PexInvariant.CheckInvariant(arrayList);
// call to PUT
ArrayListTest(arrayList, null); }
How many tests do you expect to cover all reachable branches for this scenario?