1. Advances in Unit Testing: Theory and Practice
Tao Xie Nikolai Tillmann2 Pratap Lakshman 2
1 University of Illinois at Urbana-Champaign
2 Microsoft Tools for Software Engineers/Developer Division
Materials:

2. What is a Unit Test? A unit test is a small program with assertions.
[TestMethod] public void Add(HashSet s,int x, int y) {
Assume.IsTrue(!s.IsMember(x));
int oldCount = s.GetSize();
s.Add(x);
Assert.AreEqual(set.Count, oldCount + 1); }
Many developers write such unit tests by hand. This involves
determining a meaningful sequence of method calls,
selecting exemplary argument values (the test inputs),
stating assertions.
What does exist today that is close? … You have seen how to pull out a value, let’s be a little bit more extreme…
Different partitions possible

3. Unit Testing: Benefits
Design and specification
by example
Code coverage and regression testing
confidence in correctness
preserving behavior
Short feedback loop
unit tests exercise little code
failures are easy to debug
Documentation

4. Unit Testing: Measuring Quality
Coverage: Are all parts of the program exercised?
statements
basic blocks
explicit/implicit branches …
Assertions: Does the program do the right thing?
test oracle
Experience:
Just high coverage or large number of assertions is no good quality indicator.
Only both together are!

5. Advantages of tests as specs
Concrete, easy to understand
Don’t need new language
Easy to see if program meets the spec
Making tests forces you to talk to customer and learn the problem
Making tests forces you to think about design of system (classes, methods, etc.)

6. Disadvantages of tests as specs
Too specific
Hard to test that something can’t happen
Can’t withdraw more money than you have in the system
Can’t break into the system
Can’t cause a very long transaction that hangs the system
Tends to be verbose

7. Parameterized Unit Test
A parameterized unit test is a small program that takes some inputs and states assumptions and assertions.
Parameterized Unit Test

8. Parameterized Unit Testing
Parameterized Unit Tests
serve as specifications
can be leveraged by (automatic) test input generators
fit in development environment, evolve with the code
Parameterized Unit Test

9. Test Generation Process
xUnit Attributes
Pex Attributes
Generated
// FooTest.cs [TestClass, PexClass]
partial class FooTest {
[PexMethod]
void Test(Foo foo) {…}
// FooTest.Test.cs partial class FooTest {
[TestMethod]
void Test_1()
{ this.Test(new Foo(1)); }
{ this.Test(new Foo(2)); } … }
Partial Class
Pex
Parameterized Unit Test
Hand-written
User writes parameterized tests
Lives inside a test class
Generated unit tests
Pex not required for re-execution
xUnit unit tests

10. PUTs separate concerns
PUTs separate two concerns: (1) The specification of external behavior (i.e., assertions) (2) The selection of internal test inputs (i.e., coverage) In many cases, a test generation tool (e.g., Pex) can construct a small test suite with high coverage !

11. PUTs are algebraic specs
A PUT can be read as a universally quantified, conditional axiom.  int name, int data. name ≠ null ⋀ data ≠ null ⇒ equals( ReadResource(name, WriteResource(name, data)), data)

12. Parameterized Unit Tests in JUnit
import org.junit.*;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;
import static org.junit.Assert.*;
import java.util.*;
@RunWith(Parameterized.class)
public class ParamTest {
public int sum, a, b;
public ParamTest (int sum, int a, int b) { this.sum = sum; this.a = a; this.b = b; }
@Parameters public static Collection<Object[]> parameters() {
return Arrays.asList (new Object [][] {{0, 0, 0}, {2, 1, 1}}); }
@Test public void additionTest() { assertEquals(sum, a+b); }
Introduction to Software Testing (Ch 1)
© Ammann & Offutt

13. JUnit Theories These Are Unit Tests With Actual Parameters
So Far, We’ve Only Seen Parameterless Test Methods
Contract Model: Assume, Act, Assert
Assumptions (Preconditions) Limit Values Appropriately
Action Performs Activity Under Scrutiny
Assertions (Postconditions) Check Result
@Theory public void removeThenAddDoesNotChangeSet(
Set<String> set, String string) { // Parameters!
assumeTrue(set.contains(string)) ; // Assume
Set<String> copy = new HashSet<String>(set); // Act
copy.remove(string);
copy.add(string);
assertTrue (set.equals(copy)); // Assert //
// System.out.println(“Instantiated test: “ + set + “, “ + string); }
Introduction to Software Testing (Ch 1)
© Ammann & Offutt

14. Question: Where Does Data Come From?
Answer:
All Combinations of Values Annotations Where Assume Clause is True
Four (of Nine) Combinations in This Particular Case
Note: @DataPoint Format is an Array.
@DataPoints
public static String[] string = {"ant", "bat", "cat"};
public static Set[] sets = {
new HashSet(Arrays.asList("ant", "bat")),
new HashSet(Arrays.asList(“bat", “cat", “dog“, “elk”)),
new HashSet(Arrays.asList(“Snap”, “Crackle”, “Pop")) };
Introduction to Software Testing (Ch 1)
© Ammann & Offutt

15. JUnit Theories Need BoilerPlate
import org.junit.*;
import org.junit.runner.RunWith;
import static org.junit.Assert.*;
import static org.junit.Assume.*;
import org.junit.experimental.theories.DataPoint;
import org.junit.experimental.theories.DataPoints;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;
import java.util.*;
@RunWith(Theories.class)
public class SetTheoryTest {
… // See Earlier Slides }
Introduction to Software Testing (Ch 1)
© Ammann & Offutt

16. (Automated) Test Generation
Human
Expensive, incomplete, …
Brute Force
Pairwise, predefined data, etc…
Tool Automation!!

17. State-of-the-Art/Practice Test Generation Tools
Running Symbolic PathFinder ... …
====================================================== results
no errors detected
====================================================== statistics
elapsed time: :00:02
states: new=4, visited=0, backtracked=4, end=2
search: maxDepth=3, constraints=0
choice generators: thread=1, data=2
heap: gc=3, new=271, free=22
instructions:
max memory: MB
loaded code: classes=71, methods=884 …

19. Pex is Part of Visual Studio 2015 Enterprise Edition!
As new feature of “IntelliTest”

20. Pex4Fun – Turning Pex Online
1,767,012 clicked 'Ask Pex!'

21. Code Hunt Programming Game
130,000 users
as of March 2015
after 1 year launch

22. What are Behind Pex NOT Random: …
Cheap, Fast
“It passed a thousand tests” feeling …
But Dynamic Symbolic Execution: e.g., Pex, CUTE,EXE
White box
Constraint Solving

23. Behind the Scene of Pex Dynamic Symbolic Execution (DSE)
aka. Concolic Testing
[Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08]
Instrument code to explore feasible paths

24. Dynamic Symbolic Execution
Choose next path
Code to generate inputs for:
Solve
Execute&Monitor
void CoverMe(int[] a) {
if (a == null) return;
if (a.Length > 0)
if (a[0] == )
throw new Exception("bug"); }
Constraints to solve a!=null a!=null && a.Length>0 a.Length>0 && a[0]==
Data null {} {0} {123…}
Observed constraints a==null a!=null && !(a.Length>0) a.Length>0 && a[0]!= a[0]==
Negated condition
a==null F T
a.Length>0 F T
Done: There is no path left.
a[0]==123… F T

25. Explosion of Search Space
There are decision procedures for individual path conditions, but…
Number of potential paths grows exponentially with number of branches
Reachable code not known initially
Without guidance, same loop might be unfolded forever
Fitnex search strategy
[Xie et al. DSN 09]

26. Successful Case of MSR Testing Tool: Pex & Relatives
Pex (released on May 2008)
Shipped with Visual Studio 15 as IntelliTest
30,388 download# (20 months, Feb 08-Oct 09)
22,466 download# (10 months, Apr 13-Jan 14): Code Digger
Active user community: 1,436 forum posts during ~3 years (Oct 08- Nov 11)
Moles (released on Sept 2009)
Shipped with Visual Studio 12 as Fakes
“Provide Microsoft Fakes w/ all Visual Studio editions” got 1,457 community votes
A Journey of Bringing Automated Unit Test Generation to Practice

27. Lesson 1. Started as (Evolved) Dream
Moles/Fakes
Parameterized Unit Tests Supported by Pex
Code Digger
void TestAdd(ArrayList a, object o) {
Assume.IsTrue(a!=null);
int i = a.Count;
a.Add(o);
Assert.IsTrue(a[i] == o); }
Pex4Fun/Code Hunt
Surrounding (Moles/Fakes)
Simplifying (Code Digger)
Retargeting (Pex4Fun/Code Hunt)

28. Lesson 2. Chicken and Egg Tool Adoption by (Mass) Target Users
Macro Perspective
Tool Adoption by (Mass) Target Users
Tool Shipping with Visual Studio
Micro Perspective
Developer/manager: “Who is using your tool?”
Pex team: “Do you want to be the first?”
Developer/manager: “I love your tool but no.”

29. Lesson 3. Human Factors – Generated Data Consumed by Human
Developer: “Code digger generates a lot of “\0” strings as input. I can’t find a way to create such a string via my own C# code. Could any one show me a C# snippet? I meant zero terminated string.”
Pex team: “In C#, a \0 in a string does not mean zero-termination. It’s just yet another character in the string (a very simple character where all bits are zero), and you can create as Pex shows the value: “\0”.”
Developer: “Your tool generated “\0””
Pex team: “What did you expect?”
Developer: “Marc.”

30. Lesson 3. Human Factors – Generated Name Consumed by Human
Developer: “Your tool generated a test called Foo001. I don’t like it.”
Pex team: “What did you expect?”
Developer:“Foo_Should_Fail_When_Bar_Is_Negative.”

31. Lesson 3. Human Factors – Generated Results Consumed by Human
Object Creation messages suppressed (related to Covana by Xiao et al. [ICSE’11])
Exception Tree View
Exploration Tree View
Exploration Results View

32. Lesson 4. Best vs. Worst Cases
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true; }
return false;
Fitnex by Xie et al. [DSN’09]
Key observations: with respect to the coverage target
not all paths are equally promising for branch-node flipping
not all branch nodes are equally promising to flip
To avoid local optimal or biases, the fitness-guided strategy is integrated with Pex’s fairness search strategies
Our solution:
Prefer to flip branch nodes on the most promising paths
Prefer to flip the most promising branch nodes on paths
Fitness function to measure “promising” extents

33. Lesson 5. Tool Users’ Stereotypical Mindset or Habits
“Simply one mouse click and then everything would work just perfectly”
Often need environment isolation w/ Moles/Fakes or factory methods, …
“One mouse click, a test generation tool would detect all or most kinds of faults in the code under test”
Developer: “Your tool only finds null references.”
Pex team: “Did you write any assertions?”
Developer: “Assertion???”
“I do not need test generation; I already practice unit testing (and/or TDD). Test generation does not fit into the TDD process”

34. Lesson 6. Practitioners’ Voice
Gathered feedback from target tool users
Directly, e.g., via
MSDN Pex forum, tech support, outreach to MS engineers and .NET user groups
Indirectly, e.g., via
interactions with MS Visual Studio team (a tool vendor to its huge user base)
Motivations of Moles
Refactoring testability issue faced resistance in practice
Observation at Agile 2008: high attention on mock objects and tool supports

35. Lesson 7. Collaboration w/ Academia
Win-win collaboration model
Win (Ind Lab): longer-term research innovation, man power, research impacts, …
Win (Univ): powerful infrastructure, relevant/important problems in practice, both research and industry impacts, …
Industry-located Collaborations
Faculty visits, e.g., Fitnex, Pex4Fun
Student internships, e.g., FloPSy, DyGen, state cov
Academia-located Collaborations

36. Lesson 7. Collaboration w/ Academia
Academia-located Collaborations
Immediate indirect impacts, e.g.,
Reggae [ASE’09s]  Rex
MSeqGen [FSE’09]  DyGen
Guided Cov [ICSM’10]  state coverage
Long-term indirect impacts, e.g.,
DySy by Csallner et al. [ICSE’08]
Seeker [OOPSLA’11]
Covana [ICSE’11]

37. Summary of Lessons Learned
Pex  practice impacts
Moles/Fakes, Code Digger, Pex4Fun/Code Hunt
Lessons in transferring tools
Started as (Evolved) Dream
Chicken and Egg
Human Factors
Best vs. Worst Cases
Tool Users’ Stereotypical Mindset or Habits
Practitioners’ Voice
Collaboration w/ Academia

38. Experience Reports on Successful Tool Transfer
Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of ASE 2014, Experience Papers.
Jian-Guang Lou, Qingwei Lin, Rui Ding, Qiang Fu, Dongmei Zhang, and Tao Xie. Software Analytics for Incident Management of Online Services: An Experience Report. In Proceedings ASE 2013, Experience Paper.
Dongmei Zhang, Shi Han, Yingnong Dang, Jian-Guang Lou, Haidong Zhang, and Tao Xie. Software Analytics in Practice. IEEE Software, Special Issue on the Many Faces of Software Analytics,
Yingnong Dang, Dongmei Zhang, Song Ge, Chengyun Chu, Yingjun Qiu, and Tao Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. In Proceedings of ACSAC

39. Parameterized Unit Tests Supported by Pex/Pex4Fun
using System; using Microsoft.Pex.Framework; using Microsoft.Pex.Framework.Settings; [PexClass] public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….

40. Interface for IntSet
Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … } } sort IntSet imports Int, Bool signatures new : -> IntSet insert : IntSet × Int -> IntSet member : IntSet × Int -> Bool remove : IntSet × Int -> IntSet

41. (Buggy) Implementation for IntSet
Class IntSet {
public IntSet() {…};
public void insert(int e) { … }
public Bool member(int e) { … }
public void remove(int e) { … } }
See the Set.cs that can be downloaded from
Let’s copy it to
And Click “Ask Pex”

42. Parameterized Unit Tests Supported by Pex/Pex4Fun
using System; using Microsoft.Pex.Framework; using Microsoft.Pex.Framework.Settings; [PexClass] public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….
Pex4Fun supports only one PexMethod at a time;
you can write multiple PexMethods but comment out other lines of “[PexMethod]” except one

43. Axioms for IntSet
variables i, j : Int; s : IntSet Axioms: member(new(), i) = false member(insert(s, j), i) = if i = j then true else member(s, i)
Is this complete?
How do we know?

44. Guidelines for Completeness
Classify methods:
constructors: return IntSet
inspectors: take IntSet as argument, returning some other value.
Identify key constructors, capable of constructing all possible object states
e.g., insert, new.
Identify others as auxiliary,
e.g., remove is a destructive constructor
Completeness requires (at least):
every inspector/auxiliary constructor is defined by one equation for each key constructor. 

45. Add More Axioms remove(new(), i) = new()
remove(insert(s, j), i) = if i = j then remove(s, i) else insert(remove(s, i), j)
Are we done yet?
The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied.

46. Guidelines for Completeness
But does this really specify sets? Do the following properties hold?
Order of insertion is irrelevant.
insert(insert(s, i), j) = insert(insert(s, j), i)
Multiple insertion is irrelevant.
insert(insert(s, i), i) = insert(s, i)

47. Interface (Implementation) for UIntStack
Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … } }
See the UIntStack.cs that can be downloaded from

48. Take-Home Exercise: Write Parameterized Unit Tests (PUTs)
Let’s copy it to
And Click “Ask Pex”
Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … } }
Reminder: you have to comment earlier written “[PexMethod]” before you try Pex on your current PUT (Pex4Fun can handle only one PUT at a time)
See the UIntStack.cs that can be downloaded from

49. Recall: Parameterized Unit Tests Supported by Pex/Pex4Fun
using System; using Microsoft.Pex.Framework; using Microsoft.Pex.Framework.Settings; [PexClass] public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….

50. Force Pex/Pex4Fun to Display All Explored Test Inputs/Paths
using System;
using Microsoft.Pex.Framework;
using Microsoft.Pex.Framework.Settings;
[PexClass]
public class Set {
[PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)]
public static void testMemberAfterInsertNotEqual(Set s, int i, int j) {
PexAssume.IsTrue(s != null);
PexAssume.IsTrue(i != j);
bool exist = s.member(i);
s.insert(j);
PexAssert.IsTrue(exist); } ….

51. Factory Method: Help Pex Generate Desirable Object States
In class, we show the factory method as below automatically synthesized by Pex after a user clicks “1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states.
[PexFactoryMethod(typeof(UIntStack))]
public static UIntStack Create(int k_i) {
UIntStack uIntStack = new UIntStack();
uIntStack.Push(k_i);
return uIntStack;
// TODO: Edit factory method of UIntStack
// This method should be able to configure the object in all possible ways.
// Add as many parameters as needed,
// and assign their values to each field by using the API. }

52. Factory Method: Help Pex Generate Desirable Object States
Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method.
[PexFactoryMethod(typeof(UIntStack))]
public static UIntStack CreateVariedSizeAnyElemsStack(int[] elems) {
PexAssume.IsNotNull(elems);
UIntStack s = new UIntStack();
PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1));
for (int i = 0; i < elems.Length; i++)
s.Push(elems[i]);
return s; }

53. One Sample PUT
Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method.
[PexMethod]
public void TestPush([PexAssumeUnderTest]UIntStack s, int i) {
//UIntStack s = new UIntStack();
PexAssume.IsTrue(!s.IsMember(i));
int oldCount = s.GetNumberOfElements();
s.Push(i);
PexAssert.IsTrue(s.Top() == i);
PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1);
PexAssert.IsFalse(s.IsEmpty()); }

54. Pex4Fun Not Supporting Factory Method - Workaround
If you try PUTs on Pex4Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below
[PexMethod]
public void TestPush(int[] elems, int i) {
PexAssume.IsNotNull(elems);
UIntStack s = new UIntStack();
PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1));
for (int i = 0; i < elems.Length; i++)
s.Push(elems[i]);
//UIntStack s = new UIntStack();
PexAssume.IsTrue(!s.IsMember(i));
int oldCount = s.GetNumberOfElements();
s.Push(i);
PexAssert.IsTrue(s.Top() == i);
PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1);
PexAssert.IsFalse(s.IsEmpty()); }

55. Guideline of Writing PUT
Setup: basic set up for invoking the method under test
Checkpoint: Run Pex to make sure that you don't miss any Pex assumptions (preconditions) for the PUT
Assert: add assertions for asserting behavior of the method under test, involving
Adding Pex assertions
Adding Pex assumptions for helping assert
Adding method sequences for helping assert

56. Setup Select your method under test m Put its method call in your PUT
Create a parameter for your PUT as the class under test c (annotated it with [PexAssumeUnderTest])
Create other parameters for your PUT for parameters of m if any
Add Pex assumptions for preconditions for all these parameters of PUT if any

57. Setup - Example [PexMethod]
public void TestPush([PexAssumeUnderTest]UIntStack s, int i) {
s.Push(i); }
You may write your factory method to help Pex in test generation
If you get exceptions thrown
if indicating program faults, fix them
If indicating lack of PUT assumptions, add PUT assumptions
If indicating insufficient factory method assumptions or inappropriate scenarios, add PUT assumptions or improve factory method.

58. Assert Think about how you can assert the behavior
Do you need to invoke other (observer) helper methods in your assertions (besides asserting return values)?
Do you need to add assumptions so that your assertions can be valid?
Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?

59. Targets for Asserting Return value of the method under test (MUT)
Argument object of MUT
Receiver object properties being modified by MUT (if public fields, directly assertable)
How to assert them?
Think about the intended behavior!
If you couldn't do so easily, follow the guidelines discussed next

60. Cached Public Property Value
A property value before invoking MUT may need to be cached and later used.
Pattern 2.1/2.2: Assume, Arrange, Act, Assert
[PexMethod]
void AssumeActAssert(ArrayList list, object item) {
PexAssume.IsNotNull(list); // assume
var count = list.Count; // arrange
list.Add(item); // act
Assert.IsTrue(list.Count == count + 1); // assert }

61. Argument of MUT Argument value of MUT may be used
Pattern 2.3:Constructor Test
[PexMethod]
void Constructor(int capacity) {
var list = new ArrayList(capacity); // create
AssertInvariant(list); // assert invariant
Assert.AreEqual(capacity, list.Capacity); // assert }

62. Reciever or Argument of Earlier Method
Receiver or argument value of a method before invoking MUT
Pattern 2.4/5:Roundtrip
[PexMethod]
void ToStringParseRoundtrip(int value) {
// two-way roundtrip
string s = value.ToString();
int parsed = int.Parse(s);
// assert
Assert.AreEqual(value, parsed); }
value  s  parsed

63. Observer Methods
Invoking observer methods on the modified object state
Pattern 2.6: State Relation
[PexMethod]
void InsertContains(string value) {
var list = new List<string>();
list.Add(value);
Assert.IsTrue(list.Contains(value)); }
Each modified object property should be read by at least one observer method.

64. Observer Methods cont.
Forcing observer methods to return specific values (e.g., true or false) can force you to add specific assumptions or scenarios
[PexMethod]
void PushIsFull([PexAssumeUnderTest]UIntStack s, int value) {
PexAssume.IsTrue(s.GetSize() == (s.GetMaxSize()-1));
s.Push (value);
Assert.IsTrue(s.IsFull ()); }

65. Alternative Computation
Invoking another method/method sequence to produce a value to be used
Pattern 2.7: Commutative Diagram
[PexMethod]
void CommutativeDiagram1(int x, int y) {
// compute result in one way
string z1 = Multiply(x, y).ToString();
// compute result in another way
string z2 = Multiply(x.ToString(), y.ToString());
// assert equality if we get here
PexAssert.AreEqual(z1, z2); }

66. Divide and Conquer
Split possible outcomes into cases (each with pre and post condition)
Pattern 2.8: Cases
[PexMethod]
void BusinessRules(int age, Job job) {
var salary = SalaryManager.ComputeSalary(age, job);
PexAssert
.Case(age < 30)
.Implies(() => salary < 10000)
.Case(job == Job.Manager && age > 35)
.Implies(() => salary > 10000)
.Case(job == Job.Manager && age < 20)
.Implies(() => false); }

67. Class Invariant Checker
If class invariant checker (repOk) exists or you would be willing to write one, use it to assert
Pattern 2.3:Constructor Test
[PexMethod]
void Constructor(int capacity) {
var list = new ArrayList(capacity); // create
AssertInvariant(list); // assert invariant
Assert.AreEqual(capacity, list.Capacity); // assert }

68. Other Patterns Pattern 2.9: Allowed exceptions
[PexAllowedException(typeof(ArgumentNullException))]
[ExpectedException(typeof(ArgumentNullException))]
Pattern 2.10: Reachability
[PexExpectedGoals] + throw new PexGoalException();
Pattern 2.11: Parameterized Stub
No scenarios or assertions
Pattern 2.12: Input Output Test
void Add(int a, int b, out int result) { … }
int Substract(int a, int b) { … }
Pattern 2.13/14: Regression Tests
bool Parse(string input) { … }
PexStore.ValueForValidation("result", result);

70. Test-Driven Development (TDD)
Basic Idea:
Write tests before code
Refine code with new tests
In more detail, TDD is a cycle of steps:
Add a test,
Run it and watch it fail,
Change the code as little as possible such that the test should pass,
Run the test again and see it succeed,
Refactor the code if needed.

71. Note: TDD and specifications
TDD encourages writing specifications before code
Exemplary specification
Later, we will generalize TDD to Parameterized TDD
Axiomatic specifications

72. Parameterized Test- Driven Development
Write/refine Contract as PUT
Bug in PUT
Write/refine Code of Implementation
Bug in Code
Fix-it (with Pex),
Debug with generated tests
failures
Run Pex
no failures
Use Generated Tests for Regression

73. Coding Duels
1,767,012 clicked 'Ask Pex!'

74. Coding Duels Pex computes “semantic diff” in cloud
secret reference implementation vs.
code written in browser
You win when Pex finds no differences
secret
For more info, see our ICSE 2013 SEE paper:

75. Behind the Scene of Pex for Fun
behavior
Secret Impl == Player Impl
Player Implementation
class Player {
public static int Puzzle(int x) {
return x; }
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1); }
1,594,092
1,594,092
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”); }

76. Code Hunt Programming Game

77. Code Hunt Programming Game

78. Code Hunt Programming Game

79. Code Hunt Programming Game

80. Code Hunt Programming Game

81. Code Hunt Programming Game

82. Code Hunt Programming Game

83. Code Hunt Programming Game

84. Code Hunt Programming Game

85. Code Hunt Programming Game

86. Code Hunt Programming Game

87. Code Hunt Programming Game

88. Code Hunt Programming Game

89. It’s a game!
code
iterative gameplay adaptive personalized no cheating clear winning criterion
secret
test cases

90. Thank you! Questions ? http://research.microsoft.com/pex
Materials: