1. Pexxxx White Box Test Generation for
Pexxxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux
Microsoft Research

2. Unit Testing Today A unit test is a small program with assertions.
void AddTest() {
HashSet set = new HashSet();
set.Add(7);
set.Add(3);
Assert.IsTrue(set.Count == 2); }
Many developers write such unit tests by hand.

3. Vision: Parameterized Unit Testing
void AddSpec(int x, int y) {
HashSet set = new HashSet();
set.Add(x);
set.Add(y);
Assert.AreEqual(x == y, set.Count == 1);
Assert.AreEqual(x != y, set.Count == 2); }
Parameterized Unit Tests separate two concerns:
The specification of externally visible behavior. (assertions)
The selection of internally relevant test inputs. (coverage)

4. Demo: Parameterized Unit Test
Write [PexMethod] with parameters, invoke Analysis

5. Demo: Pex Exploration Results
Table: data and results

6. Demo: Generated Unit Tests
Generated Test Inputs
are stored as C# Unit Tests

7. Pex: Overview Purpose: Test input generator
Starting from parameterized unit tests
Generated tests emitted as traditional unit tests
Goal: Test suite that covers all reachable statements
Technology: Dynamic symbolic execution
Whole-program, white-box code analysis
At the level of the .NET instructions (bytecode)
Symbolic execution based on monitoring and re-execution
Path conditions obtained by monitoring dataflow, branches
Iterative selection of potential execution paths
Constraint solver determines test inputs for paths

8. Dynamic Symbolic Execution By Example
Choose next path
Solve
Execute&Monitor
Code to generate inputs for:
Constraints to solve a!=null a!=null && a.Length>0 a.Length>0 && a[0]==
Input null {} {0} {123…}
Observed constraints a==null a!=null && !(a.Length>0) a==null && a.Length>0 && a[0]!= a[0]==
void CoverMe(int[] a) {
if (a == null) return;
if (a.Length > 0)
if (a[0] == )
throw new Exception("bug"); }
Negated condition
a==null F T
Done: There is no path left.
a.Length>0 F T
a[0]==123… F T

9. Going from simple example Into the real world
How to test this code?
(Actual code from .NET base class libraries.)

10. Real World (II)

11. Real World (III) There is no detailed specification.
Most basic test oracle:
Code should not crash, i.e. throwing unexpected exceptions
Possible test case, written by Hand

12. Pex – Demo
Test input, generated by Pex 12

13. Test Input Generation by Dynamic Symbolic Execution
Initially, choose Arbitrary
Run Test and Monitor
Solve
Test Inputs
Constraint System
Execution Path
Known Paths
Choose an
Uncovered Path
Record
Path Condition

14. Test Input Generation by Dynamic Symbolic Execution…
Initially, choose Arbitrary
Run Test and Monitor
Solve
Test Inputs
Constraint System
Execution Path
Known Paths
Choose an
Uncovered Path
Record
Path Condition

15. Test Input Generation by Dynamic Symbolic Execution
Initially, choose Arbitrary
Path Condition:
… ⋀ magicNum != 0x
Run Test and Monitor
Solve
Test Inputs
Constraint System
Execution Path
Known Paths
Choose an
Uncovered Path
Record
Path Condition

16. Test Input Generation by Dynamic Symbolic Execution
Initially, choose Arbitrary
Run Test and Monitor
… ⋀ magicNum != 0x
… ⋀ magicNum == 0x
Solve
Test Inputs
Constraint System
Execution Path
Known Paths
Choose an
Uncovered Path
Record
Path Condition

17. Test Input Generation by Dynamic Symbolic Execution
Initially, choose Arbitrary
Run Test and Monitor
Solve
Test Inputs
Constraint System
Execution Path
Known Paths
Choose an
Uncovered Path
Record
Path Condition

18. Test Input Generation by Dynamic Symbolic Execution
Initially, choose Arbitrary
Run Test and Monitor
Solve
Test Inputs
Constraint System
Execution Path
Known Paths
Choose an
Uncovered Path
Record
Path Condition

19. Test Input Generation by Dynamic Symbolic Execution
Initially, choose Arbitrary
Run Test and Monitor
Solve
Test Inputs
Constraint System
Execution Path
Known Paths
Choose an
Uncovered Path
Record
Path Condition
Result: small test suite, high code coverage
Finds only real bugs
No false warnings

20. Advanced Features Command-line tool Parameterized Mock Objects
Explores your code on your build server
Generates HTML reports
Parameterized Mock Objects
Stubs and mock objects can use parameters too
Parameterized Object Factories
Factories that take parameters help Pex

21. Assumptions and Assertions
void PexAssume.IsTrue(bool c) {
if (!c) throw new AssumptionViolationException(); }
void PexAssert.IsTrue(bool c) {
if (!c) throw new AssertionViolationException();
Assumptions and assertions are explored just like all other branches; Pex will try to break them!
Executions which cause assumption violations are ignored, not reported as errors or test cases

22. Limitations Assumption: Environment is deterministic
"Environment" includes all code that is not monitored, e.g. native code, uninstrumented code, code that executes on another machine on the network…
Pex simply ignores non-deterministic behavior
Assumption: Program is single-threaded
Limitations of constraint solver
Little support for floating point arithmetic
approximation with rationals (linear arithmetic only)
No support for System.Decimal arithmetic
Bounds on time and memory consumption

23. Summary Pex automates test input generation for .NET programs
Pex enables Parameterized Unit Testing
Used in Microsoft to test core .NET components
Pex is publicly available for academic use.
Watch our website for future announcements.