1. White-Box Testing Using Pex
Snigdha Athaiya
Formal Methods in Software Engineering
*Slides based on the material on Pex by K.V. Raghavan and presentation by Tao Xie titled “Parameterized Unit Testing using Pex”

2. What is a Unit Test?
A unit test is a small program with assertions, scrutinizing the smallest testable parts of an application.
[TestMethod] public void AddTest() {
HashSet set = new HashSet();
set.Add(3);
set.Add(14);
Assert.AreEqual(set.Count, 2); }
Annotation
Meaningful Sequence of method calls
Selecting test inputs
Stating appropriate assertions
What does exist today that is close? … You have seen how to pull out a value, let’s be a little bit more extreme…
Different partitions possible

3. Unit Tests
smallest testable parts or units can be procedures, classes or even logical modules
Purpose of unit testing – to isolate sections of a program and ensure their correctness
Correctness? It is with respect to the specifications of the unit under test.
If source code is changed due to addition of specifications, then a new set of test cases are written. The new code should still pass the old tests. A growing body of test cases is always desirable.

4. Writing a Unit test - Demo

5. Unit Testing: Problems
Quality of unit tests
Do they test what is important?
Amount of unit tests
How many tests to write?

6. Unit Testing: Measuring Quality
Coverage: Are all parts of the program exercised?
statements
basic blocks
explicit/implicit branches …
Assertions: Does the program do the right thing?
Just high coverage or large number of assertions is not a good quality indicator.
Only both together are!

7. Unit Testing: Measuring Quality
Coverage: Are all parts of the program exercised?
statements
basic blocks
explicit/implicit branches …
Assertions: Does the program do the right thing?
We will focus on the first part for now, i.e. coverage

8. The Testing Problem Given, a sequential program P,
with conditionals C,
compute a set of program inputs I such that :
for all conditionals c in C and for each branch b of c,
if b is reachable in some execution of P, then
there exists an input i in I such that
the execution of P on input i goes through branch b.

9. Pex test input generation framework analyzes the execution paths
generates inputs to achieve maximum branch coverage
known C# exceptions treated as implicit conditionals
analyzes the execution paths
at the level of .NET instructions
Incremental analysis that discovers feasible execution paths
represents execution paths as constraints and uses the constraint solver Z3 for checking feasibility
uses Dynamic Symbolic Execution
a combination of static and dynamic analysis

10. Beginning execution with x=0 and y=1
Negating the condition 0 ≤ 0
Beginning execution with symbols for x and y
x = 0,y = 1
x = 1,y = 0
x = 0,y = 0
x↦ 𝑥 0 , y↦ 𝑦 0
Pex – Path Exploration b1 F T
𝑥 0 ≤0
𝑥 0 >0
m(x,y) { b1: if (x > 0) t = 1 else t = 0 b2: if (y > 0) x = x + t x = x - t b3: if (t > 0) z = x z = x + 1 }
t ↦0,
x↦ 𝑥 0 , y↦ 𝑦 0
t = 0
t ↦1
t = 1 b2 b2
𝑦 0 ≤0
𝑦 0 >0
x = 0
x↦ 𝑥 0 −0
x = 0
x↦ 𝑥 0 +0
x ↦ 𝑥 0 −1
x = 0 b3 b3 b3
0≤0
0>0
z = 1
z ↦ 𝑥 0 +1
z = 1
z ↦ 𝑥 0 +1
z ↦ 𝑥 0
z = 0
Input:
𝑥 0 = 0
𝑦 0 = 0
Input:
𝑥 0 = 0
𝑦 0 = 1
Solution of
𝑥 0 ≤0∧ 𝑦 0 >0
Input:
𝑥 0 = 1
𝑦 0 = 0

11. Pex Algorithm

12. x↦ 𝑥 0 , y↦ 𝑦 0
Key Terminology b1 F T
𝑥 0 ≤0
𝑥 0 >0
t ↦0,
x↦ 𝑥 0 , y↦ 𝑦 0
Symbolic Expression – an expression involving the initial symbolic values of variables.
Symbolic State – mapping from variables to Symbolic Expressions.
t ↦1 b2 b2
𝑦 0 ≤0
𝑦 0 >0
x↦ 𝑥 0 −0
x↦ 𝑥 0 +0
x ↦ 𝑥 0 −1 b3 b3 b3
0>0
z ↦ 𝑥 0 +1
z ↦ 𝑥 0 +1
z ↦ 𝑥 0
Input:
𝑥 0 = 0
𝑦 0 = 0
Input:
𝑥 0 = 0
𝑦 0 = 1
Input:
𝑥 0 = 1
𝑦 0 = 0

13. b1 b2 b3
x↦ 𝑥 0 , y↦ 𝑦 0
t ↦0,
x↦ 𝑥 0 −0
x↦ 𝑥 0 +0
z ↦ 𝑥 0 +1
t ↦1
x ↦ 𝑥 0 −1
z ↦ 𝑥 0
𝑥 0 >0
𝑥 0 ≤0
𝑦 0 ≤0
𝑦 0 >0
0>0 F T
Input:
𝑥 0 = 0
𝑦 0 = 0
𝑦 0 = 1
𝑥 0 = 1
Key Terminology
m(x,y) {
b1: if (x > 0)
t = 1
else
t = 0
b2: if (y > 0)
x = x + t
x = x - t
b3: if (t > 0)
z = x
z = x + 1 }
Static Conditional is a conditional node in the program. Each of the two branches out of a static conditional is called a Static Branch.
A Dynamic Conditional is a conditional node in the execution tree, and is an instance of a static conditional. Each of the two branches out of a dynamic conditional is called a Dynamic Branch

14. b1 b2 b3
x↦ 𝑥 0 , y↦ 𝑦 0
t ↦0,
x↦ 𝑥 0 −0
x↦ 𝑥 0 +0
z ↦ 𝑥 0 +1
t ↦1
x ↦ 𝑥 0 −1
z ↦ 𝑥 0
𝑥 0 >0
𝑥 0 ≤0
𝑦 0 ≤0
𝑦 0 >0
0>0 F T
Input:
𝑥 0 = 0
𝑦 0 = 0
𝑦 0 = 1
𝑥 0 = 1
Key Terminology
A Branch Constraint is a Boolean expression annotating a dynamic branch
An Execution Tree represents a set of execution traces. Each path from the root to a leaf represents a full execution trace from beginning to end of the method under test. The tree is factored, that is, any common prefix of one or more traces occurs only once in the tree.

15. Outline of Pex Algorithm
Maintain a tree T of execution traces already taken. Initially it is empty.
Run given method m using default inputs. Let t be the trace obtained.
while(not all static branches covered){
Add t to the tree T.
Compute symbolic states after assignment statements in t and branch constraints along all branches in t.
If t reached some static branch that has not yet been covered, use the input values that were used to obtain t to emit a unit test.
newTraceFound = false

16. while(there exists a dynamic conditional b in the tree T such that
one of its branches has been executed so far
the other branch is not present in the tree
AND iterations limit of the inner loop not yet hit
AND newTraceFound == false) {
Flip the constraint on the conditional b and feed the conjunction of all branch constraints from the root of the tree up to and including the flipped conditional to the solver. This conjunction is called a path constraint.
If(solver finds a solution I)
newTraceFound = true
else
Mark untaken branch out of b with an ‘X’ }
If newTraceFound == true
then execute method m on input I and let t be the resulting trace
else break

17. Branch Flipping strategies
In inner loop condition we need to pick a condition b to flip. What if there are multiple candidates?
Simple DFS and BFS are naïve possibilities. Why?
Pex uses multiple strategies, and gives each one of them a turn (e.g. round- robin)
Strategy 0:
For each static conditional node in the program maintain a count of how many times its instances have been flipped. At each turn choose the available one with the smallest count.
Strategy 1:
pick the shallowest dynamic conditional (based on path depth) i.e. BFS
When it is strategy i’s turn, if there are many dymanic branches that are in a tie, it could use strategy j to break the tie.

18. while(there exists a dynamic conditional b in the tree T such that
one of its branches has been executed so far
the other branch is not present in the tree
AND iterations limit of the inner loop not yet hit
AND newTraceFound == false) {
Flip the constraint on the conditional b and feed the conjunction of all branch constraints from the root of the tree up to and including the flipped conditional to the solver. This conjunction is called a path constraint.
If(solver finds a solution I)
newTraceFound = true
else
Mark untaken branch out of b with an ‘X’ }
If newTraceFound == true
then execute method m on input I and let t be the resulting trace
else break
while ( ((b = findCondToFlip()) ≠𝜙) 𝐴𝑁𝐷 newTraceFound == false) {

19. Try Strategy 0 on suitableCond next strategy to try will be 1
findCondToFlip() { // total number of strategies = 2, named as Strategy 0 and Strategy 1 static int nextStrat = 0; int currentStrat = nextStrat; suitableCond = {b | b is a dynamic conditional in the tree AND one of its branches has been executed so far AND the other branch is not marked ‘X’ } for i = (currentStrat mod 2) to ((currentStrat+1)mod 2) { nextStrat = (i + 1) mod 2; fromI = {b | b is the top candidate in suitableCond by Strategy i} if (fromI is a singleton) return the conditional in fromI else { //tie breaker for j = ((i+1)mod 2) to ((i-1) mod 2) { fromJ = {b | b is the top candidate in fromI by Strategy j} if (fromJ is a singleton) return the conditional in fromJ } } } return 𝜙;}
Start with Strategy 0
Try Strategy 0 on suitableCond
next strategy to try will be 1
Now Trying Strategy 1 on suitableCond
next strategy to try will be 0
in case solution not found, the loop will exit
and return 𝜙
Exit with solution of single candidate found
Else Tie Breaker!!
Try strategy 1 to choose a single candidate out of fromI
Else Tie Breaker!!
Try strategy 1 to choose a single candidate out of fromI
No more strategies to use  exit tie-breaker loop

20. Illustration of Branch Flipping Strategies
Flipping the shallowest conditional with the lowest count
b1: while (x<10) x++; b2: assert(x==11); b1
𝑥 0 < 10
𝑥 0 ≥10
Flipping the cond with the lowest count (0)
x ↦ 𝑥 0 +1 b1 b2
𝑥 0 ≠11
𝑥 0 +1<10
Strategy 0:
For each static conditional node in the program maintain a count of how many times its instances have been flipped. At each turn choose the available one with the smallest count.
Strategy 1:
pick the shallowest dynamic conditional (based on path depth) i.e. BFS b1
𝑥 0 =10
Counts: b1 :1
b2 :0
𝑥 0 =11
Counts: b1 :1
b2 :1
𝑥 0 +10≥10 b2
𝑥 0 +10≠11
currentStrat = 1
tie breaking using 0
nextStrat = 0
nextStrat = 1
will be used by the next cond finding call
currentStrat = 0
currentStrat = 0
tie breaking using 1
nextStrat = 0
currentStrat = 1
𝑥 0 =0
Count: b1 :0
b2 :0

21. Parametrized Unit Tests
A parameterized unit test is a small program that takes some inputs and states assumptions and assertions.
Takes inputs
[DataTestMethod]
public void testDiv(int m, int n) {
IntMathFunctionsSP obj = new IntMathFunctionsSP();
try
IntMathFunctionsSP.DivResult res = obj.div(m, n);
Assert.IsTrue(res.quotient * n + res.remainder == m);
Assert.IsTrue(res.remainder >= 0 && res.remainder < n); }
catch(IntMathFunctionsSP.IllegalDivision illEx){Assert.IsTrue(n == 0);}
catch(Exception e) { Assert.IsTrue(false); }
Invoking the method under test with variables, and not constants
Generalized assertions

22. Parameterized Unit Testing
It is supported by all popular testing frameworks – NUnit, MSTest, JUnit, etc.
MSTest uses the annotation [DataTestMethod] for the PUT (parameterized unit test) and different inputs to the parameterized test can be passed as a list of [DataRow(…one set of params to PUT..)] annotations. (see file DivStructurePUT.cs in the project)
PUTs can serve as specifications due to the presence of generalized assertions, which specify functional correctness of a method under test (MUT).
They can be leveraged by (automatic) test input generators, like Pex.

23. Benefit of Using PUTs with Pex
Separation of concerns
PUTs can specify the expected behavior of the MUT using assertions
Pex can focus on the selection of the test inputs with reasonably high coverage
Results in a richer test suite, as assertions are treated as branches
Pex tries to generate inputs which fail a subset of assertions
failing of an assertion in PUT ⇒ an unexpected behavior of the program according to specifications ⇒ Bug
With simple unit tests (non-parameterized), the test cases are generated only according to the MUT logic, which may not reflect the specifications of the MUT at all.