Analysis of Complex Data Structures with PEX Vadim Mutilin Evgeniy Gerlits Vladimir Fedotov Mentor: Patrice Godefroid

Microsoft Pex White box test generation for.NET Implements DART (Directed Automated Random Testing): combines concrete and symbolic executions Push button approach 2

Complex Data Structures Examples: List, Queue, Map, Tree, Graph 3

Motivation Nikolai Tillmann, et al. Test Input Generation for Programs with Pointers. TACAS Does push button approach really work on complex data structures? – class MyList private T value add(T value) void removeAT(int index) T getItem(int index) 4

Project Goals Understanding limits – Documentation – Authors – Concrete examples Making out suggestions – Usage patterns – Improvements 5

Push Button Approach 6

Push Button (Results) MyContains 6/6 (100.00%), 103 Runs MyContains 7

Push Button (MyList) 8

Push Button (Results) 9 Data StructureAlgorithmPush button Graph dijkstra7/23 (122) SkipList filterPrimes12/20 (3) BinarySearchTree search8/9 (4) List search9/9 (109) LinkedListbubble_sort10/10 (31) covered/total basic blocks (test runs)

Factory Approach 10 Pex generates template for Factory method User adds Factory method which generates the inputs Factory method may be parameterized

Factory (Results) Data StructureAlgorithm Push button Factory Graph dijkstra7/23 (122)7/23 (85) SkipList filterPrimes12/20 (3)20/20 (116) BinarySearchTree search8/9 (4)9/9 (9) List search9/9 (109) LinkedListbubble_sort10/10 (31)10/10 (44) 11 covered/total basic blocks (test runs)

Pool Approach Collection of concrete input objects Collection is created manually by the tester Pex iterates indexes Implementation is called with concrete objects by their indexes 12

Pool (Results) Data StructureAlgorithmPush buttonFactoryPool Graph dijkstra7/23 (122)7/23 (85)23/23 (6) SkipList filterPrimes12/20 (3)20/20 (116) BinarySearchTree search8/9 (4)9/9 (9) List search9/9 (109) LinkedListbubble_sort10/10 (31)10/10 (44) 13 covered/total basic blocks (test runs)

Relaxing Visibility Constraints 14 Can be done with annotation

Conclusion Pex limitations are still unclear 2 bugs reported to the Pex team Suggested – Pool approach – Relaxing visibility constraints with annotations Promising topic for further research 15