1. Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection Anthony J H Simons and Wenwen Zhao

2. Overview Lazy Systematic Unit Testing  JWalk testing concept and methodology The JWalk 1.0 toolset  JWalkTester, JWalkUtility, JWalkEditor, etc. Dynamic analysis and pruning  extending earlier work to full algebraic analysis Comparison and evaluation  measure path pruning, before and after  test result prediction, before and after http://www.dcs.shef.ac.uk/~ajhs/jwalk/

3. Motivation State of the art in agile testing  Test-driven development is good, but…  …no specification to inform the selection of tests  …manual test-sets are fallible (missing, redundant cases)  …reusing saved tests for conformance testing is fallible – state partitions hide paths, faults (Simons, 2005) Lazy systematic testing method: the insight  Complete testing requires a specification (even in XP!)  Infer an up-to-date specification from a code prototype  Let tools handle systematic test generation and coverage  Let the programmer focus on novel/unpredicted results

4. Lazy Systematic Unit Testing Lazy Specification  late inference of a specification from evolving code  semi-automatic, by static and dynamic analysis of code with limited user interaction  specification evolves in step with modified code Systematic Testing  bounded exhaustive testing, up to the specification  emphasis on completeness, conformance, correctness properties after testing, repeatable test quality http://en.wikipedia.org/wiki/Lazy_systematic_unit_testing

5. JWalk 1.0 Toolset JWalk Tester JWalk Utility JWalk Editor JWalk Marker JWalk Grapher JWalk SOAR

6. JWalk Tester Lazy systematic unit testing for Java  static analysis - extracts the public API of a compiled Java class  protocol walk (all paths) – explores, validates all interleaved methods to a given path depth  algebra walk (memory states) – explores, validates all observations on all mutator-method sequences  state walk (high-level states) – explores, validates n-switch transition cover for all high-level states http://www.dcs.shef.ac.uk/~ajhs/jwalk/ Try me

7. Baseline Approaches Breadth-first generation  all constructors and all interleaved methods (eg JCrasher, DSD-Crasher, Jov)  generate-and-filter (eg Rostra, Java Pathfinder) by state equivalence class Computational cost  exponential growth, memory issues, wasteful over- generation, even if filtering is later applied  #paths = Σ c.m k, for k = 0..n Key: c = #constructors, m = #methods, k = depth

8. Dynamic Pruning Interleaved analysis  generate-and-evaluate, pruning active paths on the fly (eg JWalk, Randoop)  remove redundant prefix paths after each test cycle, don’t bother to expand in next cycle Increasing sophistication  prune prefix paths ending in exceptions (fail again)  JWalk, Randoop (2007)  and prefixes ending in algebraic observers (unchanged)  JWalk 0.8 (2007)  and prefixes ending in algebraic transformers (reentrant)  JWalk 1.0 (2009)

9. Prune Exceptions… new push top pop push top pop push top pop push top pop Key:novel state exception top poptop pop top pop top poptop pop push Prune error-prefixes (JWalk0.8, Randoop)

10. …and Observers new push top pop push top pop push top pop push top pop Key:novel state exception unchanged state push top pop push top pop Prune error- and observer-prefixes (JWalk0.8)

11. Algebraic Pruning new push top pop top push top pop push top pop Key:novel state exception unchanged state reentrant state Prune error-, observer- and transformer-prefixes (JWalk1.0)

12. What is the Same State? Some earlier approaches  distinguish observers, mutators by signature (Rostra)  intrusive state equality predicate methods (ASTOOT)  external (partial) state equality predicates (Rostra)  subsumption of execution traces in JVM (Pathfinder) Some algebraic approaches  shallow, deep equality under all observers (TACCLE)  but assumes observations are also comparable  very costly to compute from first principles  serialise object states and hash (Henkel & Diwan)  but not all objects are serialisable  no control over depth of comparison

13. Smart State Inspection Reflection-and-hash  extract state vector from objects  compute hash code for each field  order-sensitive combination hash code Proper depth control  shallow or deep equality settings, to chosen depth  hash on pointer, or recursively invoke algorithm Fast state comparison  each test evaluation stores posterior state code  fast comparison with preceding, or all prior states  possible to detect unchanged, or reentrant states

14. Pruning: Stack Stackbaselineexcept.observ.algebr. 01111 17777 2433113 32591392519 415556674325 5933133917931 Pruned: 9,300 redundant paths Retained: 31 significant paths (best 0.33%) Table 1: Cumulative paths explored after each test cycle

15. Pruning: Reservable Book ResBookbaselineexcept.observ.algebr. 01111 19999 273 25 35855614933 4468141859741 537449memex16941 Pruned: 37,408 redundant paths Retained: 41 significant paths (best 0.12%) Table 2: Cumulative paths explored after each test cycle

16. Test Result Prediction Semi-automatic validation  the user confirms or rejects key results  these constitute a test oracle, used in prediction  eventually > 90% test outcomes predicted JWalk test result prediction rules  eg: predict repeat failure  new().pop().push(e) == new().pop()  eg: predict same state  target.size().push(e) == target.push(e)  eg: predict same result  target.push(e).pop().size() == target.size() Try me

17. Kinds of Prediction Strong prediction  From known results, guarantee further outcomes in the same equivalence class  eg: observer prefixes empirically checked before making any inference, unchanged state is guaranteed  target.push(e).size().top() == target.push(e).top() Weak prediction  From known facts, guess further outcomes; an incorrect guess will be revealed in the next cycle  eg: methods with void type usually return no result, but may raise an exception  target.pop() predicted to have no result  target.pop().size() == -1 reveals an error

18. Test Confirmation – JWalk 0.8 new push top pop push top pop push top pop push top pop Key:confirm result confirm error predicted result push top pop push top pop Confirm all observations, errors on all state- modifying paths

19. Test Confirmation – JWalk 1.0 new push top pop top push top pop push top pop Confirm all observations, errors on all primitive algebraic constructions Key:confirm result confirm error predicted result

20. Confirmations: Stack Stackv0.8 algv0.8 prov1.0 algv1.0 pro 01-1- 15-5- 24-4- 39-4- 412-4+4 526-4+8 Total57 2234 Table 3: Confirmations per test cycle (new oracle) JWalk 0.8: trained oracle after 57 confirmations JWalk 1.0: trained oracle after 34 confirmations

21. Confirmations: Reservable Book ResBookv0.8 algv0.8 prov1.0 algv1.0 pro 01-1- 12-2- 28-8- 312-6- 430-6+20 540memex- Total93 2343 Table 4: Confirmations per test cycle (inherited oracle) JWalk 0.8: trained oracle after 93 confirmations JWalk 1.0: trained oracle after 43 confirmations

22. Why Residual Confirmations? Prediction based on state equality  from state equivalence:  target.push(e).pop() == target  predict identical observations:  target.push(e).pop().size() == target.size() Novel states occur in longer protocols  JWalk has deterministic argument synthesis:  elements generated in order: e1, e2, … e n  algebraic reduction yields a novel state:  target.push(e1).pop().push(e2) == target.push(e2)  target.push(e2) != target.push(e1) from the oracle

23. Conclusions … Test path pruning  algebraic analysis effective at eliminating redundant paths  absolutely necessary when testing classes with large APIs  java.lang.Character: c = 1, m = 78; d3 base = 480,715 paths; alg = 79 paths, stable after 1 cycle  java.lang.String: c = 13, m = 64; d3 base = 54,093 paths; alg = 845 paths, stable after 1 cycle More test automation  presents user with the ideal mimimal test-set for judgement  user only has to confirm all errors and observations on all primitive algebraic constructions

24. Conclusions Faster state exploration  algebra-walking finds the leaves of the algebra-tree faster  state-walking discovers high-level states faster, by growing only primitive state-modifying paths  can afford to search to greater test depths Test result prediction  algebraic anaylsis improves predictive power as expected  but oracle must also have the reduction (and may not)  future idea: axiom generalisation? (Henkel & Diwan)

25. Thank You! And thanks also to:  Wenwen Zhao – hashing on states for comparison  Mihai-Gabriel Glont – prototype UI for JWalkTester  Arne-Michael Toersel – case studies for JWalk http://www.dcs.shef.ac.uk/~ajhs/jwalk/ Let’s go JWalking!

26. Confirmations: Library Book LibBookv0.8 algv0.8 prov1.0 algv1.0 pro 01-1- 12-2- 23-3- 32--- 43--+3 52--- Total13 69 Table 5: Confirmations per test cycle (new oracle) JWalk 0.8: depth-5 oracle after 13 confirmations JWalk 1.0: depth-5 oracle after 9 confirmations