1. Feedback-Based Specification, Coding and Testing… …with JWalk Anthony J H Simons, Neil Griffiths and Christopher D Thomson

2. Overview Lazy Systematic Unit Testing  JWalk testing tool (Simons) The JWalkEditor tool  Integrated Java editor and JWalk (Griffiths) Feedback-based methodology  Prototype, validate, specify, test Evaluation of cost-effectivness  Testing effort, time, coverage (Thomson) http://www.dcs.shef.ac.uk/~ajhs/jwalk/

3. Motivation State of the art in agile testing  Test-driven development is good, but…  …no specification to inform the selection of tests  …manual test-sets are fallible (missing, redundant cases)  …reusing saved tests for conformance testing is fallible – state partitions hide paths, faults (Simons, 2005) Lazy systematic testing method: the insight  Complete testing requires a specification (even in XP!)  Infer an up-to-date specification from a code prototype  Let tools handle systematic test generation and coverage  Let the programmer focus on novel/unpredicted results

4. Lazy Systematic Unit Testing Lazy Specification  late inference of a specification from evolving code  semi-automatic, by static and dynamic analysis of code with limited user interaction  specification evolves in step with modified code Systematic Testing  bounded exhaustive testing, up to the specification  emphasis on completeness, conformance, correctness properties after testing, repeatable test quality http://en.wikipedia.org/wiki/Lazy_systematic_unit_testing

5. JWalk Testing Tool Lazy systematic unit testing for Java  static analysis - extracts the public API of a compiled Java class  protocol walking (code exploration) - executes all interleaved methods to a given path depth  algebraic testing (memory states) - validates all observations on all mutator-method sequences  state-based testing (high-level states) - validates all state- transitions (n-switch coverage) for inferred high-level states http://www.dcs.shef.ac.uk/~ajhs/jwalk/

6. JWalkEditor Integration of JWalk with a Java-editor Editing features  Java-sensitive  Integrated with JDK compiler tools – track exceptions to source Testing features  Invokes JWalk (6 validation/testing modes)  Confirm/reject key results via dialogs  Browse test result-sets via tabbed panes

7. Snapshot – Editing a Stack Syntax sensitive text highlights Full build cycle, multiple sources Set JWalk test parameters

8. Snapshot –Testing a Stack Tabbed pane with test results for the Empty state, for all paths of depth 2 Colour-coded test outcomes Colour-coded test sequences

9. Dynamic Analysis of States Memory states  generate all interleaved method paths to depth 1..n  prune sequences ending in observers from the active edges, preserving mutator sequences  distinguish observer/mutator methods by empirical low-level state comparison (extracted by reflection) High-level states  generate all mutator-method paths (as above)  evaluate all state predicates, eg: isEmpty(), isFull()  seek states corresponding to the product of boolean outcomes: viz: {Default, Empty, Full, Full&Empty}

10. Test Result Prediction Strong prediction  From known results, guarantee further outcomes in the same equivalence class  eg: test sequences containing observers in the prefix map onto a shorter sequence  target.push(e).size().top() == target.push(e).top() Weak prediction  From known facts, guess further outcomes; an incorrect guess will be revealed in the next cycle  eg: methods with void type usually return no result, but may raise an exception  target.pop() predicted to have no result  target.pop().size() == -1 reveals an error

11. Feedback-based Methodology Coding  The programmer prototypes a Java class in the editor Validation  JWalk systematically explores method paths, providing useful instant feedback to the programmer Specification  JWalk infers a specification, building a test oracle based on key test results confirmed by the programmer Testing  JWalk tests the class to bounded exhaustive depths, based on confirmed and predicted test outcomes  JWalk uses state-based test generation algorithms

12. Example – Library Book Validation  surprise: target.issue(“a”).issue(“b”).getBorrower() == “b”  violates business rules: fix code to raise an exception Testing  all observations on chains of issue(), discharge()  n-switch cover on states {Default, OnLoan} public class LibraryBook { private String borrower; public LibraryBook(); public void issue(String); public void discharge(); public String getBorrower(); public Boolean isOnLoan(); }

13. Extension – Reservable Book Validation  surprise: target.reserve(“a”).issue(“b”).getBorrower() == “b”  violates business rules: override issue() to refuse “b” here. Testing  all obs. on chains of issue(), discharge(), reserve(), cancel()  n-switch cover on states {Default, OnLoan, Reserved, Reserved&OnLoan} public class ReservableBook extends LibraryBook { private String requester; public ReservableBook(); public void reserve(String); public void cancel(); public String getRequester(); public Boolean isReserved(); }

14. Evaluation User Acceptance  programmers find JWalk habitable  they can concentrate on creative aspects (coding) while JWalk handles systematic aspects (validation, testing) Cost of Confirmations  not so burdensome, since amortized over many test cycles  metric: measure amortized confirmations per test cycle Comparison with JUnit  propose a common testing objective for manual and lazy systematic testing; evaluate coverage and testing effort  Eclipse+JUnit vs. JWalkEditor: given the task of testing the “transition cover + all equivalence partitions of inputs”

15. Amortized Interaction Costs  number of new confirmations, amortized over 6 test cycles  con = manual confirmations, > 25 test cases/minute  pre = JWalk’s predictions, eventually > 90% of test cases Test classa1a2a3s1s2s3 LibBk con357005 LibBk pre2818 38133 ResBk con3145601183 ResBk pre62789362411649 eg: algebra-test to depth 2, 14 new confirmations eg: state-test to depth 2, 241 predicted results

16. Comparison with JUnit manual testing method  Manual test creation takes skill, time and effort (eg: ~20 min to develop manual cases for ReservableBook)  The programmer missed certain corner-cases  eg: target.discharge().discharge() - a nullop?  The programmer redundantly tested some properties  eg: assertTrue(target != null) - multiple times  The state coverage for LibraryBook was incomplete, due to the programmer missing hard-to-see cases  The saved tests were not reusable for ReservableBook, for which all-new tests were written to test new interleavings

17. Advantages of JWalk JWalk lazy systematic testing  JWalk automates test case selection - relieves the programmer of the burden of thinking up the right test cases!  Each test case is guaranteed to test a unique property  Interactive test result confirmation is very fast (eg: ~80 sec in total for 36 unique test cases in ReservableBook)  All states and transitions covered, including nullops, to the chosen depth  The test oracle created for LibraryBook formed the basis for the new oracle for ReservableBook, but…  JWalk presented only those sequences involving new methods, and all interleavings with inherited methods

18. Speed and Adequacy of Testing  Test goal: transition cover + equiv. partitions of inputs  manual testing expensive, redundant and incomplete  JWalk testing very efficient, close to complete eg: wrote 104 tests, 21 were effective and 83 not! eg: JWalk achieved 100% test coverage Test classTTETE TRTR Adeqtime min.sec LibBk manual3192290%11.00 ResBk manual104218353%20.00 LibBk jwalk10 0100%0.30 ResBk jwalk36 090%0.46

19. Conclusion Performance of JWalk testing  clearly outperformed manual testing  coverage based on all states and transitions  input equivalence partitions are not yet handled Performance of JWalkEditor  unexpected gain: automatic validation of prototype code  c.f. Alloy’s model checking from a partial specification Moral for testing  just automatically executing saved tests is not so great  need systematic test generation tools to get coverage  automate the parts that humans get wrong!

20. Any Questions? http://www.dcs.shef.ac.uk/~ajhs/jwalk/

21. Bibliography A J H Simons, JWalk: a tool for lazy systematic testing of Java classes by introspection and user interaction, Automated Software Engineering, 14 (4), December, ed. B. Nuseibeh, (Springer, USA, 2007), 369-418. SpringerLink: DOI 10.1007/s10515-007-0015-3, 8 September, 2007. Final draft version also deposited with White Rose Research Online.JWalk: a tool for lazy systematic testing of Java classes by introspection and user interaction Final draft versionWhite Rose Research Online A J H Simons and C D Thomson, Lazy systematic unit testing: JWalk versus JUnit, Proc 2nd. Testing in Academia and Industry Conference - Practice and Research Techniques, 22-24 September, eds. P McMinn and M Harman, (Cumberland Lodge, Windsor Great Park: IEEE, 2007), 138. See also the A1 poster presenting this result.Lazy systematic unit testing: JWalk versus JUnitA1 poster A J H Simons and C D Thomson, Benchmarking effectiveness for object-oriented unit testing, Proc 1st. Software Testing Benchmark Workshop, 9-11 April, eds. M Roper and W M L Holcombe, (Lillehammer: ICST/IEEE, 2008).Benchmarking effectiveness for object-oriented unit testing A J H Simons, N Griffiths and C D Thomson, Feedback-based specification, coding and testing with JWalk, Proc 3rd. Testing in Academia and Industry Conference - Practice and Research Techniques, 29-31 August, eds. L. Bottacci and G. M. Kapfhammer and M. Roper, (Cumberland Lodge, Windsor Great Park: IEEE, 2008), to appear.Feedback-based specification, coding and testing with JWalk A J H Simons, A theory of regression testing for behaviourally compatible object types, rev. and ext., Software Testing, Verification and Reliability, 16 (3), UKTest 2005 Special Issue, September, eds. M Woodward, P. McMinn, M. Holcombe, R. Hierons (London: John Wiley, 2006), 133-156.A theory of regression testing for behaviourally compatible object types A J H Simons, Testing with guarantees and the failure of regression testing in eXtreme Programming, Proc. 6th Int. Conf. on eXtreme Programming and Agile Processes in Software Engineering (XP 2005), eds. H Baumeister et al., Lecture Notes in Computer Science, 3556, (Berlin: Springer Verlag, 2005), 118-126.Testing with guarantees and the failure of regression testing in eXtreme ProgrammingProc. 6th Int. Conf. on eXtreme Programming and Agile Processes in Software Engineering Wikipedia entry for JWalkJWalk Wikipedia entry for Lazy Systematic Unit TestingLazy Systematic Unit Testing