BAiT: Adaptive Test Case Execution in Practice Jens R. Calamé Centrum voor Wiskunde en Informatica, Amsterdam

DWFTT 2007, Enschede2 Agenda 1.Introduction 2.The Toolset for BAiT 3.An Exemplary Case: The ATM 4.Structure of BAiT 5.Conclusion

DWFTT 2007, Enschede3 Introduction BAiT: Behavior Adaptation in Testing Toolset to generate test cases and test oracles as well as execute tests on nondeterministic systems with data

DWFTT 2007, Enschede4 Agenda 1.Introduction 2.The Toolset for BAiT 3.An Exemplary Case: The ATM 4.Structure of BAiT 5.Conclusion

DWFTT 2007, Enschede5 BAiT: Test Case & Oracle Generation Test generation with TGV (enumerative approach) Chaotic data abstraction for input and output parameters of events Test oracle generation for Eclipse Prolog constraint solver in order to select test data Automatic generation of variable names for the test case

DWFTT 2007, Enschede6 BAiT: Test Proxy Generation Test Proxy for Java-based SUTs Instantiation of test object within test proxy Procedure-based bidirectional communication (publisher- subscriber pattern) Mapping of datatypes from specification language to target platform

DWFTT 2007, Enschede7 BAiT: Test Execution Test execution of pre- calculated test trace (behavior and data) Adaptation to the IUT's reactions, rather than immediate assignment of verdict BAiT framework provides adapters for user-defined –trace selection and –test data selection algorithms

DWFTT 2007, Enschede8 Agenda 1.Introduction 2.The Toolset for BAiT 3.An Exemplary Case: The ATM 4.Structure of BAiT 5.Conclusion

DWFTT 2007, Enschede9 An Exemplary Case: The ATM ATM specification (partially)ATM test purpose (completely)

DWFTT 2007, Enschede10 An ATM Test Case Photos: Wikipedia and ECB !getPin(1234)?pinCorrect !getAmount(100) !initBalance(1000)?retFifty(2) ?retFifty(1)?retTwenty(2)?retTen(1) ? !initPin(1234)

DWFTT 2007, Enschede11 An ATM Test Case (2) May 25, :06:19 PM nl.cwi.sen2.bait.steps.TraceImpl merge FINE: Pruning planned test trace. May 25, :06:19 PM nl.cwi.sen2.bait.RunTest findTrace FINER: Examining: [init],initPin(Pin),getPin(PinUser),pinCorrect, initBalance(Bal),getAmount(Amt),[tau],retFifty(Fif),emitBankNotes May 25, :06:19 PM nl.cwi.sen2.bait.steps.TraceImpl solve FINEST: init(G1),initPin(G1,G2,lparam(nat(Pin))),[...] [Pin in {[-Infinity.. Infinity]} => 1234] May 25, :06:21 PM nl.cwi.sen2.bait.steps.TraceImpl solve FINEST: init(G1),initPin(G1,G2,lparam(nat(1234))),[...] [...] May 25, :06:26 PM nl.cwi.sen2.bait.RunTest executeTest INFO: Trying: [init],initPin(1234),getPin(1234),pinCorrect, initBalance(1000),getAmount(100),[tau],retFifty(2),emitBankNotes May 25, :06:26 PM nl.cwi.sen2.bait.RunTest executeTest FINE: initPin(1234) -> OK [...] May 25, :06:26 PM nl.cwi.sen2.bait.RunTest executeTest FINE: retFifty(2) -> NOT OK May 25, :06:26 PM nl.cwi.sen2.bait.RunTest executeTest INFO: Expected: retFifty(2), received: retFifty(1) -> Trace failed, trying alternative. 1. Precalculation of possible test trace 2. Iterative data selection 3. Stepwise execution of the trace… …until the IUT deviates from the precalculated trace.

DWFTT 2007, Enschede12 An ATM Test Case (3) May 25, :06:26 PM nl.cwi.sen2.bait.RunTest executeTest INFO: Trace deviated; executed so far: [init],initPin(1234),getPin(1234),pinCorrect,initBalance(1000), getAmount(100),[tau],retFifty(1),retTwenty(2),retTen(1), emitBankNotes May 25, :06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge FINE: Pruning planned test trace. May 25, :06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge FINE: Adding retFifty(1) to trace stub. [...] May 25, :06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge FINE: Adding emitBankNotes to trace stub. May 25, :06:27 PM nl.cwi.sen2.bait.RunTest executeTest INFO: Test finished; executed: [init],initPin(5),getPin(5), pinCorrect,initBalance(1000),getAmount(100),[tau],retFifty(1), retTwenty(2),retTen(1),emitBankNotes May 25, :06:27 PM nl.cwi.sen2.bait.RunTest executeTest INFO: The test case ended with verdict PASS. 5. Appending received system output to trace. 6. Solve test trace with pending system output; assign verdict or calculate new trace. 4. Pruning not yet executed part of test trace.

DWFTT 2007, Enschede13 Agenda 1.Introduction 2.The Toolset for BAiT 3.An Exemplary Case: The ATM 4.Structure of BAiT 5.Conclusion

DWFTT 2007, Enschede14 Structure of BAiT: The Tools BAiT: Behavior Adaptation in Testing Toolset to generate test cases and test oracles as well as execute tests on nondeterministic systems with data

DWFTT 2007, Enschede15 Structure of BAiT: Execution Part BAiT Execution Core Behavior Mapping Constraint Solver Behavior Mapping Impl. Under Test Data Mapping Constraint Solver Data Mapping Impl. Under Test Impl. Under Test Constraint Solver Trace Selection Data Selection Trace class: trace execution management Step class: communication with IUT Trace selection from complete test graph (IOLTS with all possible test cases for a given test purpose) Test Oracle Test Case(s) Variable class: Codec of data to constraint solver, data instantiator and IUT Data instantiation algorithm(s)

DWFTT 2007, Enschede16 Agenda 1.Introduction 2.The Toolset for BAiT 3.An Exemplary Case: The ATM 4.Structure of BAiT 5.Conclusion

DWFTT 2007, Enschede17 Conclusion Behavior Adaptation in Testing helps finding more reliable verdicts in testing non-deterministic systems A prototypical framework for BAiT has been developed procedure-based setting with atomic data Further work: –Extension of framework for other settings –Extension of framework to structured datatypes –Case Study: Mozilla Firefox CSS subsystem

DWFTT 2007, Enschede18 Further Information Background information Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Sidorova, Natalia; Data Abstraction and Constraint Solving for Conformance Testing (APSEC 2005). Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Towards Automatic Generation of Parameterized Test Cases from Abstractions. Technical Report, The toolset/framework itself Calamé, Jens R.; Adaptive Test Case Execution in Practice. Technical Report, Calamé, Jens R.; The Internals of Adaptive Test Case Execution. Technical Report, to appear. All downloadable from Contact: Jens R. Calamé CWI, Amsterdam