ISP RAS Java Specification Extension for Automated Test Development Igor B. Bourdonov, Alexei V. Demakov, Andrei A. Jarov, Alexander S. Kossatchev, Victor V. Kuliamin, Alexander K. Petrenko, Sergei V. Zelenov

Victor Kuliamin ISP RAS 2 Design Goals To preserve and to extend the techniques developed in previous verification projects Automatic generation of test oracles from pre- and postconditions Testing mechanism based of FSM traversal Test coverage metrics definition and tracking Dynamic test optimization for the target coverage To propose a specification based test development process for use in the software industry To provide a specification extension of widely used programming language To simplify the technology

Victor Kuliamin ISP RAS 3 Solutions Proposed Constraint specifications and axioms are both used for functional requirements recording Constraint specifications give rise to test oracles Axioms can be transformed into test scenarios OO architecture is used for specification reuse User should specify branches as the basic test coverage metric based on specifications. Several additional metrics can be extracted automatically. Test sequence can be generated with the help of FSM traversal mechanism. Test scenarios serve as a source for FSM description. Flexible mechanism to bind specification with target component is used

Victor Kuliamin ISP RAS 4 Verification Suite Components Requirements recording AxiomsConstraint SpecificationsTest Coverage MetricsImplementation Test effectiveness measure Test ScenariosFSM Traversal Algorithms Test sequence generation OraclesTest driversTest coverage trackers Handmade componentGenerated componentPrebuilt component DependencyAutomatic derivation Legend

Victor Kuliamin ISP RAS 5 Main Features of State based behavior specifications Object state and invariants Precondition and postcondition of a method Axioms and algebraic specifications Test coverage metrics description Abstraction management Test scenarios executed with the help of FSM based testing machine Flexible OO test suite architecture

Victor Kuliamin ISP RAS 6 Specification of Object State Attributes of class and invariants constitute the specification of state for objects of this class. Example. Specification of a queue with elements having priorities.

Victor Kuliamin ISP RAS 7 Example of Object State Specification public class PriorityQueueSpecification { public List items; // The list of queue elements public List priorities; // The list of elements’ priorities invariant I1() // The lists of elements and priorities have the same sizes { return items.size() == priorities.size(); } invariant I2() // The list of priorities contains only Priority objects { for(int i = 0; i < priorities.size(); i++) if(!(priorities.elementAt(i) instanceof Priority)) return false; return true; } invariant I3() // The values of priorities should not increase { for(int i = 1; i < priorities.size(); i++) if( ((Priority)priorities.elementAt(i)).greater( (Priority)priorities.elementAt(i-1) ) ) return false; return true; }

Victor Kuliamin ISP RAS 8 Method Precondition and Postcondition Method precondition formulates the conditions when this method can be called. Method postcondition gives the criterion of the correct method’s behavior in terms of states of method parameters and other objects before the call and after it. Example. The specifications of enq() method in the queue. The most coarse test coverage is based on branch operators, which make a section of postcondition control flow graph.

Victor Kuliamin ISP RAS 9 Example of Method Behavior Specification specification public synchronized void enq(Object obj) reads obj updates items.? { pre { return obj != null; } // We cannot add null object reference to the queue post { branch “Single branch"; List new_items = means pre-value of new_items.addLastElement(obj); return items.equals(new_items); }

Victor Kuliamin ISP RAS 10 Axioms and Algebraic Specifications Axioms are represented as special methods that return boolean result and can have precondition. Algebraic specifications are represented as special methods with several bodies. They also can have precondition. Example. Axiom and algebraic specification for a stack.

Victor Kuliamin ISP RAS 11 Example of Axiom and Algebraic Specification axiom PushAndSize() { pre { return true; } int oldSize = size(); push(new Object()); return size() == oldSize() + 1; } equivalence Object PushAndPop(Object obj) { pre { return true; } alternative { push(obj); return pop(); } alternative { return obj; } }

Victor Kuliamin ISP RAS 12 Test Coverage Metrics Four predefined test coverage metrics are extracted from the control flow of pre- and postcondition. User can describe his own metrics in addition. Test execution environment keeps track of all the metrics defined and can perform dynamic optimizations for the coverage specified as target. Example. Specification of enq() operation for the queue with additional coverage metrics.

Victor Kuliamin ISP RAS 13 Test Coverage Metrics Example specification public synchronized void enq(Object obj) reads obj updates items.? { pre { return obj != null; } // We cannot add null object reference to the queue post { switch(items.size()) { case 0: mark “Empty queue”; break; case 1: mark “Single element in the queue”; break; default: mark “Several elements in the queue”; break; } branch “Single branch"; List new_items = means pre-value of new_items.addLastElement(obj); return items.equals(new_items); }

Victor Kuliamin ISP RAS 14 Abstraction Management has the following instruments for abstraction management Inheritance of specifications Refinement of specifications Multi-level system of specifications As an example of inheritance we can present specifications of the ordinary queue and the queue with priorities, which has an additional operation enq(Object o) adding an element with minimum priority.

Victor Kuliamin ISP RAS 15 Test Scenarios Test scenarios provide a powerful tool of test design. They are used as a source for test sequence generation. Test scenarios are executed with the help of the FSM traversal mechanism. Example. Test scenario for the queue. It is based of the FSM model of the queue, where different states correspond to different numbers of elements. The test generated from this scenario performs the traversal of the FSM having 10 states and enq() and deq() transitions from every state, except for the initial and last ones. The initial state has only enq() transition, and the last state has only deq() transition.

Victor Kuliamin ISP RAS 16 Example of Test Scenario scenario public class QueueTestScenario { public QueueSpecification queue = new QueueSpecification(); // The object under test public maxNumberOfItems = 10; // The maximum number of elements in // the queue during the test public synchronized AbstractGenState getGenState() // Returns the state object { return new IntGenState(queue.items.size()); // Library class IntGenState is used } scenario public boolean enq() // Subscenario for enq() method { if( queue.items.size() < maxNumberOfItems ) queue.enq(new Object()); return true; } scenario public boolean deq() // Subscenario for deq() method { if( queue.items.size() != 0 ) queue.deq(); return true; }

Victor Kuliamin ISP RAS 17 Test Suite Architecture SpecificationModel Mediator Implementation Test ScenarioTest Engine Oracle Test driver Test coverage trackers Handmade componentGenerated componentPrebuilt component Reference Automatic derivation Test coverage trackers Java Mediator Inheritance Legend

Victor Kuliamin ISP RAS 18 — Based Test Development Framework

Victor Kuliamin ISP RAS 19 References 1. A. K. Petrenko, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT’01, India, Pune, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Using Finite State Machines in Program Testing. "Programmirovanije", 2000, No. 2 (in Russian). Programming and Computer Software, Vol. 26, No. 2, 2000, pp (English version) 3. I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications. Proceedings of World Congress of Formal Methods, Toulouse, France, LNCS, No. 1708, 1999, pp I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin, A. V. Maximov. Testing Programs Modeled by Nondeterministic Finite State Machine. (see [6] white papers)

Victor Kuliamin ISP RAS 20 Contacts Victor V. Kuliamin , B. Kommunisticheskaya, 25 Moscow, Russia. Web: Phone: 007 (095) ext 4422 Fax: 007 (095)