UniTesK: Model Based Testing in Industrial Practice Victor Kuliamin Alexander Petrenko Alexander Kossatchev Igor Burdonov Institute for System Programming Moscow, Russia

Outline Origin of UniTesK Method UniTesK Manifesto UniTesK Solutions Case Studies

Origin of UniTesK Method 1994 – 1996 ISP RAS – Nortel Networks project on functional test suite development for Switch Operating System kernel  Few hundreds of bugs found in the OS kernel, which had been in use for 10 years KVEST technology About 600K lines of Nortel code tested by 2000 BUT: Failed to be introduced in Nortel processes

UniTesK Test Construction System under Test Behavior Model Testing Model Coverage Model Test Input Behavior Correctness Checking

Specification Development Test Development Process Test Quality Criteria Requirements Interface Definition Scenario Development System under Test Mediator Development Test Execution Behavior Specifications Test Scenarios Mediators Test Reports Design Test Results Analysis

Engineering Problems How to simplify transformation of requirements into formal specifications? How to minimize human involvement in test development, but make it possible in necessary points? How to support a large variety of problem domains and test completeness criteria? How to decouple tests and implementation, but keep them working together? How to measure testing quality without implementation? To make easier their usage for ordinary software engineers To make relation between them more clear To increase productivity, but not to loose flexibility To make possible early start of test development To make specifications and tests more reusable

UniTesK Manifesto Maximum simplification of MBT techniques Automation where possible Accommodation to current industrial practice Integration with widely used tools Integration with widely used languages

Tools CTesK

UniTesK Test Architecture Oracle Test action construction System under Test Test Engine Test Action Iterator Specification  ✕ Mediator Test Scenario

Example Client Data Management System: A number of clients Clients can be related as parent-subsidiary  Client can have only one parent, but may have several subsidiaries  No cycles along parent-subsidiary links  When parent is removed, all its subsidiaries should be removed

Example : Interface class ClientManager { Client addClient ( String name ); boolean removeClient ( Client client ); } class Client { boolean addSubsidiary ( Client client ); }

Example : Client Data class Client { String name; ClientSpecification parent = null; HashSet children = new HashSet(); invariant ParentChildLink() { if(parent != null && !parent.children.contains(this)) return false; Iterator j = children.iterator(); while(j.hasNext()) if(((Client)j.next()).parent != this) return false; return true; } invariant NoCyclesAlongLinks() { Client r = this; do { r = r.parent; if(r == this) return false; } while(r != null); return true; }

Example : addClient() Method class ClientManager { specification Client addClient ( String name ) updates clients.? { post { if(name == null || clients.containsKey(name)) { // Client cannot be created return addClient == null && } else { // Client can be created HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null && addClient.name.equals(name) && addClient.parent == null && addClient.children.isEmpty() && clients.get(name) == addClient && }

Test Coverage Criteria branch “Single branch” return post { } if(a || b) branches“Single branch” predicates( a || b )!( a || b ) disjunctsa!a && b!a && !b branch “Single branch” branches “Single branch” disjuncts a!a && !b predicates ( a || b ) !( a || b ) !a && b if( a || b)

class ClientManager { specification Client addClient ( String name ) updates clients.? { post { if(name == null || clients.containsKey(name)) { branch "Client cannot be created"; return addClient == null && } else { branch "Client can be created"; HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null && addClient.name.equals(name) && addClient.parent == null && addClient.children.isEmpty() && clients.get(name) == addClient && } Example : addClient() Method

Automaton from Coverage Goals states parametersoperation domain coverage goals

Implicit Automata Description Implicit specifications cannot be resolved To decrease effort they should not be resolved Huge automata can be described shortly

Test Scenarios User can describe an automaton implicitly in the form of Test Scenario Functions: Provide the current state identifier Provide the next admissible stimulus in this state

Test Scenario : State scenario class ClientManagerTestScenario { ClientManager target; AbstractGenState getGenState() { IntToIntMapGenState genstate = new IntToIntMapGenState(); int n = 0, m = 0; Iterator j = objectUnderTest.clients.keySet().iterator(); while(j.hasNext()) { String s = (String)j.next(); n = Integer.parseInt(s); if(((Client)target.clients.get(s)).parent == null) genstate.setValue(n, 0); else { m = Integer.parseInt(((Client)target.clients.get(s)).parent.name); genstate.setValue(n, m); } return genstate; }

Test Scenario : Stimuli scenario class ClientManagerTestScenario { scenario boolean addClient() { iterate(int i = 0; i < numberOfClients; i++; ) { String name = (i == 0)?(null):("" + i); if( target.precondition_addClient( name ) ) target.addClient( name ); } return true; } scenario boolean removeClient() { iterate(int i = -1; i < numberOfClients; i++; ) { Client client = (i < 0)? ClientMediator.create(new ClientImpl("1")) : (Client)target.clients.get(("" + i)); if( target.precondition_removeClient( client ) ) target.removeClient( client ); } return true; }

Test Execution : First Stage Test Engine Test Scenario

Factorization

Testing Concurrency : Modeling ?a ?{a,b} ?b ?{a,a} ?{b,b} ?{a,a,a} ?{a,a,b} How to model responses on all possible multistimuli? Plain concurrency axiom : reaction on multistimulus is the same as on some sequence of its stimuli

Asynchronous Reaction Specification specification reaction UDPPacket UDPResponse ( ) { pre { return !ModelUDPPackets.isEmpty ( ); } post { return ( UDPResponse ) && !ModelUDPPackets.contains ( UDPResponse ); }

Providing Concurrent Inputs s 11 System under Test s 21 s 31 s 12 s 32 s 13 s 22 s 23 s 14 s 24 s 33 Multisequence is used instead of sequence of stimuli

Collecting Asynchronous Reactions Reactions form a partially ordered set System under Test r 33 r 32 r 12 r 23 r 22 r 11 r 21 r 31 Time

Evaluation of Reactions StimuliReactions Plain concurrency axiom   ✕

UniTesK Test Architecture, 2 Oracle Test input construction System under Test Test Engine Test Action Iterator Mediator Synchronization Manager Reaction Collector Serializer ✕   ✕

UniTesK Tools (C++ Link) Java (C++) / NetBeans, Eclipse (plan) CTesK C / Visual Studio 6.0, gcc C# / Visual Studio.NET 7.1 OTK Specialized tool for compiler testing

Technology Transfer Forms  Training  Pilot projects  Joint projects  Project supervising Successful transfers  Lanit-Tercom CTesK, Feb 2002  Luxoft (IBS group) Jul 2003  Intel OTK, Nov 2003

Case Studies IPv6 implementations  Microsoft Research  Mobile IPv6 (in Windows CE 4.1)  Oktet Intel compilers Web-based client management system in bank Enterprise application development framework Billing system

References 1. V. Kuliamin, A. Petrenko, N. Pakoulin, I. Bourdonov, and A. Kossatchev. Integration of Functional and Timed Testing of Real-time and Concurrent Systems. Proc. of PSI LNCS, Springer-Verlag, V. Kuliamin, A. Petrenko, I. Bourdonov, and A. Kossatchev. UniTesK Test Suite Architecture. Proc. of FME LNCS 2391, pp , Springer-Verlag, A. K. Petrenko, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT’01, India, Pune, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Using Finite State Machines in Program Testing. "Programmirovanije", 2000, No. 2 (in Russian). Programming and Computer Software, Vol. 26, No. 2, 2000, pp (English version) 5. I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications. Proceedings of World Congress of Formal Methods, Toulouse, France, LNCS, No. 1708, 1999, pp

Contacts Victor V. Kuliamin Alexander K. Petrenko , B. Kommunisticheskaya, 25 Moscow, Russia Web: Phone: Fax:

Thank you!