1. UniTesK: Model Based Testing in Industrial Practice Victor Kuliamin Alexander Petrenko Alexander Kossatchev Igor Burdonov Institute for System Programming Moscow, Russia

2. Outline Origin of UniTesK Method UniTesK Manifesto UniTesK Solutions Case Studies

3. Origin of UniTesK Method 1994 – 1996 ISP RAS – Nortel Networks project on functional test suite development for Switch Operating System kernel  Few hundreds of bugs found in the OS kernel, which had been in use for 10 years KVEST technology About 600K lines of Nortel code tested by 2000 BUT: Failed to be introduced in Nortel processes

4. UniTesK Test Construction System under Test Behavior Model Testing Model Coverage Model Test Input Behavior Correctness Checking

5. Specification Development Test Development Process Test Quality Criteria Requirements Interface Definition Scenario Development System under Test Mediator Development Test Execution Behavior Specifications Test Scenarios Mediators Test Reports Design Test Results Analysis

6. Engineering Problems How to simplify transformation of requirements into formal specifications? How to minimize human involvement in test development, but make it possible in necessary points? How to support a large variety of problem domains and test completeness criteria? How to decouple tests and implementation, but keep them working together? How to measure testing quality without implementation? To make easier their usage for ordinary software engineers To make relation between them more clear To increase productivity, but not to loose flexibility To make possible early start of test development To make specifications and tests more reusable

7. UniTesK Manifesto Maximum simplification of MBT techniques Automation where possible Accommodation to current industrial practice Integration with widely used tools Integration with widely used languages

8. Tools J@T CTesK Ch@se

15. UniTesK Test Architecture Oracle Test action construction System under Test Test Engine Test Action Iterator Specification  ✕ Mediator Test Scenario

16. Example Client Data Management System: A number of clients Clients can be related as parent-subsidiary  Client can have only one parent, but may have several subsidiaries  No cycles along parent-subsidiary links  When parent is removed, all its subsidiaries should be removed

17. Example : Interface class ClientManager { Client addClient ( String name ); boolean removeClient ( Client client ); } class Client { boolean addSubsidiary ( Client client ); }

18. Example : Client Data class Client { String name; ClientSpecification parent = null; HashSet children = new HashSet(); invariant ParentChildLink() { if(parent != null && !parent.children.contains(this)) return false; Iterator j = children.iterator(); while(j.hasNext()) if(((Client)j.next()).parent != this) return false; return true; } invariant NoCyclesAlongLinks() { Client r = this; do { r = r.parent; if(r == this) return false; } while(r != null); return true; }

19. Example : addClient() Method class ClientManager { specification Client addClient ( String name ) updates clients.? { post { if(name == null || clients.containsKey(name)) { // Client cannot be created return addClient == null && clients.equals(@clients.clone()); } else { // Client can be created HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null && addClient.name.equals(name) && addClient.parent == null && addClient.children.isEmpty() && clients.get(name) == addClient && oldClients.equals(@clients.clone()); }

20. Test Coverage Criteria branch “Single branch” return post { } if(a || b) branches“Single branch” predicates( a || b )!( a || b ) disjunctsa!a && b!a && !b branch “Single branch” branches “Single branch” disjuncts a!a && !b predicates ( a || b ) !( a || b ) !a && b if( a || b)

21. class ClientManager { specification Client addClient ( String name ) updates clients.? { post { if(name == null || clients.containsKey(name)) { branch "Client cannot be created"; return addClient == null && clients.equals(@clients.clone()); } else { branch "Client can be created"; HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null && addClient.name.equals(name) && addClient.parent == null && addClient.children.isEmpty() && clients.get(name) == addClient && oldClients.equals(@clients.clone()); } Example : addClient() Method

22. Automaton from Coverage Goals states parametersoperation domain 1 2 3 coverage goals

23. Implicit Automata Description Implicit specifications cannot be resolved To decrease effort they should not be resolved Huge automata can be described shortly

24. Test Scenarios User can describe an automaton implicitly in the form of Test Scenario Functions: Provide the current state identifier Provide the next admissible stimulus in this state

25. Test Scenario : State scenario class ClientManagerTestScenario { ClientManager target; AbstractGenState getGenState() { IntToIntMapGenState genstate = new IntToIntMapGenState(); int n = 0, m = 0; Iterator j = objectUnderTest.clients.keySet().iterator(); while(j.hasNext()) { String s = (String)j.next(); n = Integer.parseInt(s); if(((Client)target.clients.get(s)).parent == null) genstate.setValue(n, 0); else { m = Integer.parseInt(((Client)target.clients.get(s)).parent.name); genstate.setValue(n, m); } return genstate; }

26. Test Scenario : Stimuli scenario class ClientManagerTestScenario { scenario boolean addClient() { iterate(int i = 0; i < numberOfClients; i++; ) { String name = (i == 0)?(null):("" + i); if( target.precondition_addClient( name ) ) target.addClient( name ); } return true; } scenario boolean removeClient() { iterate(int i = -1; i < numberOfClients; i++; ) { Client client = (i < 0)? ClientMediator.create(new ClientImpl("1")) : (Client)target.clients.get(("" + i)); if( target.precondition_removeClient( client ) ) target.removeClient( client ); } return true; }

27. Test Execution : First Stage Test Engine Test Scenario

28. Factorization

29. Testing Concurrency : Modeling ?a ?{a,b} ?b ?{a,a} ?{b,b} ?{a,a,a} ?{a,a,b} How to model responses on all possible multistimuli? Plain concurrency axiom : reaction on multistimulus is the same as on some sequence of its stimuli

30. Asynchronous Reaction Specification specification reaction UDPPacket UDPResponse ( ) { pre { return !ModelUDPPackets.isEmpty ( ); } post { return (@ModelUDPPackets.clone()).contains ( UDPResponse ) && !ModelUDPPackets.contains ( UDPResponse ); }

31. Providing Concurrent Inputs s 11 System under Test s 21 s 31 s 12 s 32 s 13 s 22 s 23 s 14 s 24 s 33 Multisequence is used instead of sequence of stimuli

32. Collecting Asynchronous Reactions Reactions form a partially ordered set System under Test r 33 r 32 r 12 r 23 r 22 r 11 r 21 r 31 Time 11 12 2131 2232 23 33

33. Evaluation of Reactions StimuliReactions Plain concurrency axiom   ✕

34. UniTesK Test Architecture, 2 Oracle Test input construction System under Test Test Engine Test Action Iterator Mediator Synchronization Manager Reaction Collector Serializer ✕   ✕

35. UniTesK Tools J@T (C++ Link) Java (C++) / NetBeans, Eclipse (plan) CTesK C / Visual Studio 6.0, gcc Ch@se C# / Visual Studio.NET 7.1 OTK Specialized tool for compiler testing

36. Technology Transfer Forms  Training  Pilot projects  Joint projects  Project supervising Successful transfers  Lanit-Tercom CTesK, Feb 2002  Luxoft (IBS group) J@T, Jul 2003  Intel OTK, Nov 2003

37. Case Studies IPv6 implementations - 2001-2003  Microsoft Research  Mobile IPv6 (in Windows CE 4.1)  Oktet Intel compilers - 2001-2003 Web-based client management system in bank Enterprise application development framework Billing system

38. References 1. V. Kuliamin, A. Petrenko, N. Pakoulin, I. Bourdonov, and A. Kossatchev. Integration of Functional and Timed Testing of Real-time and Concurrent Systems. Proc. of PSI 2003. LNCS, Springer-Verlag, 2003. 2. V. Kuliamin, A. Petrenko, I. Bourdonov, and A. Kossatchev. UniTesK Test Suite Architecture. Proc. of FME 2002. LNCS 2391, pp. 77-88, Springer-Verlag, 2002. 3. A. K. Petrenko, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT’01, India, Pune, 2001 4. I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Using Finite State Machines in Program Testing. "Programmirovanije", 2000, No. 2 (in Russian). Programming and Computer Software, Vol. 26, No. 2, 2000, pp. 61-73 (English version) 5. I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications. Proceedings of World Congress of Formal Methods, Toulouse, France, LNCS, No. 1708, 1999, pp. 608- 621 6. http://www.ispras.ru/groups/rv/rv.html http://www.ispras.ru/groups/rv/rv.html

39. Contacts Victor V. Kuliamin Alexander K. Petrenko E-mail: kuliamin@ispras.ru, petrenko@ispras.ru 109004, B. Kommunisticheskaya, 25 Moscow, Russia Web: http://www.ispras.ru/groups/rv/rv.html Phone: 007-095-9125317 Fax: 007-095-9121524

40. Thank you!