April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 1 Model Program Based Black-Box Testing Margus Veanes Foundations of Software Engineering Microsoft Research, Redmond

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 2 System-level black-box testing ► Requires model of system behavior ► Behavior is often reactive/nondeterministic  Implementation is multi-threaded or distributed  Thread scheduling hard to control ► State space is typically infinite  Objects, unbounded value types ► Traditional FSM-based testing does often not scale

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 3 Model based testing Model Implementation Test OracleTest Cases Are run by Provides actual results for Pass Fail Provides expected results for Generates User Info

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 4 Our approach ► Behavior is described by model programs  Written in Spec# or AsmL  Describe reactive system behavior as model automata  Use alternating simulation for conformance ► Model exploration, validation and model-based testing are provided by the Spec Explorer tool developed at MSR  Supports scenario control  Provides offline as well as online testing  Views test cases as game strategies  Checks conformance violations

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 5 The big picture Model program Constrained model Test cases Pass/Fail Implementation Scenario control Modeling Test generation Test executionCoding Online testing

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 6 Conformance checking component view Constrained model Model IUT wrapper IUT Refinement checker Controllable actions Observable actions Pass/Fail d1 …… c1 Object mapping

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 7 Example: Chat IUT wrapper IUT Ch at ser ver Controllable actions: Observable actions: Login … Send Receive Refinement checker Constrained chat model b.Send(“hi”) m.Send(“hello”) b.Send(“bye”) m.Rcv(“hi”) m.Rcv(“bye”) b.Rcv(“hello”) m.Rcv(“bye”) m.Rcv(“hi”) b.Rcv(“hello”) Bob Marym b Object mapping

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 8 Refinement relation ► Conformance between the model and the system under test is alternating refinement b.Send(“hi”) m.Send(“hello”) SpecSystem b.Send(“hi”) m.Send(“hello”) m.Receive(b,“hi”)b.Receive(m,“hello”) m.Receive(b,“hi”)b.Receive(m,“hello”)

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 9 Example: Valid chat scenario BobServerMary B.Send(“hi”) M.Send(“hello”) M.Receive(B,“hi”) B.Receive(M,“hello”) Time Service should guarantee local consistency of message delivery (FIFO wrt sender) Send is a controllable action Receive is an observable action B.Send(“bye”) M.Receive(B,“bye”)

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 10 Model programs and automata ► States are mappings of variables to values  first-order structures ► Initial state is given by initial assignment to variables ► Actions are defined by method invocations  Actions are either controllable or observable ► An action m(v) is enabled is a state S if the precondition of m(v) is true in S ► An action m(v) transitions from state S 1 to state S 2 if m(v) is enabled in S 1 and its execution in S 1 yields S 2

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 11 Model program : Chat service State: Actions: class Client { bool entered = false; Map > unreceivedMsgs = Map{}; } class Client { void Send(string message) requires entered; { foreach (Client c in enumof(Client), c != this, c.entered) c.unreceivedMsgs[this] += Seq{message}; } void Receive(Client sender, string message) requires sender != this && unreceivedMsgs[sender].Length > 0 && unreceivedMsgs[sender].Head == message; { unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;} … } Controllable: Observable:

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 12 Online testing ► For large reactive systems deriving an exhaustive test-suite is infeasible ► Merge test generation and test execution into one process ► Online testing is a form of model-based stress testing ► Purely random exploration is wasteful  Use scenario control to direct online testing

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 13 Exploration & Scenario control ► Exploration is the process of unfolding a rule- based model program into a transition system  Actions move the system from state to state  State comes from variables containing values  Compound data types are possible (sets, maps, sequences, etc…)  Objects are identities ► Scenario control  Imposes limits on exploration ► Using filters, state groupings, scenario actions, …

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 14 Example Demo: exploration of a coffee machine

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 15 Scenario control ► In general the transition system is infinite, but we can impose limits ► Goal  Create a state space of manageable size that satisfies a given testing goal ► Two main tasks  Restrict action parameter domains to interesting values  Restrict the state space to interesting states ► Note: The two tasks are not necessarily independent!

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 16 Scenario control (cont.) ► The following techniques are used in Spec Explorer  State filters  Stopping conditions  State groupings  Search order  Enabling conditions on actions  Scenario actions ► Usually a combination of all (or some) of the techniques is needed to achieve desired effect

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 17 Example of scenario actions: Chat initialization ► Use a parameterized scenario action Start.  Invoking Start( n ) produces a sequence of n Create() invocations followed by n Enter() invocations [Action(Kind=Scenario)] void Start(int nrOfMembers) requires enumof(Client) == Set{}; { Seq clients = Seq{}; // 1-- The given number of // clients are created for (int i = 0; i < nrOfMembers; i++) clients += Seq{Create()}; // 2-- all clients enter the session for (int i = 0; i < nrOfMembers; i++) clients[i].Enter(); } Start(2) Create()/c0; Create()/c1; c0.Enter(); c1.Enter();

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 18 Test generation ► The result of exploration is a finite transition system with observable and controllable actions : a model automaton A. ► Testing can be viewed as a game between two players  Testing tool (TT) ► Making controllable transitions  Implementation under test (IUT) ► Making observable transitions ► The “rules of the game” are dictated by A ► Tests are generated from A. Test cases are strategies  A strategy tells what move TT should make in a state where its actions are enabled  The purpose of a strategy may be to achieve transition coverage or to reach a certain goal state

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 19 Demo: chat

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 20 Experiences ► Most tool features driven by demands of internal users at MS (mostly testers) ► Models help discover more bugs during modeling (design bugs) than testing  Testers do not get credit for finding those bugs !!! ► During testing models help discover deep system- level bugs where manual test scenarios don’t  Such bugs are hard to understand and fix

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 21 Experiences (cont.) ► Bugs appear in both models and implementations (the ratio is roughly 50-50) ► Code coverage is a poor measure for testing concurrent software, often a single execution thread provides the same coverage. ► New versions of implementation usually require only local changes to models, whereas manual tests often need to be rewritten completely ► The tool is built on.NET but has been used to test distributed C++ applications. Most test harnesses at MS are built in managed code.

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 22 Outlook ► User wish list:  Language independence for modeling ► Use C#, VB, …  Better control over exploration ► Scenario-oriented models  Model composition ► Model features separately and compose the models  Model validation ► Validate properties on models Drives the focus of current research at FSE

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 23 Related foundational work ► Abstract state machines ► Interface automata ► Input-Output COnformance (IOCO) theory ► Markov Decision Process (MDP) theory

April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 24 Thank you! ► FSE web site:  ► Downloading Spec Explorer:  Questions?