Model-Based Testing Using Spec Explorer Aditya Mathur Purdue University CS 49000 Software Testing Spring 2011 Material extracted mostly from: “Model-Based.

Slides:



Advertisements
Similar presentations
Abstract State Machines, and lessons of an ASM-based project at Microsoft Yuri Gurevich ( Erdos #2 ) Microsoft Research.
Advertisements

CS 267: Automated Verification Lecture 8: Automata Theoretic Model Checking Instructor: Tevfik Bultan.
Partial Order Reduction: Main Idea
Models of Concurrency Manna, Pnueli.
CS4231 Parallel and Distributed Algorithms AY 2006/2007 Semester 2 Lecture 6 Instructor: Haifeng YU.
Introducing Formal Methods, Module 1, Version 1.1, Oct., Formal Specification and Analytical Verification L 5.
A university for the world real R © 2009, Chapter 3 Advanced Synchronization Moe Wynn Wil van der Aalst Arthur ter Hofstede.
Automatic Verification Book: Chapter 6. What is verification? Traditionally, verification means proof of correctness automatic: model checking deductive:
PROTOCOL VERIFICATION & PROTOCOL VALIDATION. Protocol Verification Communication Protocols should be checked for correctness, robustness and performance,
CS 290C: Formal Models for Web Software Lecture 4: Implementing and Verifying Statecharts Specifications Using the Spin Model Checker Instructor: Tevfik.
Rigorous Software Development CSCI-GA Instructor: Thomas Wies Spring 2012 Lecture 13.
CS 267: Automated Verification Lecture 10: Nested Depth First Search, Counter- Example Generation Revisited, Bit-State Hashing, On-The-Fly Model Checking.
ISBN Chapter 3 Describing Syntax and Semantics.
CPSC 668Set 14: Simulations1 CPSC 668 Distributed Algorithms and Systems Spring 2008 Prof. Jennifer Welch.
Models and Security Requirements for IDS. Overview The system and attack model Security requirements for IDS –Sensitivity –Detection Analysis methodology.
1 Introduction to Computability Theory Lecture12: Reductions Prof. Amos Israeli.
61 Nondeterminism and Nodeterministic Automata. 62 The computational machine models that we learned in the class are deterministic in the sense that the.
Penn ESE 535 Spring DeHon 1 ESE535: Electronic Design Automation Day 13: March 4, 2009 FSM Equivalence Checking.
Doctoral course ’Advanced topics in Embedded Systems’. Lyngby'08 Synthesis of Test Purpose Directed Reactive Planning Tester for Nondeterministic Systems.
Software Testing and Quality Assurance
Functional Design and Programming Lecture 10: Regular expressions and finite state machines.
Specification Formalisms Book: Chapter 5. Properties of formalisms Formal. Unique interpretation. Intuitive. Simple to understand (visual). Succinct.
Software Testing and Quality Assurance
THE OBJECT-ORIENTED DESIGN WORKFLOW Statechart Diagrams.
Describing Syntax and Semantics
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 1 Model Program Based Black-Box Testing Margus Veanes Foundations.
Finite State Machines Data Structures and Algorithms for Information Processing 1.
Chapter Seven Advanced Shell Programming. 2 Lesson A Developing a Fully Featured Program.
Texas A&M University Department of Computer Science Sameh S. Sharkawi
DECIDABILITY OF PRESBURGER ARITHMETIC USING FINITE AUTOMATA Presented by : Shubha Jain Reference : Paper by Alexandre Boudet and Hubert Comon.
Foundations of Software Testing Chapter 5: Test Selection, Minimization, and Prioritization for Regression Testing Last update: September 3, 2007 These.
Chapter 14 System Testing.
Modelling III: Asynchronous Shared Memory Model Chapter 9 by Nancy A. Lynch presented by Mark E. Miyashita.
111 Protocols CS 4311 Wirfs Brock et al., Designing Object-Oriented Software, Prentice Hall, (Chapter 8) Meyer, B., Applying design by contract,
Low-Level Detailed Design SAD (Soft Arch Design) Mid-level Detailed Design Low-Level Detailed Design Design Finalization Design Document.
CSC264 Modelling and Computation 10. Modelling State Steve Riddle, John Fitzgerald, Maciej Koutny Computing Science Semester /06.
Microsoft Research, Foundations of Software EngineeringW. Grieskamp et. al: Behavioral Compositions in Symbolic Domains Behavioral Composition in Symbolic.
1 Generating FSMs from Abstract State Machines Wolfgang Grieskamp Yuri Gurevich Wolfram Schulte Margus Veanes Foundations of Software Engineering Microsoft.
CIS 842: Specification and Verification of Reactive Systems Lecture Specifications: LTL Model Checking Copyright , Matt Dwyer, John Hatcliff,
CALTECH CS137 Spring DeHon CS137: Electronic Design Automation Day 9: May 6, 2002 FSM Equivalence Checking.
Course: Software Engineering ©Alessandra RussoUnit 2: States and Operations, slide number 1 States and Operations This unit aims to:  Define: State schemas.
Chapter 12: Design Phase n 12.1 Design and Abstraction n 12.2 Action-Oriented Design n 12.3 Data Flow Analysis n Data Flow Analysis Example n
An Ontological Framework for Web Service Processes By Claus Pahl and Ronan Barrett.
Lecture 18: Object-Oriented Design – Interaction and State Diagrams Anita S. Malik Adapted from Schach (2004) Chapter 12.
7 Systems Analysis and Design in a Changing World, Fifth Edition.
Advanced Topics in Software Engineering Marjan Sirjani Tehran University Faculty of Engineering ECE Department Tehran,
Natallia Kokash (Accepted for PACO’2011) ACG, 31/05/ Input-output conformance testing for channel-based connectors 1.
Formalizing Hardware/Software Interface Specifications
2. Regular Expressions and Automata 2007 년 3 월 31 일 인공지능 연구실 이경택 Text: Speech and Language Processing Page.33 ~ 56.
Copyright © Curt Hill Finite State Machines The Simplest and Least Capable Automaton.
Programming Languages and Design Lecture 3 Semantic Specifications of Programming Languages Instructor: Li Ma Department of Computer Science Texas Southern.
CIS 540 Principles of Embedded Computation Spring Instructor: Rajeev Alur
CS1Q Computer Systems Lecture 11 Simon Gay. Lecture 11CS1Q Computer Systems - Simon Gay 2 The D FlipFlop The RS flipflop stores one bit of information.
Systems Analysis and Design in a Changing World, Fourth Edition
CS3773 Software Engineering Lecture 06 UML State Machines.
ISP RAS Java Specification Extension for Automated Test Development Igor B. Bourdonov, Alexei V. Demakov, Andrei A. Jarov, Alexander S. Kossatchev, Victor.
Program Correctness. The designer of a distributed system has the responsibility of certifying the correctness of the system before users start using.
Chapter 5 Finite Automata Finite State Automata n Capable of recognizing numerous symbol patterns, the class of regular languages n Suitable for.
Teooriapäevad, Vanaõue, Sept Synthesis of Test Purpose Directed Reactive Planning Tester for Nondeterministic Systems 1 Jüri Vain, Kullo Raiend,
Cliquez pour modifier le style du titre Cliquez pour modifier les styles du texte du masque Deuxième niveau Troisième niveau Quatrième niveau Cinquième.
Requirements Techniques, cont.
Marlon Dumas Institute of Computer Science
Executable Specifications: Foundations  MS Tools
Instructor: Rajeev Alur
Model-based Software Testing and Analysis with C#
UML Activity Diagrams & State Charts
CSCE 668 DISTRIBUTED ALGORITHMS AND SYSTEMS
Marlon Dumas Institute of Computer Science
Protocols CS 4311 Wirfs Brock et al., Designing Object-Oriented Software, Prentice Hall, (Chapter 8) Meyer, B., Applying design by contract, Computer,
ISpec: A Compositional Approach to Interface Specification
Presentation transcript:

Model-Based Testing Using Spec Explorer Aditya Mathur Purdue University CS Software Testing Spring 2011 Material extracted mostly from: “Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer”, Margus Veanes, Colin Campbell, Wolfgang Grieskamp, Wolfram Schulte, Nikolai Tillmann, and Lev Nachmanson, Published by: Springer Verlag, Lecture Notes in Computer Science, Volume 4949, Pages 39-76, 2007.

Objective The purpose of this presentation is to introduce modeling using the Spec Explorer tool from Microsoft. The example presented here is from a paper cited in the title slide. Familiarity with Chapter 3 of the textbook is assumed. 2Spec Explorer

Model Based Conformance Testing RequirementsImplementation Test harness (send inputs, receive outputs, and compare) Model Generate (manual) Tests Generate (automated) Test outcome 3Spec Explorer

Example: Chat Room Chat Room Client 4Spec Explorer

Chat Room: Operation Each client may post text messages. Each message is delivered to all clients logged into the chat room. Pending messages from a client are delivered in the order sent. Messages from multiple senders are interleaved arbitrarily. 5Spec Explorer

Client status // Client entered the chat room or not bool entered; // Queue of messages sent by other clients but not received by this client Map > unreceivedMsgs; 6Spec Explorer

Client Actions: Creator Create a new instance of a client. State changes so that Empty message queues between the new client and the previously created clients. 7Spec Explorer

Client Actions: Creator // Create a client [Action] Client() { this.unreceivedMsgs = Map; foreach (Client c in enumof(Client), c != this){ c.unreceivedMsgs[this] = Seq{}; // Empty sequence this.unreceivedMsgs[c] = Seq{}; } entered = false; } enumof(T): set of instances of type T in the current state. Denotes an action in the abstract state mach ine. 8Spec Explorer

Client Actions: Enter Changes the state of a client to indicate that it has entered the chat room. 9Spec Explorer

Client Actions: Enter // Model client entry into the chat room [Action] void Enter() requires !entered; { entered = true; } Method pre-condition 10Spec Explorer

Client Actions: Send Appends a new message to the queue of unreceived messages in all clients in the chat room. 11Spec Explorer

Client Actions: Send message // Send a message [Action] void Send(string message) requires entered; { foreach (Client c in enumof(Client), c != this, c.entered) c.unreceivedMsgs[this] += Seq{message}; } 12Spec Explorer

Client Actions: Receive Extracts a message sent from a given sender from the sender’s queue in the client. 13Spec Explorer

Client Actions: Receive [Action(Kind=ActionAttributeKind.Observable)] void Receive(Client sender, string message) requires sender != this && unreceivedMsgs[sender].Length > 0 && unreceivedMsgs[sender].Head == message; { unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail; } 14Spec Explorer

Client Model Program class Client { bool entered; Map > unreceivedMsgs; [Action] Client() [Action] void Enter() [Action] void Send(string message) [Action(Kind=ActionAttributeKind.Observable)] } 15Spec Explorer

Action types Controllable: Input by the user client, send, enter Observable: Output from the system receive 16Spec Explorer

Client Model Scenario Start Passive state: Active state: Timeout 17Spec Explorer

Model Programs in Spec Explorer Finite set of actions or update rules (Acts); e.g. client, send, enter, receive Vocabulary  function symbols State: values, or interpretations, of state vocabulary symbols Execution of an action method in a given state leads to the next state where some state variables may have changed. State variables: V in  ; (e.g., entered, unreceivedMsgs) Each action is associated with a pre- and a post-condition. 18Spec Explorer

FSM and Model Automaton FSM=(X, Y, Q, q 0, δ, O), where X is a set of input symbols, Y a set of output symbols, q0 in Q, δ is transition function and O the output function Model automaton=(Q, Q 0, Q f, δ, A ), where Q is a set of states, Q 0 is a set of initial states, Q f is a set of final states, and A is a set of actions, A=Ctrl U Obs, Ctrl is a set of control actions and Obs is a set of observable actions, Ctrl Obs = empty U Spec Explorer uses the notion of Model Automata: 19Spec Explorer

Model program and model automaton A model automaton is a complete unwinding of the model program. Exploration: Unlike an FSM with a given sets of nodes and arcs, the states and transitions of a model program must be deduced by executing sequences of atomic actions starting in the initial state. Model programModel automaton unwind 20Spec Explorer

Exploration for the Chat Example In state s 0, the precondition is true and hence Client constructor is invoked. The dynamic universe Client is updated by the addition of client c0. This is denoted by enumof(Client). The new state is denoted as s 1. The transition explored is δ(s 0, Client/c0)=s 1 21Spec Explorer

Accepting state Needed particularly in testing distributed and multithreaded programs. Why? A state is considered an accepting state if the accepting condition is true in that state. A test is allowed to terminate in an accepting state. An action the execution of which takes the implementation to a state where no actions are enabled is known as a succeed action. It forces the system into an accepting state. 22Spec Explorer

Accepting condition example enumof(Client).Size > 0 && // Exclude the initial state, and Forall{ c in enumof(Client), s in c.unreceivedMsgs.Keys; c.unreceivedMsgs[s].Length == 0 // states where pending messages have not been received. } A state that satisfies the above condition has no observable actions enabled. 23Spec Explorer

Scenario Control A model program may correspond to a large, or infinite state, automaton. Techniques: Parameter selection, method restriction, state filtering, directed search, state grouping Techniques are available to control the size of a model automata for a specific test purpose. 24Spec Explorer

Scenario Control: Parameter selection Select (s, m, v) for each state s, action m, and v sets of tuples. In Chat example: send has an implicit parameter this and explicit parameter message. These can be restricted using the pair: Set {(c in enumof(Client)); } Restrictions by triples may lead to reduction in the number of transitions and hence a smaller automaton. 25Spec Explorer

Scenario Control: Method restriction An action m is enabled in state s if all pre-conditions associated with m are satisfied. In Chat example: restriction: Clients send messages only after all configured clients are created and have entered the system enum Mode { Creating, Entering, Sending }; Mode CurrentMode { get { if (enumof(Client).Size < 2) return Mode.Creating; if(Set{cin enumof(Client),!c.entered;c}.Size<2) return Mode.Entering; return Mode.Sending } Strengthening the pre-conditions can be used to limit the scenarios. 26Spec Explorer

Scenario Control: Method restriction In Chat example: restriction: Clients send messages only after all configured clients are created and have entered the system enum Mode { Creating, Entering, Sending }; Mode CurrentMode { get { if (enumof(Client).Size < 2) return Mode.Creating; if(Set{c in enumof(Client),!c.entered;c}.Size<2) return Mode.Entering; return Mode.Sending } Enabling of actions can now be restricted using expressions such as currentMode==Mode.Creating; 27Spec Explorer

Scenario Control: State filtering A state filter is a set S f, where S init is in S f. [The subscript f stands for filter, and not for final.] In Chat example: Using state filter avoid states in which the same message is posted more than once before it is received. Forall{c in enumof(Client), s in c.unreceivedMsgs.Keys, m1 in c.unreceivedMsgs[s], m2 in c.unreceivedMsgs[s]; m1 != m2} 28Spec Explorer A transition from state s to state t is included in the automaton if t is in S f. S f is specified using a state based expression.

Test Generation Offline: Generate tests in advance from the model. 29Spec Explorer Online: Generate tests on the fly as testing progresses.

Test Suite Test suite T: Is another automaton generated from an automaton M. 30Spec Explorer States S T in T may use new state variables (test variables). Test variables make it possible to record test history; e.g., which states have been traversed so far. It contains two new methods called test actions: Observe and Timeout. Transitions corresponding to test actions are called test transitions. The Observe action encodes a decision to wait for an observable action. The Timeout action indicates that no other observable action happened. An accepting state is reachable from every state in S T.

Test Suite: Example Consider the following model program P: 31Spec Explorer enum Mode = {A,B,C} Mode mode = A; void F() requires mode == A {mode = B;} void G() requires mode == B {mode = C;} void H() requires mode == B {mode = C;} void I() requires mode == C {mode = A;} // Added to P to create P’. Accepting state: Where mode is C. Exploration: M generated from P and M’ from P’.

Test Suite: Generate Test automaton 32Spec Explorer Add a test variable n to indicate test number. T: (F, G) and (F, H )

Automaton Traversal algorithms 33Spec Explorer Algorithm(s) used in Spec Explorer: T covers all states in M T covers all transitions in M Each action is associated with a weight and cost using a state-based expression. Tests are generated to optimize the expected cost of a test.

Summary 34Spec Explorer Spec Explorer : Allows the creation of a model program P that captures the expected behavior(s) of the implementation under test (IUT). Generates one or more model automaton (M) from P using exploration subjected to scenario restrictions. Generates a test suite T from M either offline or online.