Automated Model-Based Testing of Hybrid Systems Michiel van Osch PROSE January 25, 2007 13.

Slides:



Advertisements
Similar presentations
1 Lars Frantzen, Pieter Koopman, René de Vries, Tim Willemse, Jan Tretmans Radboud University Nijmegen © Jan Tretmans Radboud University Nijmegen Testing.
Advertisements

CSCE 668 DISTRIBUTED ALGORITHMS AND SYSTEMS Fall 2011 Prof. Jennifer Welch CSCE 668 Set 14: Simulations 1.
Model Checking for an Executable Subset of UML Fei Xie 1, Vladimir Levin 2, and James C. Browne 1 1 Dept. of Computer Sciences, UT at Austin 2 Bell Laboratories,
Hybrid Systems Presented by: Arnab De Anand S. An Intuitive Introduction to Hybrid Systems Discrete program with an analog environment. What does it mean?
Supervisory Control of Hybrid Systems Written by X. D. Koutsoukos et al. Presented by Wu, Jian 04/16/2002.
UPPAAL Andreas Hadiyono Arrummaisha Adrifina Harya Iswara Aditya Wibowo Juwita Utami Putri.
Prototyping of Real-time Component Based Systems by the use of Timed Automata Trevor Jones Lancaster University, UK
MotoHawk Training Model-Based Design of Embedded Systems.
An Automata-based Approach to Testing Properties in Event Traces H. Hallal, S. Boroday, A. Ulrich, A. Petrenko Sophia Antipolis, France, May 2003.
CPSC 668Set 14: Simulations1 CPSC 668 Distributed Algorithms and Systems Spring 2008 Prof. Jennifer Welch.
Introduction Designing cost-sensitive real-time control systems for safety-critical applications requires a careful analysis of the cost/fault-coverage.
Model-based Testing of Hybrid Systems Michiel van Osch IPA Spring Days on Testing 19 April – 21 April 2006.
Model Checking. Used in studying behaviors of reactive systems Typically involves three steps: Create a finite state model (FSM) of the system design.
Behavioral Design Outline –Design Specification –Behavioral Design –Behavioral Specification –Hardware Description Languages –Behavioral Simulation –Behavioral.
1 Jan Tretmans Embedded Systems Institute Eindhoven Radboud University Nijmegen Model-Based Testing.
EECS 20 Lecture 1 (January 17, 2001) Tom Henzinger Motivation.
1 Jan Tretmans University of Nijmegen © Jan Tretmans University of Nijmegen Model Based Testing Property Checking for Real.
Design of Fault Tolerant Data Flow in Ptolemy II Mark McKelvin EE290 N, Fall 2004 Final Project.
1 Jan Tretmans Radboud University Nijmegen (NL) © Jan Tretmans Radboud University Nijmegen together with: University of Twente Enschede.
Designing Predictable and Robust Systems Tom Henzinger UC Berkeley and EPFL.
1 Adapting BPEL4WS for the Semantic Web The Bottom-Up Approach to Web Service Interoperation Daniel J. Mandell and Sheila McIlraith Presented by Axel Polleres.
Software Testing and QA Theory and Practice (Chapter 10: Test Generation from FSM Models) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory.
Cheng/Dillon-Software Engineering: Formal Methods Model Checking.
1 Jan Tretmans Embedded Systems Institute Eindhoven, NL Radboud University Nijmegen, NL Model-Based Testing with Labelled Transition.
1 EVALUATING INTELLIGENT FLUID AUTOMATION SYSTEMS USING A FLUID NETWORK SIMULATION ENVIRONMENT Ron Esmao - Sr. Applications Engineer, Flowmaster USA.
REAL-TIME SOFTWARE SYSTEMS DEVELOPMENT Instructor: Dr. Hany H. Ammar Dept. of Computer Science and Electrical Engineering, WVU.
ECE 720T5 Winter 2014 Cyber-Physical Systems Rodolfo Pellizzoni.
Protocol Architectures. Simple Protocol Architecture Not an actual architecture, but a model for how they work Similar to “pseudocode,” used for teaching.
FROM CONTROL SOFTWARE TO CONTROL SYSTEMS Wojciech Kurek.
The State of Hybrid Model-Based Testing Michiel van Osch
Formal Model for Simulations Instructor: DR. Lê Anh Ngọc Presented by – Group 6: 1. Nguyễn Sơn Hùng 2. Lê Văn Hùng 3. Nguyễn Xuân Hậu 4. Nguyễn Xuân Tùng.
Modeling Process CSCE 668Set 14: Simulations 2 May be several algorithms (processes) runs on each processor to simulate the desired communication system.
Testing with Formal Methods Ed Brinksma course 2004 A Formal Framework.
Fleetronic International easy and affordable telematics Kris Van Uffelen january 2008 This document is owned by.
Model Based Conformance Testing for Extensible Internet Protocols Anastasia Tugaenko Scientific Adviser: Nikolay Pakulin, PhD.
Benjamin Gamble. What is Time?  Can mean many different things to a computer Dynamic Equation Variable System State 2.
Slide No. 1 Course: Logic Design Dr. Ali Elkateeb Topic: Introduction Course Number: COMP 1213 Course Title: Logic Design Instructor: Dr. Ali Elkateeb.
CIS 842: Specification and Verification of Reactive Systems Lecture Specifications: Sequencing Properties Copyright , Matt Dwyer, John Hatcliff,
High Performance Embedded Computing © 2007 Elsevier Lecture 3: Design Methodologies Embedded Computing Systems Mikko Lipasti, adapted from M. Schulte Based.
Real Time Event Based Communication Team Abhishekh Padmanabhan CIS 798 Final Presentation.
Conformance Test Suites, Extensionally Arend Rensink University of Twente Dutch Workshop on Formal Testing Techniques University of Twente 13 September.
REAL-TIME SOFTWARE SYSTEMS DEVELOPMENT Instructor: Dr. Hany H. Ammar Dept. of Computer Science and Electrical Engineering, WVU.
Natallia Kokash (Accepted for PACO’2011) ACG, 31/05/ Input-output conformance testing for channel-based connectors 1.
Fall 2004EE 3563 Digital Systems Design EE 3563 VHSIC Hardware Description Language  Required Reading: –These Slides –VHDL Tutorial  Very High Speed.
Communicating Real-Time State Machines (CRSM) State machines that communicate synchronously Unique unidirectional channels are used for the communication.
Conformance Test Experiments for Distributed Real-Time Systems Rachel Cardell-Oliver Complex Systems Group Department of Computer Science & Software Engineering.
1 Black-box conformance testing for real-time systems Stavros Tripakis VERIMAG Joint work with Moez Krichen.
Electrical and Computer Engineering University of Cyprus LAB 1: VHDL.
Control Theory Control System Objectives  Establish a final condition  Provide safe operation  Eliminate the human element  Assure economical operation.
1 Model Checking of Robotic Control Systems Presenting: Sebastian Scherer Authors: Sebastian Scherer, Flavio Lerda, and Edmund M. Clarke.
1 RealProct: Reliable Protocol Conformance Testing with Real Nodes for Wireless Sensor Networks Junjie Xiong, Edith C.-Ngai, Yangfan Zhou, Michael R. Lyu.
CS4730 Real-Time Systems and Modeling Fall 2010 José M. Garrido Department of Computer Science & Information Systems Kennesaw State University.
Abstract A Structured Approach for Modular Design: A Plug and Play Middleware for Sensory Modules, Actuation Platforms, Task Descriptions and Implementations.
Automatic Testing of Neighbor Discovery Protocol Based on FSM and TTCN Zhiliang Wang, Xia Yin, Haibin Wang, Jianping Wu Department of Computer Science.
Towards Interoperability Test Generation of Time Dependent Protocols: a Case Study Zhiliang Wang, Jianping Wu, Xia Yin Department of Computer Science Tsinghua.
S IMULATION AND R EAL - TIME PROCESSING Chapter 3.7 VCN 9691 Computing Notes 1.
1 Fault tolerance in distributed systems n Motivation n robust and stabilizing algorithms n failure models n robust algorithms u decision problems u impossibility.
Winter 2007SEG2101 Chapter 121 Chapter 12 Verification and Validation.
TEMPLATE DESIGN © Crawling is the process of automatically exploring a web application to discover the states of the application.
Test Generation for Input/Output Transition Systems Ed Brinksma Course 2004.
© 2009 Artisan Software Tools. All rights reserved. Testing Solutions with UML/SysML Andrew Stuart, Matthew Hause.
Copyright 1999 G.v. Bochmann ELG 7186C ch.1 1 Course Notes ELG 7186C Formal Methods for the Development of Real-Time System Applications Gregor v. Bochmann.
CSCE 668 DISTRIBUTED ALGORITHMS AND SYSTEMS
RealProct: Reliable Protocol Conformance Testing with Real Nodes for Wireless Sensor Networks Junjie Xiong
Event-Based Architecture Definition Language
Discrete Controller Synthesis
Develop a Reliability Test in TTCN-3
A test generation framework for quiescent real-time systems
Presentation transcript:

Automated Model-Based Testing of Hybrid Systems Michiel van Osch PROSE January 25,

2 Motivation Hybrid Systems Testing might be expensive, dangerous, or resources might be limited Discrete and real-time model-based testing does not test the continuous aspects of the system

3 Content Part I: Theory –Model-based Testing –Input-Output Conformance & Discrete Tests –Hybrid Systems –Hybrid Input-output Conformance –Hybrid Tests –Results Part II:Tool –Test Architecture –Specification –Tester –The Connection with the Implementation Under Test –Adapter –Limitations and Future Work –Case Study: Vacuum Control

4 Model-Based Testing model SUT IUT conforms to model pass fail test tool test generation tool test execution tool IUT passes tests IUT conf model    soundexhaustive

5 Input-output Conformance for Discrete Systems Impl. ioco Spec. iff for all traces α: out(Impl. after α)  out(Spec. after α) If there is an output action from state s then out(s) = {o in O| s →} else out(s) = {δ} Furthermore, out(S) = U s  S out(s)

6 Test-case Generation and Execution 1.Terminate with verdict pass 2.Select an input from the specification and apply it to the implementation 3.Observe an output or a timeout from the implementation and check if it is allowed according to the specification

7 Example s1s0 s2 s3 ?Activate ?Button2 ?Button1 !XLCoffee !Coffee t0 t1 ?Activate t3 ?Button2 t2 ! δ Fail !Coffee!XLCoffee Pass !XLCoffee Fail !Coffee ! δ

8 Hybrid Systems In Practice: –Discrete behavior plus continuous behavior –Continuous behavior can be input observed through sensors or output generated by actuators In Theory: –Discrete actions plus flow of continuous variables (trajectories) –Variables can be input variables and output variables –Hybrid Transition Systems

9 Hybrid Systems (Output Only) Time Coffee 0 ?Button1 ?Button2 s1s0 s2 s3 ?Activate ?Button2 ?Button1 Coffee’ = 4 cl/sec. Δt = 8 sec. Coffee’ = 3 cl/sec. Δt = 5 sec. Coffee’ = 0 Δt = 1 sec. Coffee’ = 0 Δt = 1 sec..

10 Hybrid Systems (Including Input) Water s1s0 s2 s3 ?Activate ?Button2 ?Button1 Water’ = 4 cl/sec. Coffee’ = Water’ Δt = 8 sec. Water’ = 3 cl/sec. Coffee’ = Water’ Δt = 5 sec. Water’ = 0 Coffee’ = 0 Δt = 1 sec. Water’ = 0 Coffee’ = 0 Δt = 1 sec..

11 Hybrid Conformance For every reachable state, the set of output actions possible by the implementation is a subset of the set of output actions possible by the specification For every reachable state, the set of trajectories possible by the implementation is a subset of the set of trajectories possible by the specification In contrast to ioco, no quiescence action because there is always continuous output.

12 Continuous Output Only Impl. is input-output conform a Spec. iff for all traces α: out(Impl. after α)  out(Spec. after α) and traj(Impl. after α)  traj(Spec. after α) s1s0 s2 s3 ?Activate ?Button2 ?Button1 Coffee’ = 4 cl/sec. Δt = 8 sec. Coffee’ = 3 cl/sec. Δt = 5 sec. Coffee’ = 0 Δt = 1 sec. Coffee’ = 0 Δt = 1 sec..

13 With Continuous Input The implementation is input enabled (for both discrete behavior and continuous behavior). We do not require the specification to be input complete. Impl. is input-output conform a Spec. iff for all traces α: out(Impl. after α)  out(Spec. after α) and traj(Impl. after α)  traj(Spec. after α) Does not work!! Solution: Look at the trajectories of the Implementation with respect to the trajectories of input variables of the Specification

14 Hybrid Conformance (Continuous Input plus Output) infilter(traj(Impl. after α), traj(Spec. after α))  traj(Spec. after α) s1 s3 ?Button2 Water’ = 0 Coffee’ = 0 Δt = 1 sec.. s4 Water’ = 0 Coffee’ = 0 Δt = 3 sec.. s5 !Out of Cups Water’ = 0 Coffee’ = 0 Δt = 1 sec.. u1 u3 ?Button2 Water’ = 0 Coffee’ = 0 Δt = 1 sec.. u4 Water’ = 0 Coffee’ = 0 Δt = 3 sec.. u5 !Out of Cups Water’ = 0 Coffee’ = 0 Δt = 1 sec.. Specification Implementation Water’ = 0 Coffee’ = 0 Δt = 1 sec.. Still does not work because …

15 Hybrid Conformance (continuous input plus output) If there is a trajectory from state s then out(s) = {o in O| s →}  {ξ} else out(s) = {o in O| s →} Impl. hioco Spec. iff for all traces α: out(Impl. after α)  out(Spec. after α) and infilter(traj(Impl. after α), traj(Spec. after α))  traj(Spec. after α)

16 Hybrid Tests A Special kind of Hybrid Transition Systems: Tree like structure Two terminal states: pass and fail Deterministic for actions Strongly time deterministic for trajectories

17 Hybrid Tests 1.Terminate with verdict pass 2.Select an input from the specification and apply it to the implementation s1s0 ?Activate Water’ = 0 Coffee’ = 0 Specification t0 t1 ?Activate Test

18 Hybrid Test-case Generation 3.If an output action has to happen immediately according to the specification then observe an output action and check if it is allowed according to the specification or let time pass by selecting and applying and observing a trajectory Fail s4s5 !”Out of Cups” Coffee’ = 0 Water’ = 0 Δt = 1 Specification t4 Pass !”Out of Cups” Test-Case Coffee’ = 0 Water’ = 0 Δt = 1

19 Hybrid Test-case Generation 4.Select an input trajectory from the specification, apply it to the implementation and observe the output trajectory simultaneous, possibly interrupted by an output action. Specification s1 Water’ = 0 Coffee’ = 0 Δt = 1 Test t1 t2 Water’ = 0 Coffee’ = 0 Δt = 1 Fail !”out of cups” Water’ = 0 Coffee’ = 3 Δt = 1

20 Results A hybrid conformance theory Proven Sound and exhaustive A Natural extension of discrete and timed conformance theories

21 Content Part I: Theory –Model-based Testing –Input-Output Conformance & Discrete Tests –Hybrid Systems –Hybrid Input-output Conformance –Hybrid Tests –Results Part II: Tool –Test Architecture –Specification –Tester –The Connection with the Implementation Under Test –Adapter –Limitations and Future Work –Case Study: Vacuum Control

22 Libraries Tester Architecture Specification: The Model from which Tests are Generated Tester: Implements The Test Algorithm and Gives the Verdict Adapter: Translated Input/Output from Model to a format suitable for the Implementation Under Test and vice versa Medium: The Interface between Tester and Implementation IUT: The Implementation Under Test Spec Tester Adapter Medium IUT

23 Libraries Specification Needs to: Model Discrete behavior and Continuous Behavior Make Distinction between Input Actions, Output Actions, and Internal Actions Make Distinction between Input Variables, Output Variables and Internal Variables Model in an Intuitive way Medium Adapter Spec Tester IUT

24 Libraries Specification Medium Adapter Spec Tester proc Control(cont V: real, chan h,out: real)= |[ *(V h!!1.0; out!!1.0 ; V >= 10 -> h!!0.0; out!!0.0) ]| proc Env(cont V: real, chan h: real)= |[ var n: real = 0.0 :: V’=3.0*n | *(h?n) ]| model Spec()= |[ cont V: real = 10.0, chan h,out: real :: Control(V,h,out)|| Env(V, h) ]| IUT

25 Libraries Tester Implements: On the Fly Test Generation –Select Input from Specification –Apply Input –Observe Output –Compare the Observed Output with the Output allowed by the Specification –Give a Verdict or Continue Test Medium Adapter Spec Tester IUT

26 On the Fly Testing proc ControlS(cont V: real, chan h: real)= |[ var n: real = 0.0 :: V’=3.0*n | *(V n:=1.0; h!!1.0 ; V >= 10 -> n:=0.0; h!!0.0) ]| model Spec()= |[ cont V: real = 10.0, chan h: real :: ControlS(V,h) ]| V=10 V=2 V’=3.0* Δt = 8 sec. Select Input ( χ )(Manually/ Automatic) V=2 h!!1.0 pass fail h!!0.0 Compare Values ( χ, Maple) Pass Give Verdict (with trace) Continue Apply (Via adapter) Pressure’= -1.0 mbar/sec Δt = 8 sec. IUT Pump OFF Observe (Via adapter) IUT

27 Additional Libraries χ –stepper for computing sets of allowed transitions and current state of the specification E.g. Maple for comparing observed continuous output (samples) with specified trajectories and comparing observed discrete output values with specified send actions Libraries Medium Adapter Spec Tester IUT

28 Libraries The Connection Medium Jabber χ Model TCP/IP Labview Controller WiresElectronics Buttons/ Sensors Robot Arm Adapter Spec Tester IUT

29 Libraries Tester The Adapter Implements –Mapping of Variables/Actions of Specification to a Implementation and vice versa (e.g. channels to function calls, or variables to wires) –Translating Input/Output of Specification to Implementation and vice versa (e.g. functions to samples, or signals) Medium Adapter Spec IUT

30 Limitations and Future Work This is just a prototype, there are shortcomings! –Real Time Testing is Not Possible Yet –The complexity of Continuous behavior is limited by the Hybrid χ –stepper implementation. E.g. currently only standard differential equations. –Models are not ‘ideal’ for testing. E.g. in case of identifying input and output –For performance reasons we only deal with deterministic specifications. –We assume that the communication medium is reliable Adaptation of theory for Sampling and Inaccuracy Case Studies

31 Real Time Generating and applying input (e.g. samples) Observing output and Time at which output Occurred in the Implementation

32 Limitations and Future Work This is just a prototype, there are shortcomings! –Real Time Testing is Not Possible Yet –The complexity of Continuous behavior is limited by the Hybrid χ –stepper implementation. –Models are not ‘ideal’ for testing. –For performance reasons we only deal with deterministic specifications. –We assume that the communication medium is reliable Adaptation of theory for Sampling and Inaccuracy Case Studies

33 The Vacuum Case Lithography Process takes place in vacuum Waferstepper has Five Chambers Chambers are kept in Vacuum by a system of Pumps and Valves Pumps and Valves are Controlled by Software (discrete) Software observes Pressure in Chambers through Sensors (continuous)

34 Activities Modeling Hardware in Hybrid χ and Stand Alone Simulation Modeling (translating) Hardware in discrete (timed) χ and Integration with Software Controller Modeling (translating) in Uppaal for Model Checking Testing Models and Software Controller with the Hybrid Tester

35 Questions?