SDRL & RTG University of Pennsylvania 5/24/01 1 Run-time Monitoring and Checking Based on Formal Specifications Insup Lee Department of Computer and Information Science University of Pennsylvania 24 May 2001 Joint work with M. Kim, M. Viswanathan, S. Kannan, and O. Sokolsky

SDRL & RTG University of Pennsylvania 5/24/01 2 Objectives Specification and verification –complete analysis, all behaviors are correct –gap between abstract model and implementation Testing –tested behaviors are correct –not complete Run-time behavior checking –consistency between abstract model and implementation To provide a framework for automatic generation of monitors and checkers

SDRL & RTG University of Pennsylvania 5/24/01 3 Fundamental Issues How does a monitor gather information from a running system? How does the monitor relate to requirements? How do we integrate dynamic monitoring with static analysis? Can monitor be used to steer a system? What mathematical guarantees do monitors provide?

SDRL & RTG University of Pennsylvania 5/24/01 4 Monitorable Properties Run-time monitoring and checking checks whether properties are violated or not by observing only finite traces execution at run-time A class of monitorable properties is a Turing computable subset of safety properties –Safety property: finite trace reveals violations. –Monitorable property: violations Turing computable. Equivalent to class  1 in the Arithmetic Hierarchy Liveness Safety MEDL = Monitorable property Properties on Traces Safety closure of the halting problem

SDRL & RTG University of Pennsylvania 5/24/01 5 System Spec System Spec Requirement Spec Requirement Spec Formal verification Design System Implementation System Implementation Monitoring Script Monitoring Script Implementation Checker/ Corrector Checker/ Corrector System Filter Communication Run-time Check MaCS Framework Event Handler Event Handler Corrector Checker

SDRL & RTG University of Pennsylvania 5/24/01 6 Design Issues Filter –passive versus active –when to take snapshot Event Handler –mapping between concrete state and abstract event Checker –inclusion based on trace, ready semantics, bisimulation Corrector –how to provide feedback

SDRL & RTG University of Pennsylvania 5/24/01 7 Overview of MaCS framework Based events and conditions Two types of abstraction –time abstraction by instrument filter (IF) –data abstraction by event recognizer (ER) Scripting languages –MEDL (Meta Event Definition Language): property specification language –PEDL (Primitive EDF): Implementation-language dependent Goals: Automatic generation of IF, ER, Checker, Steerer

SDRL & RTG University of Pennsylvania 5/24/01 8 Events and Conditions Must be able to reason about both time instants and durations in a program execution. –Events and conditions are a natural division, which is also found in other formalisms like SCR. Need temporal operators combining events and conditions in order to reason about traces. CrecyPoitiersAgincourtstart(war)end(war)

SDRL & RTG University of Pennsylvania 5/24/01 9 Two-Sorted Logic Conditions interpreted over 3 values: true, false and undefined. [E,E) pairs a couple of events to define an interval. start and end define the events corresponding to the instant when conditions change their value. Primitive events and conditions

SDRL & RTG University of Pennsylvania 5/24/01 10 An example Example: correct = oldDateUsed => (f().num-retries = 1) – If old data is used then number of retries must be 1 – f().num-retries is a local variable of function f() invoke f()f().num-retries = 1 oldDataUsedreturnoldDataNotUsed correct is true, undefined, and true again.

SDRL & RTG University of Pennsylvania 5/24/01 11 MaCS prototype architecture Program (Java byte code) Monitoring Script (PEDL) Requirements (MEDL) PEDLCompiler MEDLCompiler Instrumented Code Filter Generator (JTREK) Instrumentation Information Compiled PEDL Compiled MEDL Event Recognizer Checker Steering Script (SADL) SADLCompiler Instrumentation Information Injector class (Java byte code)

SDRL & RTG University of Pennsylvania 5/24/01 12 MEDL Meta Event Definition Language (MEDL) Expresses requirements using the events and conditions, sent by event recognizer. Describes the safety requirements of the system, in terms of conditions that must always be true, and alarms (events) that must never be raised. –property periodic = (period == 1000) –alarm staleData = oldDataUsed when connected Auxiliary variables may be used to store history. –request_info -> { num_hits’ = num_hits+1; hit_ratio’ = num_hits/num_fails; } Used to automatically create checker

SDRL & RTG University of Pennsylvania 5/24/01 13 Structure of MEDL script ReqSpec HexPattern // imported events, conditions, and actions import event MAValert, init import action collapse, restore // variable declarations var int numMAValert, prevCount1, prevCount2 var float prevAverage, currAverage // definitions of events event endPeriod = start(time(MAValert)-periodStart > 3000) alarm NoPattern = start(currAverage > prevAverage*1.1+10) // guarded variable updates and action invocations updateAvg -> { currAverage’= (prevCount2+prevCount1+numMAValert)/3;} NoPattern -> { invoke collapse; } End

SDRL & RTG University of Pennsylvania 5/24/01 14 PEDL Primitive Event Definition Language (PEDL) PEDL describes a thin interface between low-level implementation and high-level requirement –The language maps the low-level state information of the system to high-level events used in describing the requirements. –Fast event recognition is key point Provides primitives to refer to values of variables and to certain points in the execution of the program Depends on system implementation language Used to create filter and event recognizer

SDRL & RTG University of Pennsylvania 5/24/01 15 Primitive Event Definition Language (PEDL) Information about the system comes in two different forms: –Conditions, which are true or false for a finite duration of time (e.g., is variable x >5?), and –Events, which are either present or absent at some instant of time (e.g., is the control right now at the end of method f?) Provides primitives to refer to values of variables and to certain points in the execution of the program. –condition IC = (50<train_position) && (train_position<100); –Event endGD = startM(Gate.gu()); Provides primitive “time” to refer to time when events happen –condition slowTrain = (time(endIC)-time(startIC)) > 3000;

SDRL & RTG University of Pennsylvania 5/24/01 16 Structure of PEDL Script MonScr MAVpattern // exported events and conditions export event MAValert, init; // Overhead Reduction Options [timestamp;] [valueabstract;] [deltaabstract;] [multithread;] // monitored object declarations monmeth void Console.createMAVs(int); monobj double MAV.run().distance; // definitions of events and conditions event init = startM( Console.createMAVs(int) ); event MAValert = start(MAV.run().distance > 40 && MAV.run().distance < 120) ; End

SDRL & RTG University of Pennsylvania 5/24/01 17 The Current MaCS Prototype System MaCS instruments Java bytecode, not a source code. Filter resides in the host of target program as a separate thread. The filter sends updated value and time stamp to the event recognizer. The event-recognizer (ER) evaluates condition and event description ER sends evaluation result to the run-time checker. MaCS works on multi-threaded applications

SDRL & RTG University of Pennsylvania 5/24/01 18 MaCSware Version 0.99 Components –Static components PEDL/MEDL/SADL compiler Bytecode Instrumentor –Dynamic components Filter/Injector, event recognizer and run-time checker GUI

SDRL & RTG University of Pennsylvania 5/24/01 19 Related Work Monitoring in process-level by input/output/message interface instrumentation –Supervisor [T.Savor98], JEM[G.Liu99],MOTEL[Log00] Monitoring in statement-level by target system instrumentation –ANNA [S.Sankar93] checks assertion for annotated Ada program –Java Runtime Timing Constraint Monitor [A.Mok97] monitors Java program using specification language RTL –ALAMO [C.Jeffery98] instruments C source code and monitors it Monitoring in instruction-level by environment instrumentation –Dynascope [R. Sosic95], JPDA [JVM99]

SDRL & RTG University of Pennsylvania 5/24/01 20 Current and future work Extend MaCS framework –Monitoring of hybrid systems –Distributed monitoring Extend MaC prototype implementation –Monitoring and checking frequency optimization –Add program checking capability –Add steering capability Develop MaC formal foundations Integrate with other tools –Network simulator –Test generator –Charon

SDRL & RTG University of Pennsylvania 5/24/01 21 Automatic Test Generation from Formal Specifications Insup Lee Department of Computer and Information Science University of Pennsylvania 24 May 2001 Joint work with H.S. Hong and O. Sokolsky

SDRL & RTG University of Pennsylvania 5/24/01 22 Specification-based Testing SpecificationSpec(I) Determines whether an implementation conforms to its specification –Hardware and protocol conformance testing –FSM and LTS For each test case (I, Spec(I)), –Apply I to the implementation –Observe the set of output sequences Imp(I) –Check Imp(I)  Spec(I) Implementation I Imp(I)

SDRL & RTG University of Pennsylvania 5/24/01 23 Specifications Specification-based test generation for reactive, real- time, and hybrid systems –EFSM FSM + data variables –Statecharts, hierarchical reactive modules EFSM + hierarchy + concurrency + communication –CHARON, hybrid systems EFSM + hierarchy + concurrency + communication + differential equations

SDRL & RTG University of Pennsylvania 5/24/01 24 Test Coverage Criteria for EFSMs Structural information in EFSMs (application-independent) –Control flow: state or transition coverage –Data flow: all-definition, all-use, or all-def-use-path Properties to be validated (application-dependent) –MEDL scripts, Scenarios, MSCs, temporal logics empty nonempty t1: increment /money:=1 t4: decrement [money=1] /money:=0 t2: increment [money<MAX] /money:=money+1 t3: decrement [money>1] /money:=money-1

SDRL & RTG University of Pennsylvania 5/24/01 25 Test Generation from EFSM Model checking based approach to generate tests –Finding counterexamples during the model checking of EFSMs –State coverage: !EF nonempty –Transition coverage: !EF t4 –All-definition coverage: !EF (t1 & EF (!(t1|t2|t3|t4) U t4)) Explore other approaches empty nonempty t1: increment /money:=1 t4: decrement [money=1] /money:=0 t2: increment [money<MAX] /money:=money+1 t3: decrement [money>1] /money:=money-1

SDRL & RTG University of Pennsylvania 5/24/01 26 Test Generation for CHARON Adapt EFSM-based test generation to hybrid systems Transforming differential equations into EFSMs –Qualitative reasoning and predicate abstraction techniques for CHARON Determining the executability of test cases –Symbolic execution techniques for CHARON

SDRL & RTG University of Pennsylvania 5/24/01 27 Other issues Test generation –How to meet coverage criteria –Test suite optimization Test execution –Integration of testing with MaCS Effectiveness of coverage criteria in practice Apply to real examples

SDRL & RTG University of Pennsylvania 5/24/01 28 Q&AQ&A