1. 4/22/02VU '021 Specification-Based Techniques for Validation at Run-time and Design-time* Insup Lee SDRL (Systems Design Research Lab) RTG (Real-Time Systems Group) Department of Computer and Information Science University of Pennsylvania, Philadelphia, PA * Joint work with H.S. Hong, S. Kannan, M. Kim, O. Sokolsky, M. Viswanathan

2. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 2 Embedded Systems Difficulties –Increasing complexity –Decentralized –Safety critical –Resource constrained Non-functional: power, size, etc. Development of reliable and robust embedded software

3. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 3 Software Development Process Requirements capture and analysis –Informal to formal –Consistency and completeness –Assumptions and interfaces between system components –Application-specific properties Design specifications and analysis –Formal modeling notations –Abstractions –Analysis techniques (simulation, model checking, equivalence checking, testing, etc.) Implementation –Manual/automatic code generation –Validation (testing, model extraction, etc.) –Run-time monitoring and checking Requirements Design specification Implementation

4. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 4 Goals of the HASTEN Project High Assurance Systems Tools and ENvironments (HASTEN) Develop tools for “end-to-end” software engineering –Requirements capture –Specification, analysis, simulation –Implementation testing –Deployed system monitoring and checking Integrated use of tools –Vertical integration –Horizontal integration Case studies –automotive controllers, mobile robots, medical devices, real- time Java, embedded Linux www.cis.upenn.edu/hasten/

5. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 5 Abstraction/ Reengineering Analysis: - model checking - equiv. checking Diagnostic Monitoring Verification Informal Requirements Engineering Formal Requirements Informal Design Diagrams (UML) Implementation Prototype Formal Specification System Artifacts Requirements Artifacts Test Generation Testing Test Suites Test Results Testing Rapid Prototyping/ Simulation Evaluator Evaluation Report Prototyping Instrumentation Event Recognizer Checker Running System/ Filter Abstract Events Checking Output

6. 4/22/02VU '026 Java-MaC: a Run-time Validation Tool for Java Programs M. Kim, S. Kannan, I. Lee, O. Sokolsky University of Pennsylvania M. Viswanathan UIUC

7. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 7 Motivation Weaknesses of formal verification –gap between an abstract model and the implementation –scalability challenge (software size and complexity) Two approaches to implementation validation –Testing widely used limited coverage lack of formal guarantee run-time correctness is not guaranteed –Run-time monitoring and checking w.r.t. formal specification (also known as run-time verification) complementary methodology to formal verification and program testing validate properties on the current execution

8. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 8 Outline Overview of the MaC architecture The MaC languages Java-MaC: the MaC prototype for Java programs Example: stock client Conclusion and future work

9. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 9 Overview of the MaC Architecture Program Static Phase Run-time Phase low-level behavior high-level behavior Program Filter Automatic Instrumentation Human Formal Requirement Spec Low-level Specification High-level Specification Event Recognizer Event Recognizer Automatic Translation Run-time Checker Run-time Checker Automatic Translation Input Informal Requirement Spec

10. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 10 Design of the MaC Languages Must be able to reason about both time instants and information that holds for a duration of time in a program execution. –Events and conditions are a natural division, which is also found in other formalisms such as SCR. –Conditions, which are true or false for a finite duration of time (e.g., is variable x >5?) –Events, which are either present or absent at some instant of time (e.g., is the control right now at the end of method f?). Need temporal operators combining events and conditions in order to reason about traces. start(position==100)end(position==100) 1:00:101:00:301:00:15 raiseGate Time position == 100

11. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 11 Logical Foundation Conditions interpreted over 3 values: true, false and undefined. [.,.) pairs a couple of events to define an interval. start and end define the events corresponding to the instant when conditions change their value.

12. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 12 The MaC Languages Meta Event Definition Language(MEDL) –Describes the safety requirements of the system, in terms of conditions that must always be true, and alarms (events) that must never be raised. –Target program implementation independent. Primitive Event Definition Language (PEDL) –Maps the low-level state information of the system to high- level events. –Provides primitives to refer to values of variables and to certain points in the execution of the program. –PEDL is defined so that events can be recognized in time linear to the size of the PEDL specification –Depends on target program implementation

13. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 13 Meta Event Definition Language (MEDL) ReqSpec /* Import section */ import event ; import condition ; /*Auxiliary variable */ var int ; /*Event and condition */ event =...; condition =...; /*Property and violation */ property =...; alarm =...; /*Auxiliary variable update*/ -> { :=... ; } End Expresses requirements using the events and conditions, sent by event recognizer. Expresses the subset of safety properties. Describes the safety requirements of the system, in terms of conditions that must always be true, and alarms (events) that must never be raised. –property safeRRC = IC -> GD; –alarm violation = start (!safeRRC); Auxilliary variables may be used to store history. –endIC-> { num_train_pass’ = num_train_pass + 1; }

14. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 14 Java-MaC Prototype Requirement Specification Program (Java source code) Program (Java byte code) Monitoring Script (PEDL) Requirements (MEDL) PEDLCompiler MEDLCompiler Instrumented Code Run-time Checker Event Recognizer Filter Generator (JTREK) Instrumentation Information

15. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 15 PEDL for Java The language maps the low-level state information of the system to high-level events and conditions used in describing the requirements. Provides primitives to refer to –primitive variables –beginnings/endings of methods Primitive conditions are constructed from –boolean-valued expressions over the monitored variables Ex: condition IC = (position == 100); Primitive events are constructed from –update(x) –startM(f)/endM(f) Ex: event raiseGate= startM(Gate.gu()); MonScr /* Export section */ export event ; export condition ; /* Overhead reduction */ [timestamp;] [valueabstract;] [deltaabstract;] [multithread;] /* Monitored entities */ monobj ; monmeth ; /* Event and condition*/ event =...; condition =...; End

16. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 16 PEDL for Java (cont.) Events can have two attributes - time and value time(e) gives the time of the last occurrence of event e –used for expressing temporal properties value(e,i) gives the i th value in the tuple of values of e –value of update(var) : a tuple containing the current value of var –value of startM(f) : a tuple containing parameters of the method f –value of endM(f) : a tuple containing parameters and a return value of the method f

17. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 17 Monitoring Objects Java handles an object via a reference pointing to the object Object aliasing: a monitored object can be updated by several references –all references (ex: a.b1.b’, a.b2 ) which possibly point to the monitored object need to be tested whether they are actually pointing to monitored objects at run-time. –this testing may not be feasible due to Java scoping rules Ex: b2 is declared as private in the class A. We cannot test whether a.b1 == a.b2 outside of class A We need a globally visible table containing the addresses of monitored objects

18. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 18 Monitoring Objects (cont.) Address table contains addresses of monitored objects and monitored object names to test references –Ex: Suppose that we monitor a.b2 and the address of a.b2 is 8200. If a.b1.b’ is 8400, we know that a.b1.b’ is not pointing to the monitored variable Java-MaC assumes that the reference of the monitored variable do not change –The update of the address table can cause too much overhead one reference change can cause the large substitution of all descendent nodes already in the address table AddressVar Name 8200a.b2 Address Table

19. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 19 Instrumentation Java-MaC instruments Java executable code Java-MaC instrumentor detects instructions –variable updates putstatic/putfield for global variable updates store and iinc for local variable updates –execution points instruction located at the beginning of method definition return of method definition At the each detected instruction, Java-MaC instrumentor inserts a probe invoking –sendObjMethod(Object parentAddress, value, String varName)

20. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 20 Overview of Java-MaC

21. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 21 Run-time Components of Java-MaC (1) Filter –A filter consists of a communication channel probes inserted into the target system a filter thread which flushes the communication buffers to the event recognizer Event recognizer –evaluates the abstract syntax tree generated from a PEDL specification whenever it receives snapshots from the filter. –If an event or a condition changing its value is detected, the event recognizer sends the event or the condition to the run-time checker.

22. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 22 Run-time Components of Java-MaC (2) Run-time checker –evaluates the abstract syntax tree generated from a MEDL specification whenever it receives events and conditions from the event recognizer. –Detects a violation defined as alarm or property and raises a signal. Connection among run-time components –TCP socket connection –FIFO file connection –User implemented connection using InputStream and OutputStream obtained by Java-MaC API

23. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 23 Specifications for Stock Clients MonScr StockClient export event startPgm, periodStart, conFail, queryResend, oldDataUsed; monmeth void Client.main(String[]); monmeth void Client.run(); monmeth void Client.failConnection(ConnectTry); monmeth Object Client.retryGetData(int); monmeth Object Client.processOldData(); event startPgm = startM(Client.main(String[])); event periodStart = startM(Client.run()); event conFail = startM(Client.failConnection(ConnectTry)); event queryResend = startM(Client.retryGetData(int)); event oldDataUsed = startM(Client.processOldData()); End ReqSpec StockClient import event startPgm, periodStart, conFail, queryResend, oldDataUsed; var long periodTime; var long lastPeriodStart; var int numRetried; var int numConFail; alarm violatedPeriod = end((perioidTime’ >= 900) && (periodTime’ <= 1100)); alarm wrongFT = oldDataUsed when ( (numRetries’ < 4)|| (numConFail’ < 3)); startPgm -> {periodTime’ = 1000; lastPeriodStart’ = time(startPgm) -1000; numRetries’ = 0; numConFail’ = 0;} periodStart ->{ numREtries’ = 0; numConFail’ = 0; periodTime’ =time(periodStart)-lastPeriodStart; lastPeriodStart’ = time(periodStart);}... End

24. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 24 Conclusion and Future Work The MaC architecture provides a lightweight formal methodology for assuring of the correct execution of a target program at run-time –Rigorous analysis, Flexible, Automatic Reduction of monitoring overhead Systematic extension of the MaC architecture to platforms other than Java http://www.cis.upenn.edu/~rtg/mac

25. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 25 The MaCS (MaC with Steering) system

26. 4/22/02VU '0226 A Temporal Logic Based Theory of Test Coverage and Generation Hyoung S. Hong 1,3, Insup Lee 1, Oleg Sokolsky 1, Hasan Ural 2 1 SDRL (Systems Design Research Lab) RTG (Real-Time Systems Group) Department of Computer and Information Science University of Pennsylvania, Philadelphia, PA 2 School of Information Technology and Engineering University of Ottawa, Ottawa, Canada 3 Department of Computer Science KAIST, Korea

27. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 27 Specification-Based Testing Determines whether an implementation conforms to its specification –Hardware and protocol conformance testing –Widely-used specifications Finite state machines and labeled transition systems Two main steps –Test generation from specifications What to test, how to generate test –Test execution of implementations Applies tests to implementations and validates the observed behaviors

28. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 28 Overall Structure Specification Test Output Test Suite Implementation Test Generation Test Execution Test Evaluator input output

29. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 29 Our Approach The problem: automatic test generation from specifications –Specifications: deterministic EFSMs –Coverage criteria Control flow: all-states, all-transitions Data flow: all-defs, all-uses, all-inputs, all-outputs Test generation Specification Input to model checker Coverage criterion A set of formulas Model checker A set of tests

30. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 30 Outline EFSM Testing coverage criteria in WCTL Test generation using model checker Test suite optimization problems Current and future work

31. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 31 Specifications: EFSM –S : a set of states –S 0 : initial state –E : a set of input/output events –V : variables are manipulated by transitions –T : a set of transitions IDLEBUSY t1: insert[m+x<=5] /m:=m+x t4: display/y:=m t5: display/y:=m t3: done t2: coffee[m>1] /m:=m-1

32. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 32 Coverage criteria in WCTL Each coverage criterion is represented by a set of temporal logic formulas –WCTL: a subset of CTL Atomic propositions p 1,…,p n Temporal operators EX, EU, EF Conjunctions have at most one non-atomic conjuncts Negations can be applied only to atomic propositions Unrestricted disjunctions E.g.: EF(p 1 & EFp 2 ) –WCTL formulas have linear witnesses

33. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 33 All-states coverage criterion IDLEBUSY t1: insert[m+x<=5] /m:=m+x t4: display/y:=m t5: display/y:=m t3: done t2: coffee[m>1] /m:=m-1 Requires every state be covered at least once With every state s, associate EF( s & EF exit ) EF(idle & EFexit) EF(busy & EFexit) IDLEBUSY

34. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 34 All-transitions coverage criterion Requires every transition be covered at least once With every transition t, associate EF( t & EF exit ) IDLEBUSY t1: insert[m+x<=5] /m:=m+x t4: display/y:=m t5: display/y:=m t3: done t2: coffee[m>1] /m:=m-1 EF(t1 & EFexit) EF(t2 & EFexit) EF(t3 & EFexit) EF(t4 & EFexit) EF(t5 & EFexit) t1: insert[m+x<=5] /m:=m+x t4: display/y:=m t5: display/y:=m t3: done t2: coffee[m>1] /m:=m-1

35. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 35 Data flow: definitions and uses Central notions in the data-flow analysis Definition: a value is assigned to a variable Use: a value of a variable is used in an expression Variables are defined and used in transitions Definition-use pair: (v,t,t’) –v is defined by t –v is used by t’ –There is a path from t to t’ free from other definitions of v

36. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 36 Covering a definition-use pair With a definition-use pair (v, t, t’), associate –EF( t & EXE[! def ( v ) U ( t’ & EF exit )]) –def ( v ) : disjunction of all transitions that define v IDLEBUSY t1: insert[m+x<=5] /m:=m+x t4: display/y:=m t5: display/y:=m t3: done t2: coffee[m>1] /m:=m-1 t1: insert[m+x<=5] /m:=m+x t2: coffee[m>1] /m:=m-1 EF(t1 & EXE[!(t1 | t2) U (t2 & EFexit)])

37. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 37 Data-flow coverage criteria All-defs coverage criterion –Requires a definition-clear path from every definition to some use be covered at least once All-uses coverage criterion –Requires a definition-clear path from every definition to every use be covered at least once IDLEBUSY t1: insert[m+x<=5] /m:=m+x t4: display/y:=m t5: display/y:=m t3: done t2: coffee[m>1] /m:=m-1 All-uses coverage criterion EF(t1 & EXE[!(t1 | t2) U (t1 & EFexit)]) EF(t1 & EXE[!(t1 | t2) U (t2 & EFexit)]) EF(t1 & EXE[!(t1 | t2) U (t4 & EFexit)]) EF(t1 & EXE[!(t1 | t2) U (t5 & EFexit)]) EF(t2 & EXE[!(t1 | t2) U (t1 & EFexit)]) EF(t2 & EXE[!(t1 | t2) U (t2 & EFexit)]) EF(t2 & EXE[!(t1 | t2) U (t4 & EFexit)]) EF(t2 & EXE[!(t1 | t2) U (t5 & EFexit)])

38. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 38 Test Generation Model checker System model Logic formula True or false Witness or counterexample Coverage criterion Model checker System model A set of logic formulas A set of witnesses

39. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 39 Witness Generation Generating a witness for a formula –Cost: the length of a witness –A minimal-cost witness for a formula Existing model checkers generate a minimal-cost witness by breadth-first search of state space E[ U ]

40. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 40 Test Generation A set of witnesses for a set of formulas –Costs The total length of witnesses or The number of witnesses (reset operation is expensive) –Both optimization problems are NP-hard (Hitting Set Problem) E[ U ]

41. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 41 Test Generation Heuristics –Generate a witness for each formula and remove redundant witnesses Use model checker to construct witnesses –Generate all witnesses for each formula and intersect them Need to extend model checker to construct all witnesses for a given formula –A greedy algorithm described in the paper.

42. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 42 Related Work J. Callahan, F. Schneider, S. Easterbrook, “Specification-based testing using model checking”, 1996. D. Geist, et al., “Coverage-directed test generation using symbolic techniques,” 1996. A. Gargantini and C. Heitmeyer, “Using model checking to generate tests from requirements specifications,” 1999. Many more … (see the paper)

43. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 43 Conclusions Test generation from specifications using model checking –Basic idea Express a test coverage criterion as a set of CTL formulas Generate a set of witnesses for the formula set using a model checker –Advantages Language-independent –Can be applied to other EFSM-based specifications Executable –Only executable tests are generated –Limitations Finite state specifications Deterministic specifications

44. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 44 Current Research Test generation from formal specifications –Reactive systems Extended finite state machines (EFSMs) –FSM + data variables Hierarchical reactive modules –EFSM + hierarchy + concurrency –Hybrid systems CHARON –EFSM + hierarchy + concurrency + differential equations Effectiveness evaluation & tool support Case studies: –electronic throttle controller, infusion pump system

45. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 45 Challenge Problem Understanding test coverage criteria as abstraction Specifications Reachability graphs Fully semantic Flow graphs Fully syntactic Abstraction graphs Abstract interpretation, Predicate abstraction, ……

46. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 46 Abstraction/ Reengineering Analysis: - model checking - equiv. checking Diagnostic Monitoring Verification Informal Requirements Engineering Formal Requirements Informal Design Diagrams (UML) Implementation Prototype Formal Specification System Artifacts Requirements Artifacts Test Generation Testing Test Suites Test Results Testing Rapid Prototyping/ Simulation Evaluator Evaluation Report Prototyping Instrumentation Event Recognizer Checker Running System/ Filter Abstract Events Checking Output

47. 4/22/02VU '0247 Q & A

48. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 48 Vertical integration scenario SCR*Charon MEDL generator MEDL interface code generation Mocha MaCS discrete abstraction diagnostics

49. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 49 Horizontal integration scenario UML-RTParagonCharon scheduling assumptions task model

50. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 50 Data flow chains Affect pair (v, t, v’, t’) –value of v used by t affects the value of v’ defined at t’ (v,t) directly affects (v’,t) Either (v,t) directly affects (v’,t’) or there is a definition- use pair (v’’,t,t’’) such that (v,t) directly affects (v’’,t) and (v’’,t’’) affects (v’,t’) t5: display/y:=m IDLEBUSY t1: insert[m+x<=5] /m:=m+x t4: display/y:=m t3: done t2: coffee[m>1] /m:=m-1 (x, t1) directly affects (m, t1) t1: insert[m+x<=5] /m:=m+x (x, t1) affects (y, t5) t1: insert[m+x<=5] /m:=m+x t5: display/y:=m

51. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 51 Data flow chain coverage Affect pair (v, t, v’, t’) –May consist of an arbitrary number of definition-use pairs –We extend CTL with least fixpoint operators Alternatively, we can use (alternation-free) mu-calculus All-inputs coverage criterion –Requires a path from every input to some output be covered at least once All-outputs coverage criterion –Requires a path from every input to every output be covered at least once

52. SDRL & RTG University of Pennsylvania 4/22/02 VU '02 52 Properties of embedded systems Adherence to safety-critical properties Meeting timing constraints Satisfaction of resource constraints Confinement of resource accesses Supporting fault tolerance Domain specific requirements –Mobility –Software configuration