1. Run-time Verification of Software Systems
Oleg Sokolsky
Department of Computer and Information Science
University of Pennsylvania
January 21, 2003

2. Acknowledgements Collaboration: Professors: Students: Support:
Insup Lee
Sampath Kannan
Students:
Mahesh Viswanathan
Moonjoo Kim
Usa Sammapun
Support:
ONR, NSF, ARO
9/18/2018

3. Outline Introduction Why I work in formal methods? Overview
What is run-time verification, and why do we care?
Main body
A framework for run-time verification and its implementation
Conclusions
Beyond run-time verification
9/18/2018

4. What are formal methods?
Behavioral properties, e.g. temporal logic
Models provide a higher-level view of the system
“Check before you build”
“Correct by construction”
Debug and fix errors
9/18/2018

5. Advantages of formal methods
Modeling as means of reuse and lifecycle support
Analyze before you build
Early detection of errors
Correct by construction
Implementation by refinement preserves properties of the model
Analysis provides counterexamples
Debug and fix errors easier
9/18/2018

6. Do formal methods deliver?
Modeling is tedious and hard to get right
State explosion
Seldom goes all the way
Cryptic and large
9/18/2018

7. Formal methods drawbacks
“Garbage in, garbage out”
Large detailed models are almost as hard to craft as code, and have less tool support
Analysis is as good as the model
Only small models can be exhaustively analyzed
Rather than proving correctness, formal methods provide sophisticated debugging support
Only simple models can be refined into code
If implementation is not fully automatic, what have we learned from analyzing the model?
9/18/2018

8. Is it hopeless? Of course not!
What do we need?
Better modeling languages
More domain-specific, more declarative, better tool support
Model-driven implementation techniques
Model-driven code generation rather than refinement
Model-driven test generation
Novel applications of formal methods
Run-time verification
9/18/2018

9. Outline Introduction Why I work in formal methods? Overview
What is run-time verification, and why do we care?
Main body
A framework for run-time verification and its implementation
Conclusions
Beyond run-time verification
9/18/2018

10. Motivation
Problem:
No matter what we do, the system can misbehave at run time
We check models, not implementations
We cannot always check the model exhaustively
Solution:
Validate system executions at run-time
9/18/2018

11. Advantages and disadvantages+
Check implementation, not model
Avoid state explosion -
Only one execution is checked
Assess coverage metrics?
And what if it breaks?
Provide feedback for recovery?
9/18/2018

12. What is run-time verification?
Check the same properties
Simpler checking algorithm
Extract only relevant information
Low-level execution data vs. high-level requirements
9/18/2018

13. Monitoring behavioral properties
Formulas in a temporal logic
Always evaluated over a finite execution trace
Safety properties
“something bad does not happen”
Raise alarm when the bad happens
Liveness properties
Requires non-traditional interpretation
Ultimately, properties determine what observations are relevant
9/18/2018

14. Checking a property of a trace
Satisfaction relation t:
Simple algorithm, linear in the trace length
At each step, trace becomes longer t’:
Furthermore, traces are too big to store
Need a different approach
9/18/2018

15. Incremental checking of a trace
In fact, we do not need to check the whole trace over and over again
Keep a checker state
values of all subformulas
Upon each observation, update checker state
9/18/2018

16. Requirements vs. observations
System requirements are high-level and independent of an implementation
Run-time observations are low-level and implementation-specific
Software: variable assignments, function calls, exceptions, etc.
Network: send, receive, route packets, update routing tables, etc.
Need an abstraction layer to match the two
9/18/2018

17. Trace extraction Too much information is just too much!
Trace is a sequence of observations
A temporal projection of execution
Observation is a projection of system state
Keep only relevant state components
Too little information is a problem, too
Did you miss anything important?
Can you see everything you need?
Software state is observable
Networks/hardware may have unobservable state
Can you see well enough?
9/18/2018

18. Outline Introduction Why I work in formal methods? Overview
What is run-time verification, and why do we care?
Main body
A framework for run-time verification and its implementation
Conclusions
Beyond run-time verification
9/18/2018

19. MaC: Monitoring and Checking
Designed at UPenn
Components:
Architecture for run-time verification
Languages for monitoring properties and trace abstraction
Steering in response to alarms
Prototype implementation for Java programs
Implementation of checking algorithms
Automatic instrumentation
9/18/2018

20. The MaC framework Requirement Program Spec Human Monitoring Script
low level
description
high level
Input
System
Filter
Automatic
Instrumentation
Event
Recognizer
Automatic
Translation
Run-time
Checker
Automatic
Translation
The inputs to the MaC framework are program and informal requirement specification. Based on these two input, human user writes down the monitoring script. The monitoring script consists of both low-level description and high-level description. Based on the low level description and program, the program is instrumented automatically. Instrumented program has a filter as a component. Also, based on the low level description, event recognizer is automatically generated. Similarly, based on the high level description, run-time checker is generated. At run-time, filter extracts low level information like values of variables and sends it to the event recognizer. The event recognizer detects high level events and sends these high level events to the run-time checker. Run-time checker detects violation of properties based on these high level events.
As you see, we separate the low level activity and high level activity both in descriptions and run-time components of the MaC framework. This separation has three benefits. First, whenever the target program is changed, we have to change only the low level description, not the high level description. The high level description of properties in the MaC framework is reusable.
Second, the MaC framework causes less overhead to the target system because most of components are running on different machines.
Third, the run-time components are modularized so that easy to implement the MaC freamework.
I concentrate on left half of the MaC framework. Mahesh Viswanathan concentrates on the right half of the framework and theoretical foundation for run-time formal analysis.
As you see, the monitoring script is the core of the MaC framework. Let us see the MaC language.
Static Phase
low-level
observations
high-level
events
Run-time Phase
steering
9/18/2018

21. MaC languages PEDL Abstract state: Run-time state: SADL MEDL
control locations
object state
local variables
Abstract state:
events
conditions
auxiliary variables
SADL
MEDL
PEDL: Primitive Event Definition Language
abstraction
MEDL: Meta Event Definition Language
abstract transformation
SADL: Steering Action Definition Language
feedback
9/18/2018

22. Properties: events and conditions
Natural distinction for monitoring properties: instantaneous vs. durational
Instantaneity depends on time granularity
Motivation for the distinction:
Cannot monitor every time instance
If you saw something in an observation, is it still there while you are not looking?
Yes – it is a condition
No – it is an event
9/18/2018

23. Example: hundred years’ war
The war is a condition
Battles are events
Battle durations notwithstanding
Events change the state of conditions
end(War)=FallOfBordeaux
FinalDefeat = [FallOfParis,FallOfBordeaux)
9/18/2018

24. Property specification
Program
Requirement
Spec
Human
Monitoring Script
low level
description
high level
Input
System
Filter
Automatic
Instrumentation
Event
Recognizer
Automatic
Translation
Run-time
Checker
Automatic
Translation
The inputs to the MaC framework are program and informal requirement specification. Based on these two input, human user writes down the monitoring script. The monitoring script consists of both low-level description and high-level description. Based on the low level description and program, the program is instrumented automatically. Instrumented program has a filter as a component. Also, based on the low level description, event recognizer is automatically generated. Similarly, based on the high level description, run-time checker is generated. At run-time, filter extracts low level information like values of variables and sends it to the event recognizer. The event recognizer detects high level events and sends these high level events to the run-time checker. Run-time checker detects violation of properties based on these high level events.
As you see, we separate the low level activity and high level activity both in descriptions and run-time components of the MaC framework. This separation has three benefits. First, whenever the target program is changed, we have to change only the low level description, not the high level description. The high level description of properties in the MaC framework is reusable.
Second, the MaC framework causes less overhead to the target system because most of components are running on different machines.
Third, the run-time components are modularized so that easy to implement the MaC freamework.
I concentrate on left half of the MaC framework. Mahesh Viswanathan concentrates on the right half of the framework and theoretical foundation for run-time formal analysis.
As you see, the monitoring script is the core of the MaC framework. Let us see the MaC language.
Static Phase
low-level
observations
high-level
events
Run-time Phase
steering
9/18/2018

25. Meta Event Definition Language
Express requirements using the events and conditions, gathered from an execution
define events and conditions using incoming primitive events and checker state variables
Describe the safety requirements
properties (conditions that must always be true)
alarms (events that must never be raised)
Independent of the monitored system
9/18/2018

26. MEDL: events and conditions
Grammar:
Obvious tautologies:
c = [start(c),end(c))
start([e1,e2))=e1
end([e1,e2))=e2
9/18/2018

27. Semantics of events and conditions
2-sorted past temporal logic:
Interpreted over timed traces
Semantic function defines
The value of a condition at a given time instance
When an event occurs
Two-valued or three-valued interpretation
9/18/2018

28. MEDL: auxiliary variables
Checker state, in addition to values of events and conditions, contains auxiliary variables
Used to define additional events and conditions: event 5th_e = e when (e_count==5)
Updates triggered by events: e -> { e_count += 1 }
Traces are interpreted over both system state variables and checker state variables
9/18/2018

29. Trace extraction and abstraction
Program
Requirement
Spec
Human
Monitoring Script
low level
description
high level
Input
System
Filter
Automatic
Instrumentation
Event
Recognizer
Automatic
Translation
Run-time
Checker
Automatic
Translation
The inputs to the MaC framework are program and informal requirement specification. Based on these two input, human user writes down the monitoring script. The monitoring script consists of both low-level description and high-level description. Based on the low level description and program, the program is instrumented automatically. Instrumented program has a filter as a component. Also, based on the low level description, event recognizer is automatically generated. Similarly, based on the high level description, run-time checker is generated. At run-time, filter extracts low level information like values of variables and sends it to the event recognizer. The event recognizer detects high level events and sends these high level events to the run-time checker. Run-time checker detects violation of properties based on these high level events.
As you see, we separate the low level activity and high level activity both in descriptions and run-time components of the MaC framework. This separation has three benefits. First, whenever the target program is changed, we have to change only the low level description, not the high level description. The high level description of properties in the MaC framework is reusable.
Second, the MaC framework causes less overhead to the target system because most of components are running on different machines.
Third, the run-time components are modularized so that easy to implement the MaC freamework.
I concentrate on left half of the MaC framework. Mahesh Viswanathan concentrates on the right half of the framework and theoretical foundation for run-time formal analysis.
As you see, the monitoring script is the core of the MaC framework. Let us see the MaC language.
Static Phase
low-level
observations
high-level
events
Run-time Phase
steering
9/18/2018

30. Primitive Event Definition Language
Abstracts low-level run-time observations into high-level events
each primitive event or condition used in MEDL must have a PEDL definition
Semantically, a subset of MEDL
Single-state definitions
By necessity, PEDL is system dependent
9/18/2018

31. PEDL for Java Monitored objects Monitored methods
Class variables and local variables of methods
Updates of monitored objects define primitive events
Expressions over monitored objects define primitive conditions
Monitored methods
Calls and returns define primitive events
“inside a call to a monitored method” is a primitive condition
9/18/2018

32. Instrumentation Process
Variable names are available inside classfile
Bytecode instructions to be instrumented:
local variables : <T>store, <T>store_<n> and iinc
global variables : putfield and putstatic
exec points : beginning and return points of the method
Instrumentation algorithm
inserts code for monitoring
updates target addresses of jump instructions
updates the size of operand stack
9/18/2018

33. Filter Probes inserted into the target system
Thread for handling update buffers
Updates reports are synchronized for correct event ordering.
We need a guard against the race condition due to multi threads in order to ensure the correct order of event reports. Without synchronization, the order of reports can be incorrect. We use a lock for synchronization mechanism.
9/18/2018

34. Event recognition Requirement Program Spec Human Monitoring Script
low level
description
high level
Input
System
Filter
Automatic
Instrumentation
Event
Recognizer
Automatic
Translation
Run-time
Checker
Automatic
Translation
The inputs to the MaC framework are program and informal requirement specification. Based on these two input, human user writes down the monitoring script. The monitoring script consists of both low-level description and high-level description. Based on the low level description and program, the program is instrumented automatically. Instrumented program has a filter as a component. Also, based on the low level description, event recognizer is automatically generated. Similarly, based on the high level description, run-time checker is generated. At run-time, filter extracts low level information like values of variables and sends it to the event recognizer. The event recognizer detects high level events and sends these high level events to the run-time checker. Run-time checker detects violation of properties based on these high level events.
As you see, we separate the low level activity and high level activity both in descriptions and run-time components of the MaC framework. This separation has three benefits. First, whenever the target program is changed, we have to change only the low level description, not the high level description. The high level description of properties in the MaC framework is reusable.
Second, the MaC framework causes less overhead to the target system because most of components are running on different machines.
Third, the run-time components are modularized so that easy to implement the MaC freamework.
I concentrate on left half of the MaC framework. Mahesh Viswanathan concentrates on the right half of the framework and theoretical foundation for run-time formal analysis.
As you see, the monitoring script is the core of the MaC framework. Let us see the MaC language.
Static Phase
low-level
observations
high-level
events
Run-time Phase
steering
9/18/2018

35. Event recognition
Event Recognizer evaluates a PEDL script on each message from the filter
store the last seen value of each monitored variable
Specified by PEDL condition foo = x + y ≤ 5
x y foo
undefined
1 undefined
2 true send event to checker
4 true
2 false send event to checker
9/18/2018

36. MEDL evaluation Requirement Program Spec Human Monitoring Script
low level
description
high level
Input
System
Filter
Automatic
Instrumentation
Event
Recognizer
Automatic
Translation
Run-time
Checker
Automatic
Translation
The inputs to the MaC framework are program and informal requirement specification. Based on these two input, human user writes down the monitoring script. The monitoring script consists of both low-level description and high-level description. Based on the low level description and program, the program is instrumented automatically. Instrumented program has a filter as a component. Also, based on the low level description, event recognizer is automatically generated. Similarly, based on the high level description, run-time checker is generated. At run-time, filter extracts low level information like values of variables and sends it to the event recognizer. The event recognizer detects high level events and sends these high level events to the run-time checker. Run-time checker detects violation of properties based on these high level events.
As you see, we separate the low level activity and high level activity both in descriptions and run-time components of the MaC framework. This separation has three benefits. First, whenever the target program is changed, we have to change only the low level description, not the high level description. The high level description of properties in the MaC framework is reusable.
Second, the MaC framework causes less overhead to the target system because most of components are running on different machines.
Third, the run-time components are modularized so that easy to implement the MaC freamework.
I concentrate on left half of the MaC framework. Mahesh Viswanathan concentrates on the right half of the framework and theoretical foundation for run-time formal analysis.
As you see, the monitoring script is the core of the MaC framework. Let us see the MaC language.
Static Phase
low-level
observations
high-level
events
Run-time Phase
steering
9/18/2018

37. Property checking
A MEDL specification can be seen as an automaton with auxiliary store running on a stream of events provided by the event recognizer
aux. variables
9/18/2018

38. Efficient evaluation of MEDL
Checker state:
Value of each condition and auxiliary variable
Timestamp of most recent occurrence of each event
Representation: forest of dependencies between events and conditions
Lazy bottom-up evaluation
Based on statically assigned height
9/18/2018

39. Steering Requirement Program Spec Human Monitoring Script System Event
low level
description
high level
Input
System
Filter
Automatic
Instrumentation
Event
Recognizer
Automatic
Translation
Run-time
Checker
Automatic
Translation
The inputs to the MaC framework are program and informal requirement specification. Based on these two input, human user writes down the monitoring script. The monitoring script consists of both low-level description and high-level description. Based on the low level description and program, the program is instrumented automatically. Instrumented program has a filter as a component. Also, based on the low level description, event recognizer is automatically generated. Similarly, based on the high level description, run-time checker is generated. At run-time, filter extracts low level information like values of variables and sends it to the event recognizer. The event recognizer detects high level events and sends these high level events to the run-time checker. Run-time checker detects violation of properties based on these high level events.
As you see, we separate the low level activity and high level activity both in descriptions and run-time components of the MaC framework. This separation has three benefits. First, whenever the target program is changed, we have to change only the low level description, not the high level description. The high level description of properties in the MaC framework is reusable.
Second, the MaC framework causes less overhead to the target system because most of components are running on different machines.
Third, the run-time components are modularized so that easy to implement the MaC freamework.
I concentrate on left half of the MaC framework. Mahesh Viswanathan concentrates on the right half of the framework and theoretical foundation for run-time formal analysis.
As you see, the monitoring script is the core of the MaC framework. Let us see the MaC language.
Static Phase
low-level
observations
high-level
events
Run-time Phase
steering
9/18/2018

40. Steering
Steering is a way to provide feedback to the system that a violation has occurred
Requires exact knowledge of the system functionality
System should be ready to receive feedback
User specifies when it is safe to steer and what is the appropriate action
Not a recovery mechanism, but a vehicle to invoke recovery mechanism
9/18/2018

41. Steering process system checker action executed detection action
invocation
received
steering
condition
satisfied
violation
action executed
system
event
received
action
invoked
detection
checker
9/18/2018

42. Steering Action Definition Language
Identifies objects used in steering
Defines steering actions
Specifies steering conditions
locations in the code where the actions can be executed
steering action change2SC = { call (IP:dm).setSC() }
before read DecisionModule:volts
Invocation in response to events (in MEDL) SEviolation -> { invoke change2SC }
9/18/2018

43. Implementation of steering
Injector class:
Action bodies as methods
Listener thread
accepts action invocations and sets invocation flags
Calls to action bodies at fixed locations
defined in the steering script, added by instrumentation
Injector class
Checker
execution
invoke
Invocation flags
test
steering conditioni satisfied i n
Action bodies
steering conditioni satisfied i
call n
9/18/2018

44. MaC Prototype with steering
Steering Script
(SADL)
Program
(Java byte code)
Monitoring Script
(PEDL)
Requirements
(MEDL)
SADL
Compiler
PEDL
Compiler
MEDL
Compiler
Instrumentation
Information
Injector class
(Java byte code)
Instrumentation
Information
Filter Generator
(JTREK)
Compiled PEDL
Compiled MEDL
Instrumented
Code
Event Recognizer
Checker
9/18/2018

45. Case studies I Particle formations based on artificial physics
Balance attraction and repulsion
Susceptible to disturbances
Monitoring
Check the progress of pattern formation
Restart pattern formation if disturbance is observed
Suspend and then restore repulsion
Statistically improves quality of formations
9/18/2018

46. Case studies II MaC-based Simplex architecture
Controller for an inverted pendulum with hot-swapping of experimental controllers
Enhanced performance, uncertain stability
Use checker to compute safety envelope
Use steering to switch between controllers
9/18/2018

47. Inverted Pendulum in MaC
Experimental
Controller
Experimental
Controller
Experimental
Controller
Safety
Controller
angle,
track
monitor
Device
Drivers J N I
Decision Module
Switching
logic
volts
steer
9/18/2018

48. Outline Introduction Why I work in formal methods? Overview
What is run-time verification, and why do we care?
Main body
A framework for run-time verification and its implementation
Conclusions
Beyond run-time verification
9/18/2018

49. Summary
Run-time verification is a recent (since 1999) direction in formal methods
Bridge the gap between model and implementation
Avoid state explosion in checking algorithms
Annual workshop on run-time verification
4th workshop is a satellite to TACAS ’04
9/18/2018

50. Summary MaC is a framework for run-time verification Architecture
Monitoring languages
Checking algorithms
Applications for checking and steering
Java programs
Network simulations
QoS requirements (on-going)
9/18/2018

51. Related work Other run-time verification approaches
Java Path Explorer, Time Rover
Incremental checking algorithms, collection of statistics, …
Model-based diagnostics
State estimation
Event definition languages
Monitoring tools
Automatic instrumentation
Post-mortem analysis
9/18/2018

52. Future directions Integration of static and run-time analysis
Suppose we did partial model checking?
Synchronous checking
Faster detection of problems
More efficient extraction of data
Distributed MaC
Collection from distributed platforms
Decentralized checking
Monitoring coverage estimation
How much confidence have we gained?
9/18/2018

53. Other research directions
Modeling approaches
Hierarchical hybrid systems
Uniform resource modeling
Code generation from hybrid models
Feasibility constraints on models
Model-based test generation
Use model checkers to generate the “right” counterexamples
Modeling of genetic pathways by hybrid systems
9/18/2018