1. Parametric Trace Slicing and Monitoring
Feng Chen and Grigore Rosu
Department of Computer Science
University of Illinois at Urbana-Champaign
Thank *** for the introduction. Today I am going to discuss some techniques of using trace analysis to build reliable software.

2. Runtime Verification - Idea
Approach to analyze programs by observing and checking their executions at runtime …
event1
event2
event3
event4
Execution Trace
Program
Execute & Observe ?
Analyze
First, let me give a short introduction to runtime verification. Briefly, runtime verification is an approach to analyze software programs. It is a quite new research domain and getting more and more interest recently; there has been a workshop since 2000 and it will become a conference the next year. And in major conferences, such as ICSE and OOPSLA, we can see many related papers in recent years. There are many different runtime verification techniques but they all share some common process, that is, given a program and we want to analyze its behavior, using RV, we will execution the program and observe its execution, then we extract a runtime model from the observation, which can be analyzed to draw some conclusions about the program. A very common and simple example of runtime models is execution traces, but as I will show later, we can compute other models for different purposes.

3. Runtime Verification - Problem
Given a program execution Exec and a specification Prop,
Does Exec satisfy Prop?
It scales!
Does not prove correctness “per se”, but it can when used in combination with correct recovery
Many runtime verification systems
JPaX (NASA), MAC (UPenn), MOP (UIUC), Tracematches (Oxford), PQL (Stanford), etc.

4. Overview Runtime Verifiable Properties Parametric Properties
Parametric Traces
Parametric Trace Slicing
Parametric Trace Monitoring
Implementation and Evaluation

5. Runtime Verification - Properties
Typically must be “monitorable”
Different specification formalisms; examples:
LTL : Authenticate before use
Always(use -> Eventually-past authenticate)
ERE (Extended regular expression) : Wrong close
openGood (! closeGood)* closeBad
CFG (Context free grammar): Matched locks
S -> ε | S S | begin S end | acq S rel
etc.

6. Parametric Properties
Many runtime verification systems today support parametric properties
Tracematches – parametric regular patterns
PQL – parametric context-free grammars
J-LO – parametric linear temporal logic
MOP – all the above and more
PTQL and Eagle – nested parameters, complex
However, current implementations are adhoc, mathematical foundation necessary

7. Parametric Properties - Examples
LTL : Authenticate before use for any resource
Always(use<r> ->
Eventually-past authenticate<r>)
ERE : Wrong close for any file
openGood<f> (! closeGood<f>)* closeBad<f>
CFG : Matched locks – any lock in any task
S -> ε | S S
| begin<t> S end<t> | acq<l> S rel<l>
etc.

8. Running Example: Safe Enumeration
One should not change a vector while being accessed via one or more enumeration objects …
Vector v 1 2 3 4 5 6
Data …
Enumeration e 1 2 3 4

9. Example of Unsafe Enumeration
Bug found in jHotDraw
The underlying vector should not be changed when one of its enumeration is being used!
Main Thread:
Vector v = //initialization; …
Enumeration e = v.elements();
Object obj = e.nextElement();
Task Thread: …
v.remove(0);
May cause unexpected behaviors, e.g., a NoSuchElement Exception or some errors later.
Now let me use a concrete real-life example to show how RV can help error detection. …
This kind of errors is more difficult to detect and debug because it may not cause visible failures of the program. RV provides a feasible solution to this problem. In fact, this bug is a real bug we found in an open source program using our technique.
But before we can detect this error, we first need to know why it is an error. Actually, this error is related to an …

10. Safe Enumeration as Parametric Property
Violation pattern (using regular expressions) of three events
update<v> : change vector v
createEnum<v,e> : create enumeration e from vector v
useEnum<e> : use enumeration e
update* createEnum useEnum* update+ useEnum
Monitor
update
useEnum
update
update
createEnum
useEnum 1 2 3
Violation state

11. Runtime Verification of Safe Enum…
Main Thread:
Vector v = //initialization; …
Enumeration e = v.elements();
Object obj = e.nextElement();
Task Thread: …
v.remove(0);
createEnum
update
useEnum
After explaining the example: This example is not complicated but it shows the basics of using runtime verification for error detection. First, we need some formalism to specify properties and algorithms to generate monitors from the specification; like the regular expression and state machine I used in this example. And we may use many other logics and other algorithms, such LTL. We also need some mechanism to observe the execution and building the runtime model. All these together gives us an automated RV technique. In the remaining talk, I will mainly discuss two important challenges we encountered when applying RV in practice applications.
update
useEnum
update
update
createEnum
useEnum 1 2 3

12. Overview Runtime Verifiable Properties Parametric Properties
Parametric Traces
Parametric Trace Slicing
Parametric Trace Monitoring
Implementation and Evaluation

13. Parametric Traces Like properties, traces also need to be parametric
Parametric trace = trace with parametric events

14. Lack of Parameters Leads to False Alarms…
Main Thread:
Vector v = //initialization; …
Enumeration e = v.elements();
Object obj = e.nextElement();
Task Thread: …
v.remove(0);
createEnum
update
v2.remove(0);
useEnum
The first challenge originally came from object-oriented programs. We have seen that we can detect the error in this execution; but what if the task thread changes a different vector v2? In such case, there is no violation of the property. But we may have the same execution trace and it causes a warning, which is obviously a false alarm. The problem is clear: this trace is too abstract and cannot tell the difference between the two programs. So we need to add more information to the trace; but that brings new challenges about analyzing the trace, as I will discuss in a couple slides.
Appear to be a violation but it is not; false alarm!

15. Adding Parameters to Events…
Main Thread:
Vector v = //initialization; …
Enumeration e = v.elements();
Object obj = e.nextElement();
Task Thread: …
v2.remove(0);
createEnum(v, e)
update(v2)
update(v)
v.remove(0);
useEnum(e) …
Start with the first problem: parametric traces.
Parametric traces: traces containing events with parameters;
Abundant in practice, especially in object-oriented programs

16. Overview Runtime Verifiable Properties Parametric Properties
Parametric Traces
Parametric Trace Slicing
Parametric Trace Monitoring
Implementation and Evaluation

17. Checking Parametric Traces
parametric monitor
non-parametric monitor
update(v1)
update
useEnum
createEnum(v1,e1)
createEnum 1
update(v2)
update
useEnum(e1)
useEnum 2 3
createEnum(v1,e2)
We need to close the gap. How? One solution, which is commonly used in other existing approaches, is to extend the monitor with support for parameters, but …
update(v1)
update
useEnum(e1)

18. Parametric Monitors
Other approaches: Monolithic (centralized) monitors
Tracematches [Oxford], Program Query Language (PQL) [Stanford], Eagle [NASA], etc.
Bound to specific formalisms/checking mechanisms
Limited expressiveness, specific to application domains
Our solution: decentralized monitors
Formalism-independent, work with any formalism
More expressive, adaptive to different domains
Facilitates optimization (separation of concerns)
Evaluation shows better performance

19. Parametric Trace Slicing
v1, e1
v1, e2
v2, e1
v2, e2
update
update(v1)
update(v1)
update
createEnum(v1,e1)
createEnum(v1,e1)
createEnum
update(v2)
trace slice
update
useEnum(e1)
useEnum(e1)
useEnum
useEnum
createEnum(v1,e2)
createEnum
update(v1)
update(v1)
update
useEnum(e1)
useEnum(e1)
useEnum
useEnum
For given parameters (v, e)

20. Naïve monitoring of Parametric Traces
Every parametric trace contains multiple non-parametric trace slices, each corresponding to a particular parameter binding
update
useEnum
update
v1, e1
createEnum
update
useEnum 1 2 3
update
useEnum
update
v1, e2
update
createEnum
useEnum 1 2 3

21. Parametric Trace Slicing - Challenges
v1, e1
v1, e2
v2, e1
update(v1)
update(v1)
update
update
createEnum(v1,e1)
createEnum(v1,e1)
How to do it efficiently?
createEnum
update(v2)
update
useEnum(e1)
useEnum(e1)
useEnum
useEnum
What if the trace is not complete?
More concretely, the parameter trace slicing is that, given a set of parameters and a parametric trace, how to build all the parameter bindings that can be seen in the parametric trace, together with corresponding traces.
There is a natural solution, if we have the whole trace at hand: first, we go through the trace to construct all the parameter bindings; then we extract events for each binding we found and build the trace slices.
But this solution is not efficient enough in many cases: the trace can be very long, for example, in some our experiments, the observed executions contained millions of events.
(click) So doing the slicing efficiently is very important.
Also, sometimes, we need to check the execution when we observe it, instead of checking it after the execution finishes; (click) in such cases, we don’t have a complete trace to analyze.
createEnum(v1,e2)
createEnum
update(v1)
update(v1)
update
useEnum(e1)
useEnum(e1)
useEnum
useEnum
For given parameters (v, e)

22. Overview Runtime Verifiable Properties Parametric Properties
Parametric Traces
Parametric Trace Slicing
Parametric Trace Monitoring
Implementation and Evaluation

23. Online Parametric Trace Slicing
Online: process events as receiving them and do not look back for the previous events
Efficient
Scan the trace once
Events discarded immediately after being processed
What information should be kept for the unknown future?
We proposed an online parametric trace slicing algorithm.
The whole algorithm is quite complicated and I will not go into all the details here because of the time limit. I will try to give a high-level introduction using an running example.

24. Slicing Example v1
v1, e1 v2 e1
v2, e1
v1, e2
update
update(v1)
update
update
Optimization: based on static property analysis, generate
specialized slicing code for the given specification
createEnum(v1,e1)
createEnum
update(v2)
update
update
At the end, mention that the two grey ones are useless and how can we improve it?
useEnum(e1)
useEnum
useEnum
useEnum
createEnum(v1,e2)
createEnum
For given parameters (v, e)

25. Overview Runtime Verifiable Properties Parametric Properties
Parametric Traces
Parametric Trace Slicing
Parametric Trace Monitoring
Implementation and Evaluation

26. Implementation and Evaluation
Implemented in Monitoring Oriented Programming (MOP) , a runtime verification framework
Extensible with specification formalisms, presently support extended regular expressions, past(future)-time linear logics, context-free grammars, …
Two instances: JavaMOP and BusMOP
JavaMOP evaluated in more than 100 program-property pairs
Dacapo benchmark, Tracematches benchmark, Eclipse, …
Monitoring-intensive experiments
traces containing millions of events/parameter bindings
This parametric trace slicing technique has been implemented in our monitoring oriented programming tool, which is a runtime monitoring framework.
Runtime monitoring means that we check the execution of the program when the program is running.
We have evaluated our implementation in more than 100 program-property pairs, including many non-trivial, real life examples, such as the dacapo benchmark and the tracematches benchmark.
And we also cover some properties that requires very intensive monitoring work.

27. Numbers are percent overhead with a 3% margin of error
JavaMOP Evaluation
Overhead less than 8% in most cases; close to hand-optimized monitoring
SafeEnum
SafeIterator
HashMap
HasNext
LeakingSync
ClosedReader
antlr
0.0
1.5
1.1
0.4
5.8
chart
3.6
4.8
0.5
eclipse
4.1
2.8
1.4
3.7
3.8
3.0
3.1
2.2
2.4
fop
1.2
0.6
0.8
1.0
hsqldb
3.3
0.9
2.1
jython
0.2
0.3
2.3
luindex
1.6
1.9
1.8
3.2
1.7
lusearch
0.1
pmd
44.8
11.3
25.4
13.7
5.4
8.0
xalan
3.5
4.4
6.7
4.7
6.5
4.5
MOP generated monitors VS. hand-optimized monitors
Numbers are percent overhead with a 3% margin of error

28. JavaMOP Comparison
Outperform other techniques, provide better expressiveness
SafeIterator
HashMap
SafeFileInput
SafeFileWriter
ImprovedLeakingSync
MOP TM
PQL
antlr 2 82 3 6
113
N/E 22 1
chart 11 50 29 37 4
eclipse
fop 5 24 58 47
hsqldb 17 78 15 21
280 95
jython 16 12 23
937
0.6 41
luindex 9
181 8
233 33
lusearch 34
132 7
137 49
pmd
178
175
1334
547
658 36
xalan 19 53 90
164
Comparison among MOP, Tracematches, and PQL
Numbers are percent overhead with a 3% margin of error; N/E means non-expressible

29. Conclusion
Parametric properties are useful and feasible to runtime verify
Parametric trace slicing
Generic approach to handle parametric traces
Online algorithm for parametric trace slicing
Optimized using static property analysis
Evaluation
Low runtime overhead