Parametric Trace Slicing and Monitoring Feng Chen and Grigore Rosu Department of Computer Science University of Illinois at Urbana-Champaign Thank *** for the introduction. Today I am going to discuss some techniques of using trace analysis to build reliable software.

Runtime Verification - Idea Approach to analyze programs by observing and checking their executions at runtime … event1 event2 event3 event4 Execution Trace Program Execute & Observe ? Analyze First, let me give a short introduction to runtime verification. Briefly, runtime verification is an approach to analyze software programs. It is a quite new research domain and getting more and more interest recently; there has been a workshop since 2000 and it will become a conference the next year. And in major conferences, such as ICSE and OOPSLA, we can see many related papers in recent years. There are many different runtime verification techniques but they all share some common process, that is, given a program and we want to analyze its behavior, using RV, we will execution the program and observe its execution, then we extract a runtime model from the observation, which can be analyzed to draw some conclusions about the program. A very common and simple example of runtime models is execution traces, but as I will show later, we can compute other models for different purposes.

Runtime Verification - Problem Given a program execution Exec and a specification Prop, Does Exec satisfy Prop? It scales! Does not prove correctness “per se”, but it can when used in combination with correct recovery Many runtime verification systems JPaX (NASA), MAC (UPenn), MOP (UIUC), Tracematches (Oxford), PQL (Stanford), etc.

Overview Runtime Verifiable Properties Parametric Properties Parametric Traces Parametric Trace Slicing Parametric Trace Monitoring Implementation and Evaluation

Runtime Verification - Properties Typically must be “monitorable” Different specification formalisms; examples: LTL : Authenticate before use Always(use -> Eventually-past authenticate) ERE (Extended regular expression) : Wrong close openGood (! closeGood)* closeBad CFG (Context free grammar): Matched locks S -> ε | S S | begin S end | acq S rel etc.

Parametric Properties Many runtime verification systems today support parametric properties Tracematches – parametric regular patterns PQL – parametric context-free grammars J-LO – parametric linear temporal logic MOP – all the above and more PTQL and Eagle – nested parameters, complex However, current implementations are adhoc, mathematical foundation necessary

Parametric Properties - Examples LTL : Authenticate before use for any resource Always(use<r> -> Eventually-past authenticate<r>) ERE : Wrong close for any file openGood<f> (! closeGood<f>)* closeBad<f> CFG : Matched locks – any lock in any task S -> ε | S S | begin<t> S end<t> | acq<l> S rel<l> etc.

Running Example: Safe Enumeration One should not change a vector while being accessed via one or more enumeration objects … Vector v 1 2 3 4 5 6 Data … Enumeration e 1 2 3 4

Example of Unsafe Enumeration Bug found in jHotDraw The underlying vector should not be changed when one of its enumeration is being used! Main Thread: Vector v = //initialization; … Enumeration e = v.elements(); Object obj = e.nextElement(); Task Thread: … v.remove(0); May cause unexpected behaviors, e.g., a NoSuchElement Exception or some errors later. Now let me use a concrete real-life example to show how RV can help error detection. … This kind of errors is more difficult to detect and debug because it may not cause visible failures of the program. RV provides a feasible solution to this problem. In fact, this bug is a real bug we found in an open source program using our technique. But before we can detect this error, we first need to know why it is an error. Actually, this error is related to an …

Safe Enumeration as Parametric Property Violation pattern (using regular expressions) of three events update<v> : change vector v createEnum<v,e> : create enumeration e from vector v useEnum<e> : use enumeration e update* createEnum useEnum* update+ useEnum Monitor update useEnum update update createEnum useEnum 1 2 3 Violation state

Runtime Verification of Safe Enum … Main Thread: Vector v = //initialization; … Enumeration e = v.elements(); Object obj = e.nextElement(); Task Thread: … v.remove(0); createEnum update useEnum After explaining the example: This example is not complicated but it shows the basics of using runtime verification for error detection. First, we need some formalism to specify properties and algorithms to generate monitors from the specification; like the regular expression and state machine I used in this example. And we may use many other logics and other algorithms, such LTL. We also need some mechanism to observe the execution and building the runtime model. All these together gives us an automated RV technique. In the remaining talk, I will mainly discuss two important challenges we encountered when applying RV in practice applications. update useEnum update update createEnum useEnum 1 2 3

Overview Runtime Verifiable Properties Parametric Properties Parametric Traces Parametric Trace Slicing Parametric Trace Monitoring Implementation and Evaluation

Parametric Traces Like properties, traces also need to be parametric Parametric trace = trace with parametric events

Lack of Parameters Leads to False Alarms … Main Thread: Vector v = //initialization; … Enumeration e = v.elements(); Object obj = e.nextElement(); Task Thread: … v.remove(0); createEnum update v2.remove(0); useEnum The first challenge originally came from object-oriented programs. We have seen that we can detect the error in this execution; but what if the task thread changes a different vector v2? In such case, there is no violation of the property. But we may have the same execution trace and it causes a warning, which is obviously a false alarm. The problem is clear: this trace is too abstract and cannot tell the difference between the two programs. So we need to add more information to the trace; but that brings new challenges about analyzing the trace, as I will discuss in a couple slides. Appear to be a violation but it is not; false alarm!

Adding Parameters to Events … Main Thread: Vector v = //initialization; … Enumeration e = v.elements(); Object obj = e.nextElement(); Task Thread: … v2.remove(0); createEnum(v, e) update(v2) update(v) v.remove(0); useEnum(e) … Start with the first problem: parametric traces. Parametric traces: traces containing events with parameters; Abundant in practice, especially in object-oriented programs

Overview Runtime Verifiable Properties Parametric Properties Parametric Traces Parametric Trace Slicing Parametric Trace Monitoring Implementation and Evaluation

Checking Parametric Traces parametric monitor non-parametric monitor update(v1) update useEnum createEnum(v1,e1) createEnum 1 update(v2) update useEnum(e1) useEnum 2 3 createEnum(v1,e2) We need to close the gap. How? One solution, which is commonly used in other existing approaches, is to extend the monitor with support for parameters, but … update(v1) update useEnum(e1)

Parametric Monitors Other approaches: Monolithic (centralized) monitors Tracematches [Oxford], Program Query Language (PQL) [Stanford], Eagle [NASA], etc. Bound to specific formalisms/checking mechanisms Limited expressiveness, specific to application domains Our solution: decentralized monitors Formalism-independent, work with any formalism More expressive, adaptive to different domains Facilitates optimization (separation of concerns) Evaluation shows better performance

Parametric Trace Slicing v1, e1 v1, e2 v2, e1 v2, e2 update update(v1) update(v1) update createEnum(v1,e1) createEnum(v1,e1) createEnum update(v2) trace slice update useEnum(e1) useEnum(e1) useEnum useEnum createEnum(v1,e2) createEnum update(v1) update(v1) update useEnum(e1) useEnum(e1) useEnum useEnum For given parameters (v, e)

Naïve monitoring of Parametric Traces Every parametric trace contains multiple non-parametric trace slices, each corresponding to a particular parameter binding update useEnum update v1, e1 createEnum update useEnum 1 2 3 update useEnum update v1, e2 update createEnum useEnum 1 2 3

Parametric Trace Slicing - Challenges v1, e1 v1, e2 v2, e1 update(v1) update(v1) update update createEnum(v1,e1) createEnum(v1,e1) How to do it efficiently? createEnum update(v2) update useEnum(e1) useEnum(e1) useEnum useEnum What if the trace is not complete? More concretely, the parameter trace slicing is that, given a set of parameters and a parametric trace, how to build all the parameter bindings that can be seen in the parametric trace, together with corresponding traces. There is a natural solution, if we have the whole trace at hand: first, we go through the trace to construct all the parameter bindings; then we extract events for each binding we found and build the trace slices. But this solution is not efficient enough in many cases: the trace can be very long, for example, in some our experiments, the observed executions contained millions of events. (click) So doing the slicing efficiently is very important. Also, sometimes, we need to check the execution when we observe it, instead of checking it after the execution finishes; (click) in such cases, we don’t have a complete trace to analyze. createEnum(v1,e2) createEnum update(v1) update(v1) update useEnum(e1) useEnum(e1) useEnum useEnum For given parameters (v, e)

Overview Runtime Verifiable Properties Parametric Properties Parametric Traces Parametric Trace Slicing Parametric Trace Monitoring Implementation and Evaluation

Online Parametric Trace Slicing Online: process events as receiving them and do not look back for the previous events Efficient Scan the trace once Events discarded immediately after being processed What information should be kept for the unknown future? We proposed an online parametric trace slicing algorithm. The whole algorithm is quite complicated and I will not go into all the details here because of the time limit. I will try to give a high-level introduction using an running example.

Slicing Example v1 v1, e1 v2 e1 v2, e1 v1, e2 update update(v1) update update Optimization: based on static property analysis, generate specialized slicing code for the given specification createEnum(v1,e1) createEnum update(v2) update update At the end, mention that the two grey ones are useless and how can we improve it? useEnum(e1) useEnum useEnum useEnum createEnum(v1,e2) createEnum For given parameters (v, e)

Overview Runtime Verifiable Properties Parametric Properties Parametric Traces Parametric Trace Slicing Parametric Trace Monitoring Implementation and Evaluation

Implementation and Evaluation Implemented in Monitoring Oriented Programming (MOP) , a runtime verification framework Extensible with specification formalisms, presently support extended regular expressions, past(future)-time linear logics, context-free grammars, … Two instances: JavaMOP and BusMOP http://fsl.cs.uiuc.edu/mop JavaMOP evaluated in more than 100 program-property pairs Dacapo benchmark, Tracematches benchmark, Eclipse, … Monitoring-intensive experiments traces containing millions of events/parameter bindings This parametric trace slicing technique has been implemented in our monitoring oriented programming tool, which is a runtime monitoring framework. Runtime monitoring means that we check the execution of the program when the program is running. We have evaluated our implementation in more than 100 program-property pairs, including many non-trivial, real life examples, such as the dacapo benchmark and the tracematches benchmark. And we also cover some properties that requires very intensive monitoring work.

Numbers are percent overhead with a 3% margin of error JavaMOP Evaluation Overhead less than 8% in most cases; close to hand-optimized monitoring SafeEnum SafeIterator HashMap HasNext LeakingSync ClosedReader antlr 0.0 1.5 1.1 0.4 5.8 chart 3.6 4.8 0.5 eclipse 4.1 2.8 1.4 3.7 3.8 3.0 3.1 2.2 2.4 fop 1.2 0.6 0.8 1.0 hsqldb 3.3 0.9 2.1 jython 0.2 0.3 2.3 luindex 1.6 1.9 1.8 3.2 1.7 lusearch 0.1 pmd 44.8 11.3 25.4 13.7 5.4 8.0 xalan 3.5 4.4 6.7 4.7 6.5 4.5 MOP generated monitors VS. hand-optimized monitors Numbers are percent overhead with a 3% margin of error

JavaMOP Comparison Outperform other techniques, provide better expressiveness SafeIterator HashMap SafeFileInput SafeFileWriter ImprovedLeakingSync MOP TM PQL antlr 2 82 3 6 113 N/E 22 1 chart 11 50 29 37 4 eclipse fop 5 24 58 47 hsqldb 17 78 15 21 280 95 jython 16 12 23 937 0.6 41 luindex 9 181 8 233 33 lusearch 34 132 7 137 49 pmd 178 175 1334 547 658 36 xalan 19 53 90 164 Comparison among MOP, Tracematches, and PQL Numbers are percent overhead with a 3% margin of error; N/E means non-expressible

Conclusion Parametric properties are useful and feasible to runtime verify Parametric trace slicing Generic approach to handle parametric traces Online algorithm for parametric trace slicing Optimized using static property analysis Evaluation Low runtime overhead