Dongyun Jin, Patrick Meredith, Dennis Griffith, Grigore Rosu Garbage Collection for Monitoring Parametric Properties Dongyun Jin, Patrick Meredith, Dennis Griffith, Grigore Rosu This talk is about how to monitor parametric properties “efficiently”. University of Illinois at Urbana-Champaign

Outline Motivation Property Static Analysis Double Lazy Garbage Collection Evaluation Conclusion First, I will give you the motivation

? Monitoring Scalable technique for ensuring software reliability Observe run of a program Analyze execution trace against desired properties React/Report using handlers (if needed) … Program Event 1 Execute & Observe Analyze Event 2 ? Event 3 Event 4 Monitoring is a scalable technique for ensuring software reliability. It observes run of a program, then generates an execution trace consisting of events. We analyze execution trace against desired properties and react or report according to the analysis result, by using handlers. Execution Trace …

Applications of Monitoring Development Debugging Testing Deployment Security Reliability Runtime Verification Monitoring can be used in many ways. In a system development, it can help in debugging or testing. Also, monitoring can be a part of a deployed system for security, enhanced reliability, or runtime verification.

Parametric Properties Properties referring to object instances The following property describes a bad behavior between Each Collection c and Iterator i : Generalize typestates Typestates are parametric properties with one parameter update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) Parametric properties are properties referring to object instances. For example, this property describes an error pattern between each collection and iterator. It says you should not update the collection while you are using its iterator. It’s a generalization of typestates. Then, a typestate is a parametric property with only one parameter. Note that parametric property can have multiple parameters.

Parametric Property Monitoring Systems Tracematches [de Moor et al. 05] Regular Patterns PQL [Lam et al. 05] Context-Free Patterns PTQL [Aiken et al. 05] SQL-like language Eagle then RuleR [Barringer et al. 04] Rule-Based Rewrite Properties (e.g. Context-Free Patterns, ..) MOP [Rosu et al. 03] Formalism Independent Approach many others There are many parametric property monitoring systems. Tracematches monitors regular patterns. PQL and RuleR monitor Context-Free Patterns PTQL uses a SQL-like language. While many systems fix their formalism, MOP is formalism independent. It supports multiple formalisms that a user can choose.

Parametric Monitoring in MOP Keep one monitor for each parameter instance A parameter instance binds parameters to objects E.g., (𝑐 ⟼ 𝑐 2 , 𝑖⟼ 𝑖 3 ) Each monitor knows nothing of parameters; operates exclusively on only one trace slice Here’s how we monitor parametric properties in MOP. We keep one monitor for each parameter instance. A parameter instance is the bindings of objects to a parameter. For examples, this parameter instance binds c2 to c and i3 to i. Then, each monitor does not need to know anything about parameters. It can focus on one trace slice.

Parametric Monitoring in MOP Keep one monitor for each parameter instance A parameter instance binds parameters to objects E.g., (𝑐 ⟼ 𝑐 2 , 𝑖⟼ 𝑖 3 ) Each monitor knows nothing of parameters; operates exclusively on only one trace slice Trace: create(c1, i1) create(c1, i2) next(i1) update(c1) next(i2) Here’s a possible trace with parameters. We have two creation events, next, update, and next.

Parametric Monitoring in MOP Keep one monitor for each parameter instance A parameter instance binds parameters to objects E.g., (𝑐 ⟼ 𝑐 2 , 𝑖⟼ 𝑖 3 ) Each monitor knows nothing of parameters; operates exclusively on only one trace slice Trace: create(c1, i1) create(c1, i2) next(i1) update(c1) next(i2) (c1, i1) Trace Slice: create next update Then, the trace slice for <c1, i1> is create next update. and the trace slice for <c1, i2> is create update next. (c1, i2) Trace Slice: create update next

Parametric Monitoring in MOP Keep one monitor for each parameter instance A parameter instance binds parameters to objects E.g., (𝑐 ⟼ 𝑐 2 , 𝑖⟼ 𝑖 3 ) Each monitor knows nothing of parameters; operates exclusively on only one trace slice Trace: create(c1, i1) create(c1, i2) next(i1) update(c1) next(i2) update(c ) next(i ) create(c , i ) (c1, i1) Trace Slice: create next update We keep a monitor for each trace and update them with the corresponding trace slice. In this example, the monitor for the second slice reached the final state. It means it has an error. (c1, i2) Trace Slice: create update next update(c ) next(i ) create(c , i )

Parametric Monitoring in MOP Keep one monitor for each parameter instance A parameter instance binds parameters to objects E.g., (𝑐 ⟼ 𝑐 2 , 𝑖⟼ 𝑖 3 ) Each monitor knows nothing of parameters; operates exclusively on only one trace slice Trace: create(c1, i1) create(c1, i2) next(i1) update(c1) next(i2) update(c ) next(i ) create(c , i ) (c1, i1) Trace Slice: create next update Our challenge here is that we have a large number of monitors. Because, theoretically, there can be unbounded number of parameter instances, there can be unbounded number of monitors. (c1, i2) Trace Slice: create update next update(c ) next(i ) create(c , i ) Challenge: Large number of monitors

Previously… Benchmark based on DaCapo 9.12 [Blackburn et al. 06] On average for 65 cases (13 examples × 5 properties) Tracematches [de Moor et al. 05] One of the most memory efficient monitoring systems 142% Runtime overhead 1 12% Memory overhead 1 MOP [Rosu et al. 03] 33% Runtime overhead 140% Memory overhead Let’s see how bad it was previously. We did benchmark two systems based on DaCapo 9.12 It has 13 examples and we used 5 properties. To sum, we have 65 combinations. We got average overheads of 65 cases. Tracematches is one of the most memory efficient monitoring systems. It shows 142% runtime overhead and only 12% memory overhead. On the contrary, MOP, which has the lowest runtime overhead of any runtime monitoring system that we know. It shows only 33% runtime overhead and 140% memory overhead. Tracematches manages memory quite well but its memory management is formalism dependent and it spends too much time for that. MOP is fast but it has a memory issue. 1: Except 6(of 65) crashes

Worst Cases – Runtime Overhead From DaCapo 9.12 and DaCapo 2006-10 MR2 2119% 448% 19194% 569% 637% 311% OOM 1203% OOM That was average performances based on new DaCapo. Let’s see 7 worst cases including examples from the old version of DaCapo. This graph shows runtime overheads. As you can see, Tracematches shows much higher runtime overhead. Although JavaMOP shows better runtime performance, but those numbers are still high in our opinion. 571% 1359% ~ 4M Monitors 746% 1942% 716%

Worst Cases – Memory Overhead From DaCapo 9.12 and DaCapo 2006-10 MR2 1059% 294% 57% 2896% 5% 2799% OOM 3439% This graph shows memory overhead in logarithmic scale. Overall, Tracematches shows better memory management except three cases including two crashes. We want to improve memory and runtime performances in MOP. 1082% OOM 39% 2510% 41% 1030%

Goal: Reduce the Runtime/Memory Overhead Static Program Analysis [Bodden et al. 09, 10], [Dwyer et al. 10], … Property Analysis Dynamic By using information from property analysis, garbage collect monitors when they become useless during monitoring There are many approaches both statically and dynamically. In static optimization, we can either analyze program or property. There are many research going on program analysis. It statically analyze the target program and apply monitoring only where needed. Also, we can analyze properties statically and use the result to optimize the system. Our approach consists of two parts. First, we statically analyze the property. Second, we use that information to garbage collect monitors when they become useless.

Outline Motivation Property Static Analysis Double Lazy Garbage Collection Evaluation Conclusion Let’s see Property Static Analysis, first.

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) Let’s consider the parametric property we saw in the beginning. Remind that MOP is formalism independent. Some formalisms cannot be analyzed based on state, like Context-Free Grammar We analyze properties based on the last event so that any formalism can be analyzed in the same way. In this analysis, we’d like to know what parameters are essential for a monitor to trigger. If a monitor has no way to trigger, we can collect it. The Last Event Parameter instances that need to be alive

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) If the last event is create, The Last Event Parameter instances that need to be alive create

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) It must be in this state The Last Event Parameter instances that need to be alive create

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) Needed Events Then, we need these two events for this monitor to trigger. The Last Event Parameter instances that need to be alive create

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) Needed Events Therefore, we need both parameters. The Last Event Parameter instances that need to be alive create {c , i }

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) Needed Events If any of parameters are garbage collected, this monitor has no future and can be collected. The Last Event Parameter instances that need to be alive create {c , i } If any of these parameters are garbage collected, this monitor cannot trigger; can be collected

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) When the last event is update, The Last Event Parameter instances that need to be alive create {c , i } update

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) There are two cases: it can be in this state, or that state. The Last Event Parameter instances that need to be alive create {c , i } update

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) Needed Events If it is in this state, we need next event. The Last Event Parameter instances that need to be alive create {c , i } update

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) Needed Events In this case, we need the parameter i The Last Event Parameter instances that need to be alive create {c , i } update {i }

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) Needed Events In this state, we need three events: create, update and next. The Last Event Parameter instances that need to be alive create {c , i } update {i }

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) Needed Events Then, we need both parameters. Since we don’t know which state the monitor would be in, we should consider both cases. When c is garbage collected, the first case still has a hope, so we don’t collect the monitor. When i is garbage collected, the both cases have no hope, so we collect the monitor. The Last Event Parameter instances that need to be alive create {c , i } update {i }

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) In a similar way, for next event, we need both parameters. The Last Event Parameter instances that need to be alive create {c , i } update {i } next

Property Static Analysis Property analysis tells us when monitors are unnecessary Our analysis is formalism independent; we only show it for FSM monitors (see paper for CFG monitors) update(c ) next(i ) update(c ) create(c , i ) update(c ) next(i ) In a similar way, for next event, we need both parameters. The Last Event Parameter instances that need to be alive create {c , i } update {i } next If any of these are dead, then the current monitor can be collected

Outline Motivation Property Static Analysis Double Lazy Garbage Collection Evaluation Conclusion Now, we know how to decide whether to collect a monitor. Let’s see how we actually collect monitors efficiently.

Double Lazy Garbage Collection Indexing Trees map parameter instances to monitors Lazy propagation of info about dead parameter instances Eager propagation yields high runtime overhead (like TM) Idea: propagate when monitors are updated for other reasons Lazy removal of useless monitors Removal is expensive (monitor can belong to many trees) Idea: just mark unnecessary monitors, then clean up later (technical; see paper for details and proof of correctness) We use double lazy garbage collection. We propagate parameter collections lazily and we remove monitors lazily. To understand this, we should know that we use multi-level indexing trees to map parameter instances to corresponding monitors. Propagating parameter collection information through the indexing tree in real results in too much overhead. Therefore, we propagate when monitors are updated for other reasons In this way, we don’t need to spend extra time to access monitors. We use lazy removal. Removing a monitor is expensive since it can belong to many indexing trees. So, we just mark unnecessary monitors, then remove them whenever we see them later for other reasons. If we are lucky, indexing trees can be garbage collected, then there is no need to remove monitors from them.

Outline Motivation Property Static Analysis Double Lazy Garbage Collection Evaluation Conclusion Let’s see how we improve the performance.

RV (MOP + Garbage Collection) Evaluation – Overall On average, based on the DaCapo 9.12 Benchmark 13 Programs and 5 Properties Tracematches MOP RV (MOP + Garbage Collection) Runtime Overhead 142%1 33% 15% Memory Overhead 12%1 140% 39% We named our new system, RV. On the same setting, we improve the runtime overhead from 33% to 15% and memory overhead from 140% to 39%. We could improve the runtime performance because we do not have to update useless monitors and collecting monitors will result faster garbage collection. 1: Except 6(of 65) crashes

Evaluation – Worst Cases Runtime Overhead From DaCapo 9.12 and DaCapo 2006-10 MR2 2119% 448% 116% 19194% 569% 251% 637% 311% 118% OOM 1203% 178% OOM Let’s see how we improve on the worst cases. As we can see our new system RV shows much less runtime overhead than MOP and Tracematches. 571% 188% 1359% ~ 4M Monitors 746% 212% 1942% 716% 130%

Evaluation – Worst Cases Memory Overhead From DaCapo 9.12 and DaCapo 2006-10 MR2 1059% 294% 169% 57% 2896% 1512% 5% 2799% 236% OOM 3439% 1045% OOM 1082% This graph is about memory overhead, again in logarithmic scale. We can see that RV shows always less memory overhead than MOP, but it shows higher memory overhead than Tracematches in many cases. 420% 39% 2510% 886% 41% 1030% 159%

Conclusions Garbage collection for parametric monitoring Formalism-independent Directed by property static analysis Double Lazy Garbage Collection Lazy propagation of info Lazy removal of monitors Evaluation Lowest runtime overhead parametric monitoring system Reasonable memory overhead (When I have enough Time) Our garbage collection for parametric monitoring is formalism-independent, directed by property static analysis and it’s efficient. It uses double lazy garbage collection to collect monitors efficiently. We reduce overhead of alerting monitors of garbage collected parameters We also reduce overhead of removing monitors. Our evaluation shows that it has the lowest runtime overhead of any parametric monitoring system And it shows reasonable memory overhead. (2) (In a rush) Our garbage collection for parametric monitoring is formalism-independent, directed by property static analysis and efficient.