http://runtimeverification.com RV-Monitor: Efficient Parametric Runtime Verification with Simultaneous Properties Qingzhou Luo, Yi Zhang, Choonghwan Lee, Dongyun Jin, Patrick O’Neil Meredith, Traian Florin Serbanuta and Grigore Rosu University of Illinois at Urbana-Champaign Runtime Verification, Inc.

Motivation RV can significantly increase system reliability However, RV still not very used in production Why? Lack of specifications Not easy to write specifications to check against High monitoring overhead Many RV systems incur large overhead when checking multiple properties simultaneously in real world software

Our Contributions Large collection of formal Java API specifications A comprehensive set (179 in total) of formal specifications covering io, network, lang, util packages Scalable Runtime Verification Monitoring all 179 specifications simultaneously Large scale evaluation - Compare with other state-of-art RV tools

Java API Specifications (Informal) Specifications for Java API Usage If map is modified while an iteration over the set is in progress ... the results of the iteration are undefined. Unintuitive, ambiguous and hard to monitor We manually formalized properties implied in such Java API documentation text covering Four packages (only): io, lang, util and net Tried to mine them, but the didn’t work well

Formal Specifications From Java API Separated specification-implying text - If the map is modified while an iteration over the set is in progress ... the results of the iteration are undefined. is modified: put(...)/clear()/… iteration … in progress: hasNext()/next() undefined: error Wrote specifications using various formalisms Extended regular expressions (most common) FSM, LTL, CFG, SRS (Turing complete) Classified formal specifications as Error, Warning, Suggestion

http://fsl.cs.illinois.edu/annotated-java/index_spec.html

Reducing Monitoring Overhead Most RV Systems are not capable of monitoring multiple properties simultaneously Monitor lookup overhead Memory / Garbage collection overhead Proposing techniques to reduce RV overhead Global Weak Reference Table (GWRT) with Caches Combining Indexing Trees Fine grained locking

Indexing Trees An indexing tree is a multi-level map that, at each level, indexes each concrete object of the parameter binding. One can retrieve the related monitor instance by searching for corresponding concrete object at each level. Map_UnsafeIterator(Map m, Collection c, Iterator i) { creation event getC after(Map m) returning(Collection c) .... event getI after(Collection c) returning(Iterator i) … event modifyM before(Map m) ...   event modifyC before(Collection c) ... event useI before(Iterator i) ... ere : getC (modifyM | modifyC)* getI useI* (modifyM | modifyC)+ use @match { print("Map was modified while being iterated"); } }

Global Weak Reference Table (GWRT) Problem: when constructing indexing tree , multiple weak references for a single object are created. Solution: A GWRT for each parameter type that stores only one weak reference for each distinct object.

Combining Indexing Trees - Problem: Monitoring a real program causes millions of parameter and monitor instances and thus the size of indexing trees becomes quite large Solution: Combine indexing trees originating from the same specification, if they share the same prefix

Combining Indexing Trees Before After

Fine-grained Locking Problem: JavaMOP uses a single global lock throughout all the operations for handling an event. Solution: Remove the global lock, and instead use fine-grained locking given that each of the GWRTS and indexing trees can be processed independently

(Re)Implementation

Evaluation Runtime Overhead Comparison with previous JavaMOP with all 179 specifications Comparison with MOPBox on Collection_UnsafeIterator, the most heavily used spec Running on all DaCapo benchmarks Violations Finding Exceptions thrown / thread unsafe object / ...

Evaluation – Runtime Overhead

Evaluation – Violation Finding

Ineffectual Approaches Combining Indexing Trees across Specifications Wasted space for uncommon parts Enhanced Indexing Cache Hit ratio is already high Indexing Tree Cleaning by GWRT Costs more than cleaning by indexing tree themselves, although GWRT is more effective Statistics-Based Indexing Tree Cleaning Saving by partial cleanup is relatively small

Conclusion Collection of 179 formal specifications covering most commonly used Java packages; assistance tool RV-Monitor: new monitoring system, outperforms old JavaMOP when monitoring multiple properties; new JavaMOP implemented on top Comprehensive evaluation, showing that new optimizations pay off Future? Parallelize: indexing, monitoring Static analysis: do not instrument unnecessarily