Retrospective computation makes past states available inline with current state in a live system What is the language for retrospective computation? What can be expressed? What is the performance of retrospection in a live system? Our approach provides retrospection within split snapshots model Application declares transactionally-consistent snapshots at any time with any frequency All applications and access methods run unmodified on persistent, on-line snapshots Explore this model through representative examples and preliminary results Performs random updates in a set of 1000 objects Throttle updates by pausing between transactions Example trigger detects all change events in a subset of the objects The trigger must keep up with updates We arbitrarily limit trigger to run for 5s or less, restarting it every 5s Measure the average number of objects which can be monitored during each invocation of the trigger The Retrospection Abstraction in Storage Systems (teaching your pet database to retrospect) Ross Shaull Liuba Shrira Laboratory for Experimental Software Systems Brandeis University Motivation: applications need access to past states Example 2: Background triggers (monitoring the recent past) Known technique using background thread to batch event detection Conventionally, this can be achieved by programming custom history tables Instead, we envision using language of retrospection to use any query as an event detector over recent past by periodically running it with ATE Performance evaluation Background: how do split snapshots work? In a standard storage system model (left), a page table relates page names (e.g., “P1”) to physical page addresses using a page table Split snapshots (right) modify this indirection layer to provide snapshot page tables which point either into the current state or into the snapshot store where old versions of pages are located Layers above the cache (e.g., an application) are unaware of this translation, they just request to run on a particular snapshot P1P1 P1P1 P2P2 P1P1 P2P2 P3P3 Snapshot Page Table for S2 Page tables are constructed on-demand from a low-cost log-structured store of page pointers Our previous work has examined how to optimize this search operation without sacrificing high- performance snapshot creation (the read/write optimized Skippy index) Model and Language Paper trail Shaull, R., Shrira, L., and Xu, H. Skippy: a New Snapshot Indexing Method for Time Travel in the Storage Manager. SIGMOD Shrira, L., van Ingen, C., and Shaull, R. Time Travel in the Virtualized Past. SYSTOR Shrira, L. and Xu, H. Thresher: An Efficient Storage Manager for Copy-on-write Snapshots. USENIX Shrira, L. and Xu, H. Snap: Efficient Snapshots for Back-In-Time Execution. ICDE The writer in the current state is minimally impacted by the background trigger For relatively light update workloads, 100% (1000) of objects can be monitored, but if BDB makes updates as quickly as possible, the trigger can only monitor 1% (10) of objects Triggers using retrospection are convenient, and provide powerful semantics beyond change detection Example 1: Forensic retrospection (exploring the distant past) An important ability enabled by retrospection is forensic querying in a live production system Often you do not know what will be of interest in the future, so forensics requires ad-hoc retrospection Leverage BITE and ATE for convenient interface to “total recall” of the past Performance evaluation Simulate forensic query that scans portions of three medium-sized tables (> 100MB) (db_perf [Sullivan. PhD Thesis. 2003]) Perform query over old windows of time to simulate search for anomalies Forensic query does not significantly impact performance of update workload ATE scales with number of snapshots Snapshots are on separate disk, so joint xput is close to sum of each Example BDB code for retrospection with ATE: for(txn->as_of = first_snap; txn->as_of <= last_snap; ++txn->as_of) { db->get(db, txn, &key, &val, 0); if(! strcmp(val.data, V)) { react(); } Integrated retrospective computations run inside the storage system containing current state and snapshots Existing interface (e.g., query language) can be used Simple iteration constructs used to search time windows with back-in-time execution (BITE) and across-time execution (ATE) Current state, recent past, and distant past can be accessed together, allowing associations between objects and their history Snapshots are retained incrementally using copy-on-write (COW), without reorganizing database Achieve good performance in same manner as the database: leverage db recovery to defer snapshot writes Test platform implemented as modification to Berkeley DB (BDB) Vertical axis scaled by number of snapshots analyzed