Jolyon White GEC9, 4 th November 2010 Measurement Flow Architecture in OML

OML = Measurement Flows 2 Rutgers University, New Jersey NICTA, Sydney Deutsche Telekom TU Berlin BOWL Testbed National Broadband Network 100Mbs FTTH VoD Trial IREEL Network Education Teaching Platform Rail Bridge Monitoring Sensors NSW Road Traffic Authority Parking Discovery Rutgers Marco Gruteser

Current OML data pipeline Application or Service Measurement pointsFiltersMeasurement streams OML Server Database (SQL) Database (SQL) Database tables File OML client library 3

Schemas Schemas enable: –Provenance –Processing in the pipeline (data crunching) Measurement Stream schema == Combination of schemas of filter outputs Each MS stored in its own DB table 4 MP (A, B, C) A B C (S, T) (U, V, W) (X, Y) (S, T, U, V, W, X, Y) MS Schema

Schemas Example: app name is “otr2” SQL issued to the database: Schema names + metadata define provenance 5 avg avg : DOUBLE max : DOUBLE min : DOUBLE ts : DOUBLE flow_id : INT32 seq_no : UINT32 pkt_length : UINT32 src_host : STRING src_port : STRING MP udp_in: CREATE TABLE otr2_udp_in ([METADATA COLS], pkt_length_avg REAL, pkt_length_max REAL, pkt_length_min REAL);

Measurement Collection Graph Modularize producers + consumers Measurement Point (MP) – data source Processing Point (PP) – buffer, select, filter, join, forward Termination Point (TP) – persistent storage 6 MP PP TP PP Metadata Store Services API MDA (Measurement Data Archive)

Resource provisioning OML has no concept of resource provisioning Experimenter obtains resources for I&M identically to experimental resources –i.e. no distinction between I&M and experiment resources User has full control over how resources used Useful defaults, but allow more if experimenter wants it Can’t always cleanly separate I&M from experiment –Mobile wireless testbeds where I&M must share compute + network with experiment –E.g. Parknet Almost all wireless traffic was measurement flows 7

Transports OML currently supports two custom procotols –Text version –Binary version Standard transports are good! We like IPFIX, aiming to support it (near future) Why? Several reasons but: –Template support  self-describing measurement streams 8 Metadata headers (schemas) Measurement flow Metadata headers (schemas) Measurement flow

Processing Point Applications 9

Proxy Server Buffer measurements on command –Don’t transmit to remote server Same protocol as server –Transparent to client applications Proxy server OML Server Application CMD_BUFFERCMD_REPLAY 10

Hierarchical Measurement Collection High-resolution measurements lose value over time Local storage may be limited Measuring at different granularities Inspired by existing research in Streaming Databases –Numerous VC-backed startups in financial data feed processing space 11

Context-Driven Experiment Steering Dynamic experiments need measured context feedback E.g. Geographic trip lines, link state feedback 12

Context-Driven Measurement Environment feedback can be used to influence the measurement process itself 13