Streaming Data Warehouses Theodore Johnson

Data Streams in AT&T Packets flowing across a network. –At the core –At a Point Of Presence (POP) –At the network edge. Periodic reports –Network traffic –Router performance –Router configuration –Alerts and alarms –….

Stream Processing Systems Gigascope –Data Stream Management System (DSMS) for monitoring network traffic. –Massive real-time query processing on network traffic. –Reduce the data flow and extract knowledge. DataDepot –Data Warehouse generator with a strong temporal flavor. –Near-real time processing and archival storage of massive data sets. –Store data feeds, transform and correlate them for canned reports and ad-hoc queries.

Gigascope Extremely fast processing of complex query sets on massive data streams. –DSL monitoring : 28 complex queries on Gigeth stream. –10X faster than other systems 1.2M packets/sec (in 2003) vs. 120K packets/sec. for Streambase. Applications throughout AT&T –Broadband (DSL) and IPTV (Lightspeed) –Core router health (OC48 for 2+ years, required for OC-768 deployment). –Other: security, data center monitoring, specialized customer applications.

Gigascope Schema PROTOCOL IP (Layer2) { uint ipversion get_ip_ipversion (snap_len 23); } PROTOCOL IPV4 (IP){ uint hdr_length get_ipv4_hdr_length (snap_len 34); uint service_type get_ipv4_tos (snap_len 34); uint total_length get_ipv4_total_length (snap_len 34); uint id get_ipv4_id (snap_len 34); bool do_not_fragment get_ipv4_do_not_fragment (snap_len 34); bool more_fragments get_ipv4_more_fragments (snap_len 34); uint offset get_ipv4_offset (snap_len 34); uint ttl get_ipv4_ttl (snap_len 34); uint protocol get_ipv4_protocol (snap_len 34); …continues.. GSQL queries receive raw data from low level schemas which are: –Defined on the packet level –Use inheritance from lower network layer Current schemas cover –Layer 2: ETH/HDLC –Layer 3: IP/IPv4 –Layer 4: UDP/TCP/ICMP –Layer 5-7: Packet data New schemas can be defined by: –Specifying the schema in the GSQL configuration file –Providing C parser functions for the fields in the schema RTS and GSQL compiler highly optimize the field extraction

Gigascope Queries (GSQL) Select tb, srcIP, sum(len) From IPv4 Where protocol=6 Group by time/60 as tb, srcIP Having count(*)>5 GSQL is a SQL-like language specialized for data streams. GSQL queries can be defined over: –Low level schemas –The output of other GSQL queries. Tumbling window semantics –Easy to understand, fast implementation, almost always what the user wants. –Support for a form of landmark queries.

Gigascope Architecture Bulk of the processing performed at the RTS. Low-level queries read directly from the packet buffer. –Avoid copying the packet data to multiple queries. Low-level queries are small and light-weight –Selection, projection, partial aggregation. –Ensure timely processing, small cache footprint. NIC q1q2q3 … Q2 Q1 App RTS Circular buffer

Prefilter Optimization Issue: the function call for invoking a query on a packet is expensive (at 500K pkt/sec). Most queries have a collection of inexpensive predicates (e.g. Protocol = TCP). Push all cheap predicates to a prefilter. Combine shared predicates. Evaluate a query on a packet only if its cheap predicates are satisfied. Results in doubled throughput. Packet prefilter Predicate 1 Predicate 2 … Predicate N q1q2q3q4

DataDepot Turn raw data feeds into data warehouse tables. Automate complex processing –Allow the definition of multiple levels of derived tables (materialized views). Stream-like processing –Assume that data tends to arrive in timestamp order. But not always And there might be several timestamps. –Use the timestamp order to localize processing. Applications –Darkstar : repository for network performance data. –DPI : Broadband (DSL, Lightspeed) data.

Defining Databases Two types of tables: –Raw : defined over a collection of data files. Ultimate data source(s). –Derived : Defined by a query over one or more (Raw or Derived) tables. Incremental Maintenance –All tables must have a timestamp attribute. –The partition of a record is a function of its timestamp attribute. –Tables are maintained over a window on the timestamp attribute.

Raw Tables Assume: data is added by dumping new files into a directory. –The directory and file name might contain metadata. –E.g. /wnet/data/foo/yyyy/mm/dd/bar_hhmm.dat.gz Assume : There is a program which converts the file into a collection of records. –E.g. gzcat /wnet/data/foo/2006/06/06/bar_0606.dat.gz Raw table specification: –Fields –How to find data files. And extract the metadata. –How to convert the data files. –The timestamp attribute and its partitioning function –Assorted other things, e.g. window size.

RAW_TABLE COMPASS_PPS_RAW { OPTIONS{Location '/wnet/darkstar/data/compass_pps'; comment 'Foo';} FIELDS { DATE Timestamp_date [Comment 'bar';]; CLOCK Timestamp_time_bin_start; UINT Timestamp_exact_unix_ts; …. } INPUT_SEP '|'; INPUT_PROGRAM ‘~darkstar/compass_pps/scripts/convert_compass $file'; FIND_FILES 'find_compass_files.pl /ipso/compass TTS INOUTPKTS.*\.gz 10'; PARTITION_SPEC{ FILE_PATTERN '/([0-9]+)\/TTS\/([0-9]+)\/INOUTPKTS[0-9]+.([0-9]+).bgt.gz$/'; STRING Poller_year; STRING Poller_date; STRING Poller_time; } VISIBLE_TIME 2.0; TS_ATTRIBUTE { FIELD Part_TS; FUNCTION hour_of(Timestamp_exact_unix_ts); NOW_FCN hour_of(now())+2; } NUM_TS_PARTITIONS 20; }

Utilities for Raw Tables File Finding –Two pre-defined programs for finding files: find_files.pl : search a directory tree. find_dt_files.pl : search a date/time limited portion of a directory tree. –Generated if referenced. Input verification –Input records must precisely conform to the FIELDS specification. –HomeDepot generates an input verification filter. E.g., verify_raw_COMPASS_PPS_RAW.c

Derived Tables A Derived table is computed from one or more source tables. –Egil query –External program –Merge If a source table is updated, automatically update the derived table. Incremental maintenance: only update the affected parts of the Derived table. –Will need a specification of how a partition of a source table affects partitions of the Derived table.

DERIVED_TABLE COMPASS_PPS { OPTIONS{UseCompressedTypes; COMMENT 'blah';} QUERY #{ select Timestamp_date, Timestamp_time_bin_start, Timestamp_exact_unix_ts, Router, Interface, Interface_suffix, from COMPASS_PPS_RAW }# INDICES{ Compass_interface_id; Router; Timestamp_time_bin_start; } LOCATION '/wnet/joel_2/darkstar/compass_pps/hd2/data'; TS_ATTRIBUTE { FIELD Part_TS; FUNCTION hour_of(Timestamp_exact_unix_ts);// 1 hour bins NOW_FCN hour_of(now())+2; LOWER_BOUNDS 24*day_of(Timestamp_date); UPPER_BOUNDS 24*day_of(Timestamp_date)+23; } SOURCE_LOWER_BOUND COMPASS_PPS_RAW: P; SOURCE_UPPER_BOUND COMPASS_PPS_RAW: P; NUM_TS_PARTITIONS 25;//

Semantic Partitioning Tables generated by HD V2.1 or higher will have semantic partitions. –That is, data is partitioned by an attribute’s value. Semantic partitions act like indices –A table can have thousands of partitions. Egil will automatically translate restrictions on the value of a semantic partition’s attribute to restrictions on the partitions to use in the query. –NOT done in DSQL or Cymbal. HomeDepot version is V2.2 … TS field is Part_TS, computed by hour_of__hugeint(Timestamp_exact_unix_ts) more lower bounds on Part_TS are: (24)*(day_of__datet(Timestamp_date)) more upper bounds on Part_TS are: ((24)*(day_of__datet(Timestamp_date)))+(23)

Update Propagation Similar to Makefile processing, but the dependencies are determined on the fly. Invert the upper, lower bounds to determine the partitions of the derived table affected by a partition of the source table SYS_SUMMARYLOG_SUMMARY SYS_LOG_SUMMARY

Research Issues Gigascope –Out-of-order processing (Jin Li) –Parallel and distributed stream processing (Vlad) –Event detection (Ted, Vlad, Lukasz) DataDepot –Scaling for petabyte warehouses (DPI warehouse) –Large scale data quality monitoring (Flip et al.) –Near-real time scheduling (Ted, Vlad, Lukasz) –Data stream optimization techniques in a data warehouse. (TDB)

Event Detection Motivation: detect dropped IPTV packets –Find all video glitches –Just compare current packet sequence number to previous. Related work: Sequence languages –Often phrased as regular expression satisfaction “Find an A event followed by a C event with no intermediate B event”. –Connections: data quality monitoring languages, alerting in darkstar Issue: –Are REs the right language? –Alternative: query over last K tuples. –Very high speed implementation in Gigascope

Select tb, destIP, SSRC, Last(Timestamp,1),Last(Timestamp), Last(Sequence,1), Last(Sequence), IF(GT(Last(Sequence), Last(Sequence,1)+1),'gap','ooop') From Video Group By time/60 as tb, destIP, SSRC Output When Last(Sequence) != Last(Sequence,1) Closing When count(*) = 0 Phrase as an aggregation query automatic partitioning implicit state maintenance Output When : condition under which a tuple is generated. Closing When : delete old state. Last(x,k) : value of x in the k’th previous tuple. Advantage : by playing some tricks, we can support an LFTA/HFTA split. Disadvantage : state management is UDAF writer’s burden.

Near-Real Time Data Warehouse Critical application in DataDepot. –Update RT tables every 5 minutes. –Supported by Multiversion Concurrency Control. Issue: handling overload situations –Can’t drop tasks : must bring the warehouse up-to-date eventually. –But we can let the schedule slip until the overload passes. –We don’t have full control over server load. Issue: hierarchy of derived tables. –Delaying the update of one table delays the update of all dependent tables. Current idea –Allow slippage in update schedule Table T goes from 0-5 minutes from current to 5-10 minutes from current. Damage is slippage * table_priority Schedule so as to minimize cumulative damage.