Add Real-time Streaming SQL to Your MySQL Skill Set Julian Hyde - Chief Architect Steve Herskovitz – Director of Professional Services

The Data Crunch » Data volumes rising fast » Human-originated data (e.g. e-commerce purchases) rising fast » Machine-generated data (e.g. e-commerce events and network packets) rising even faster » Every business needs answers with lower latency » Every significant problem is distributed: » Geographically distributed organizations » Multiple boxes for scale » Exploit multiple cores

Data management is hard » If you make a mistake, the system won’t be fast enough » Can’t afford to lose data » New technologies are very difficult to use » MapReduce » NoSQL » Multi-threaded programming in Java, C++, Erlang, Scala, … » Collaborate, interoperate, evolve

Today’s Computing Model 4 Databas e $$$$ Transaction Processing Application Infrastructu re Application Log file Real-Time Application Business Events Business Transactions Batch Load Process Polling

Stream Computing Model 5 Databas e $$ Transaction Processing Application Infrastructu re Application Real-Time Application Stream Process or Business Events Business Transactions Real-Time Answers Traditional BI Application

Case study: Mozilla

Demo: Mozilla downloads

SQL – life in the old dinosaur yet » Widely spoken » Rich » Orthogonal » Declarative » Tune your system without changing your logical schema » Apps don’t interfere with each other » Adaptive » Route around failure » Exploit available resources » Make tradeoffs to meet QoS goals

Streaming SQL: example #1 Tweets about this conference: » SELECT STREAM ROWTIME, author, text FROM Tweets WHERE text LIKE ‘%#MySQL%'

Demo: studio & simple query

Streaming SQL basics » Streams: » CREATE STREAM Tweets ( author VARCHAR(20), text VARCHAR(140)); » Relational operators have streaming counterparts: » Project (SELECT) » Filter (WHERE) » Union » Join » Aggregation (GROUP BY) » Windowed aggregation (e.g. SUM(x) OVER window) » Sort (ORDER BY)

Streaming SQL: example #2 » Each minute, return the number of clicks on each web page: » SELECT STREAM ROWTIME, uri, COUNT(*) FROM PageRequests GROUP BY FLOOR(ROWTIME TO MINUTE), uri

Streaming SQL: Time » ROWTIME pseudo-column » Provided by source application or generated by system » WINDOW » Present in regular SQL (e.g. SQL:2003) but more important in streaming SQL » Defines a ‘working set’ for streaming JOIN, GROUP BY, windowed aggregation » Monotonicity (“sortedness”) » Prerequisite for certain streaming operations

Streaming SQL: example #3 Find all orders from New York that shipped within an hour: » CREATE VIEW compliant_orders AS SELECT STREAM * FROM orders OVER sla JOIN shipments ON orders.id = shipments.orderid WHERE city = 'New York' WINDOW sla AS (RANGE INTERVAL '1' HOUR PRECEDING)

Streaming SQL: other stuff » Schemas, views, tables » Ability to nest queries » User-defined functions and transforms » Adapters make external systems look like read/write streams

Streaming SQL: example #4 Find all stock trades where the average price over the last ten trades is two standard deviations higher than the average over the last hour: » SELECT STREAM * FROM ( SELECT STREAM ticker, price, volume, AVG(price) OVER lastHour AS avgHr, STDDEV(price) OVER lastHour AS stddevHr, AVG(price) OVER lastTenTrades AS avg10 FROM Trades WINDOW lastTenTrades AS (PARTITION BY ticker ROWS 10 PRECEDING)), lastHour AS (PARTITION BY ticker RANGE INTERVAL ’1’ HOUR PRECEDING) WHERE avg10 > avgHr + 2 * stddevHr

Streaming SQL for business intelligence Conventional BI: » Star schema: » Fact table » Dimension tables » Aggregate tables » Data warehouse populated using an ETL process » OLAP servers (e.g. Mondrian) provide a top-down view of data Challenge: » Keep all of these systems up to date in real time » Alert when key metrics are outside acceptable range

ETL Process for OLAP OLAP Operational database Data warehouse Conventional ETL Aggregate tables populated from DW OLAP cache flushed after load

Real-time OLAP: Challenges OLAP imperatives Highly aggregated data – e.g. one number computed from 10M rows Therefore: 1.Use a cache 2.Materialize results as aggregates Real-time imperatives View latest version of the data Maintaining N aggregates requires ~N blocks of I/O per incoming row Therefore: 1.Don’t use a cache 2.Don’t maintain aggregates

Real-time OLAP: Solutions 1.Notify cache when underlying data has changed 1.Populate cache from data warehouse 2.Continuous ETL process 2.Build aggregates in memory 1.Flush to disk intermittently 2.OLAP engine looks for aggregates in memory first

Continuous ETL for Real-time OLAP OLAP Operational database Data warehouse SQLstream Aggregate tables populated incrementally OLAP cache flushed proactively

Demo: » Use mozilla data, show raw parsed input with geoip stuff added, per-second rollups per country. » In MySQL, show an aggregate table growing.

Summary 1.Data problems are getting harder 2.People are trying – and failing – to solve these problems with SQL databases 3.Stream computing is a powerful new kind of platform 4.Streaming SQL is pragmatic and powerful

Any questions?

Thank you for attending! Further reading: » “Data in Flight” by Julian Hyde (Communications of the ACM, Vol. 53 No. 1, Pages 48-52)Data in Flight Blog: