1. Add Real-time Streaming SQL to Your MySQL Skill Set Julian Hyde - Chief Architect Steve Herskovitz – Director of Professional Services

2. The Data Crunch » Data volumes rising fast » Human-originated data (e.g. e-commerce purchases) rising fast » Machine-generated data (e.g. e-commerce events and network packets) rising even faster » Every business needs answers with lower latency » Every significant problem is distributed: » Geographically distributed organizations » Multiple boxes for scale » Exploit multiple cores

3. Data management is hard » If you make a mistake, the system won’t be fast enough » Can’t afford to lose data » New technologies are very difficult to use » MapReduce » NoSQL » Multi-threaded programming in Java, C++, Erlang, Scala, … » Collaborate, interoperate, evolve

4. Today’s Computing Model 4 Databas e $$$$ Transaction Processing Application Infrastructu re Application Log file Real-Time Application Business Events Business Transactions Batch Load Process Polling

5. Stream Computing Model 5 Databas e $$ Transaction Processing Application Infrastructu re Application Real-Time Application Stream Process or Business Events Business Transactions Real-Time Answers Traditional BI Application

6. Case study: Mozilla

7. Demo: Mozilla downloads

8. SQL – life in the old dinosaur yet » Widely spoken » Rich » Orthogonal » Declarative » Tune your system without changing your logical schema » Apps don’t interfere with each other » Adaptive » Route around failure » Exploit available resources » Make tradeoffs to meet QoS goals

9. Streaming SQL: example #1 Tweets about this conference: » SELECT STREAM ROWTIME, author, text FROM Tweets WHERE text LIKE ‘%#MySQL%'

10. Demo: studio & simple query

11. Streaming SQL basics » Streams: » CREATE STREAM Tweets ( author VARCHAR(20), text VARCHAR(140)); » Relational operators have streaming counterparts: » Project (SELECT) » Filter (WHERE) » Union » Join » Aggregation (GROUP BY) » Windowed aggregation (e.g. SUM(x) OVER window) » Sort (ORDER BY)

12. Streaming SQL: example #2 » Each minute, return the number of clicks on each web page: » SELECT STREAM ROWTIME, uri, COUNT(*) FROM PageRequests GROUP BY FLOOR(ROWTIME TO MINUTE), uri

13. Streaming SQL: Time » ROWTIME pseudo-column » Provided by source application or generated by system » WINDOW » Present in regular SQL (e.g. SQL:2003) but more important in streaming SQL » Defines a ‘working set’ for streaming JOIN, GROUP BY, windowed aggregation » Monotonicity (“sortedness”) » Prerequisite for certain streaming operations

14. Streaming SQL: example #3 Find all orders from New York that shipped within an hour: » CREATE VIEW compliant_orders AS SELECT STREAM * FROM orders OVER sla JOIN shipments ON orders.id = shipments.orderid WHERE city = 'New York' WINDOW sla AS (RANGE INTERVAL '1' HOUR PRECEDING)

15. Streaming SQL: other stuff » Schemas, views, tables » Ability to nest queries » User-defined functions and transforms » Adapters make external systems look like read/write streams

16. Streaming SQL: example #4 Find all stock trades where the average price over the last ten trades is two standard deviations higher than the average over the last hour: » SELECT STREAM * FROM ( SELECT STREAM ticker, price, volume, AVG(price) OVER lastHour AS avgHr, STDDEV(price) OVER lastHour AS stddevHr, AVG(price) OVER lastTenTrades AS avg10 FROM Trades WINDOW lastTenTrades AS (PARTITION BY ticker ROWS 10 PRECEDING)), lastHour AS (PARTITION BY ticker RANGE INTERVAL ’1’ HOUR PRECEDING) WHERE avg10 > avgHr + 2 * stddevHr

17. Streaming SQL for business intelligence Conventional BI: » Star schema: » Fact table » Dimension tables » Aggregate tables » Data warehouse populated using an ETL process » OLAP servers (e.g. Mondrian) provide a top-down view of data Challenge: » Keep all of these systems up to date in real time » Alert when key metrics are outside acceptable range

18. ETL Process for OLAP OLAP Operational database Data warehouse Conventional ETL Aggregate tables populated from DW OLAP cache flushed after load

19. Real-time OLAP: Challenges OLAP imperatives Highly aggregated data – e.g. one number computed from 10M rows Therefore: 1.Use a cache 2.Materialize results as aggregates Real-time imperatives View latest version of the data Maintaining N aggregates requires ~N blocks of I/O per incoming row Therefore: 1.Don’t use a cache 2.Don’t maintain aggregates

20. Real-time OLAP: Solutions 1.Notify cache when underlying data has changed 1.Populate cache from data warehouse 2.Continuous ETL process 2.Build aggregates in memory 1.Flush to disk intermittently 2.OLAP engine looks for aggregates in memory first

21. Continuous ETL for Real-time OLAP OLAP Operational database Data warehouse SQLstream Aggregate tables populated incrementally OLAP cache flushed proactively

22. Demo: » Use mozilla data, show raw parsed input with geoip stuff added, per-second rollups per country. » In MySQL, show an aggregate table growing.

23. Summary 1.Data problems are getting harder 2.People are trying – and failing – to solve these problems with SQL databases 3.Stream computing is a powerful new kind of platform 4.Streaming SQL is pragmatic and powerful

24. Any questions?

25. Thank you for attending! Further reading: » “Data in Flight” by Julian Hyde (Communications of the ACM, Vol. 53 No. 1, Pages 48-52)Data in Flight Blog: http://julianhyde.blogspot.com/http://julianhyde.blogspot.com/ Twitter: @julianhyde