1. A new model and architecture for data stream management

2. Why on earth would one need it? Data Stream Management

3. The Problem: Tokyo Traffic Control

4. Stream Processing for Traffic Control  24-hour real-time control  1.000 traffic intersections  15.154 traffic signals  Input  Cameras  Helicopters  Police  Citizen reports  17.000 vehicle detectors  Onboard vehicle sensors  Traffic jams, accidents & closed streets  Output  Central monitors  300 traffic information boards  Digital speed signs  Route signs  Affectors  Adjusted traffic signal lights (7.000)  Communications with officers on site

5. TTC: Center Display Board

6. TTC: Information Board

7. Example Domains  Smart Energy Grid Management  Network Traffic Management  System Monitoring  Road Traffic Monitoring  Military Logistics  Online Auctions  Habitat Monitoring  Immersive Environments

8. Stream Processing Engines  HADP vs DAHP  Events & Triggers  Continuous Queries  Real-time processing  Transient data  Lossy information

9. Overview Aurora

10. The Topic  Aurora  The prototype  DBMS / SPE / DSMS  UI  The query language  The project  The authors

11. The Authors  M.I.T., Department of EECS and Laboratory of Computer Science  Michael Stonebraker  Brandeis University, Department of Computer Science  Daniel J. Abadi  Mitch Cherniack  Brown University, Department of Computer Science  Don Carney  Uğur Çetintemel  Christian Convey  Sangdon Lee  Nesime Tatbul  Stan Zdonik

12. Talk Overview  Stream Processing Engines  SQuAl  Runtime  Related work

13. SQuAl (Stream Query Algebra) Aurora

14. SQuAl Overview  Connection Points  Models  Continuous Query  View  Ad-hoc Query  Operators  Order-agnostic  Order-sensitive

15. SQuAl Operators  Order-agnostic  Filter  Map  Union  Order-sensitive  BSort  Aggregate  Join  Resample  Quirks!

16. Union (Unordered)

17. BSort (Ordered)

18. SQuAl: Example

19. Runtime Aurora

20. Query Optimization  Dynamic Continuous Query Optimization  Inserting projections  Combining boxes  Reordering boxes  Ad-hoc query optimization

21. Real-time Scheduling  Timestamped Tuples  Train scheduling  Interbox nonlinearities  Intrabox nonlinearities  Superboxes  Introspection  Static  Run-time

22. Handling overload  QoS specifications  Response times  Tuple drops  Values produced  Load Shedding  Not Implemented at the time

23. Related work Aurora

24. Related work  STREAM  Stanford University, 2000-2006  Telegraph  UC Berkley, 2000-2007?  SASE  UC Berkley / Mass Amherst, 2006-2008?  Cayuga  Cornell University, 2005-2007?  PIPES  University of Marburg, 2003-2007?  NiagaraCQ  University of Wiscon-Madison, 1999-2002

25. Aurora’s Evolution TimespanProject 2002-2004Aurora (and Aurora*) 2003-2005Medusa 2005-2008Borealis (Medusa + Aurora*) 2003-presentStreamBase (Commercialized)

26. Complex Event Processing Today  Oracle  Oracle CEP  Microsoft  MS SQL Server StreamInsight  Open Source  OpenPDC  Aleri  Coral8  TruViso  StreamBase  Aurora’s Grandchild  IBM  SPADE  Active Middleware Technology

27. Summary  SPEs address different problems  e.g. dynamic realtime monitoring  Data Active, Human Passive  Realtime, transient, even lossy data  Aurora evolved into StreamBase  SQuAl evolved into StreamSQL  Many production-quality alternatives

28. Filter (Unordered)

29. Map (Unordered)

30. Aggregate (Ordered)

31. Join (Ordered)

32. Resample (Ordered)  Based on RRDTool’s philosophy?  Paper:  Simple interpolation  Use The Force, Read The Source:  Average  Count  Sum  Max  Min  LastVal