1. Dieter Gawlick, Oracle 03-06 October, 2005 (GGF15 in Boston)
Notes About Streams
Dieter Gawlick, Oracle
03-06 October, 2005 (GGF15 in Boston)

2. Examples of Streams Network monitoring and traffic engineering
Sensor networks, RFID tags
Telecom call records
Financial applications
Web logs and click-streams
Manufacturing processes
Queues
RSS

3. Streams are Everywhere
Application
Application
Application
Application
Stream
File/DB
File/DB
Stream
Sensor Data
Actuators
Stream
Stream
Feeds
Feeds
Stream
Propagations
Propagations
A virtual stream; potentially based on a CQ
A materialized stream
Stream

4. Definition
… a body of water, confined within a bed and banks and having a detectable current – What you get from Google
Continuous, unbounded, rapid, time-varying streams of data elements – Jennifer Widom
A semi ordered and ever growing set of related data – Anonymous
… make up your own definition …

5. Specifications of Streams
Data model
Data structure of elements (NVP, CSV, SQL99, XML, RDF, ..) - the easy part
Relation between elements has to be added
Sequencing, other
Review presentation from Stanford Streams project and others
Data access
Continuous Queries; e.g., Subscriptions, CQL
Single elements – well understood
Windows, joins, aggregates – (very) limited understanding
Any language has to deal with evolution of streams
Ad-hoc data access assumes a static/stable data source
Ad-hoc access
Current languages are very weak on temporal support

6. Specifications of Streams
Client model and tools
We need an additional client pattern
Use consumption of queues as guide
(Exactly once) consumption
E.g., make sure that ‘new’ data are processed exactly once even in the presence of application and system failures
State management - to protect against failures
E.g., prevent extended locking – commit after consumption of new data element
Management (logical properties)
Retention
Other

7. The Creation of Streams
Creation of (initial) streams
Applications
Fixed
Based on parameters
Always, on demand
Temporal states
Continuous query must be able to identify events
Creation of (derived) streams by evaluation of other streams:
Single element (classical subscription)
Windows – CQL like languages
Joining of streams – CQL like language with extensions
Joining stream with non-stream data – tricky (temporal reference)
Many policies
See next slide

8. Distribution and consumption of Streams
Messaging, RSS, …
Propagation
Consumption
Single client, group of clients
Policies
Best effort, auditable, non-reputable
Fair
According to community policies
According to constraints defined by publisher and consumer
Best effort
Non transactional, transactional (exactly once)

9. Operational Characteristics
Scalability/performance
Large number of (streams) elements, publishers, consumers (clients)
Reliability/Recoverability
Streams may represented business interaction; e.g., may represent most valuable business (data) asset
Security
Should be aligned with database/file security; e.g., fine grain, context oriented
Transactional
Elements of some streams need to be created, propagated consumed exactly once
Fair
Everyone gets information at the same time
… and more

10. Streams and Events The elements of streams are messages
Message report about events
Event: A change of a state that is of interest
Event and message specification
Newton Model: Events and messages are pre-efined – take what you get, select a subset
Heisenberg Model: Events and messages can be defined by subscriber/consumer
Requires temporal support
Requires a language to support event/message specification
It is a significant challenge to identify important events and related messages