1. QCon.ai, San Francisco April, 11th 2018
Streaming SQL to Unify Batch and Stream Processing: Theory and Practice with Apache Flink at Uber
Fabian Hueske
Shuyi Chen
QCon.ai, San Francisco April, 11th 2018

2. What is Apache Flink? Stateful Computations Over Data Streams
Data Stream Processing
realtime results from data streams
Batch Processing
process static and historic data
Event-driven
Applications
data-driven actions and services
Stateful Computations Over Data Streams
Flink is the is a system that handles the full breadth of stream processing starting from bounded (or batch) data over streaming analytics to streaming data applications.
The important point here is that Flink addresses the analytical as well as the transactional spectrum of stream processing.

3. Stateful computations over streams
What is Apache Flink?
Stateful computations over streams
real-time and historic fast, scalable, fault tolerant, in-memory, event time, large state, exactly-once
Application
Queries
Applications
Streams
Database
Devices
Stream
etc.
Historic Data
File / Object Storage
Flink achieves that by providing an extensive feature set such as: exactly-once state semantics, handling very large state, and support for event-time processing.

4. Hardened at scale Streaming Platform Service
billions messages per day
A lot of Stream SQL
Streaming Platform as a Service
3700+ container running Flink,
1400+ nodes, 22k+ cores, 100s of jobs
100s jobs, 1000s nodes, TBs state,
metrics, analytics, real time ML,
Streaming SQL as a platform
Fraud detection
Streaming Analytics Platform
All of these features are proven to work at large scale in terms of data volume, cluster size, state size, number of jobs and for a wide variety of use cases.
Many companies and enterprises run their stream processing workloads on Flink.
And some, such as Alibaba and Uber, built their internal stream processing platforms based on Flink and more specifically on Flink’s SQL interface.

5. Powerful Abstractions
Layered abstractions to navigate simple to complex use cases
High-level Analytics API
SQL / Table API (dynamic tables)
val stats = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum((a, b) -> a.add(b))
Stream- & Batch Data Processing
DataStream API (streams, windows)
Process Function (events, state, time)
Stateful Event- Driven Applications
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = {
// work with event and state
(event, state.value) match { … }
out.collect(…) // emit events
state.update(…) // modify state
// schedule a timer callback
ctx.timerService.registerEventTimeTimer(event.timestamp + 500) }
Flink offers APIs for different levels of abstraction that all can be mixed and matched.
On the lowest level, ProcessFunctions give precise control about state and time, i.e., when to process data.
The intermediate level, the so-called DataStream API provides higher-level primitives such as for window processing
Finally, on the top level the relational API, SQL and the Table API, are centered around the concept of dynamic tables. This is what I’m going to talk about next.

6. Apache Flink’s Relational APIs
ANSI SQL
LINQ-style Table API
SELECT user, COUNT(url) AS cnt
FROM clicks GROUP BY user
tableEnvironment .scan("clicks")
.groupBy('user)
.select('user, 'url.count as 'cnt)
Unified APIs for batch & streaming data
A query specifies exactly the same result regardless whether its input is static batch data or streaming data.

7. Input data is unbounded
Query Translation
tableEnvironment .scan("clicks")
.groupBy('user)
.select('user, 'url.count as 'cnt)
SELECT user, COUNT(url) AS cnt
FROM clicks
GROUP BY user
Input data is bounded
(batch)
Input data is unbounded
(streaming)

8. What if “clicks” is a file?
Input data is read at once
Result is produced at once
user
cTime
url
Mary
12:00:00
Bob
12:00:02
Liz
12:00:03
SELECT
user,
COUNT(url) as cnt FROM clicks GROUP BY user
user
cnt
Mary 2
Bob 1
Liz

9. What if “clicks” is a stream?
Input data is continuously read
Result is continuously
produced
user
cTime
url
SELECT
user,
COUNT(url) as cnt FROM clicks GROUP BY user
user
cnt
Mary
12:00:00
Mary 1
Mary 2
Bob
12:00:00
Bob 1
Mary
12:00:02
Liz 1
Liz
12:00:03
The result is identical!

10. Why is stream-batch unification important?
Usability
ANSI SQL syntax: No custom “StreamSQL” syntax.
ANSI SQL semantics: No stream-specific results.
Portability
Run the same query on bounded and unbounded data
Run the same query on recorded and real-time data
Do we need to soften SQL semantics for streaming?
now
bounded query
unbounded query
past
future
start of the stream

11. DBMSs Run Queries on Streams
Materialized views (MV) are similar to regular views, but persisted to disk or memory
Used to speed-up analytical queries
MVs need to be updated when the base tables change
MV maintenance is very similar to SQL on streams
Base table updates are a stream of DML statements
MV definition query is evaluated on that stream
MV is query result and continuously updated

12. Continuous Queries in Flink
Core concept is a “Dynamic Table”
Dynamic tables are changing over time
Queries on dynamic tables
produce new dynamic tables (which are updated based on input)
do not terminate
Stream ↔ Dynamic table conversions

13. Stream ↔ Dynamic Table Conversions
Append Conversions
Records are only inserted/appended
Upsert Conversions
Records are inserted/updated/deleted and have a (composite) unique key
Changelog Conversions
Records are inserted/updated/deleted

14. SQL Feature Set in Flink 1.5.0
SELECT FROM WHERE
GROUP BY / HAVING
Non-windowed, TUMBLE, HOP, SESSION windows
JOIN
Windowed INNER, LEFT / RIGHT / FULL OUTER JOIN
Non-windowed INNER JOIN
Scalar, aggregation, table-valued UDFs
SQL CLI Client (beta)
[streaming only] OVER / WINDOW
UNBOUNDED / BOUNDED PRECEDING
[batch only] UNION / INTERSECT / EXCEPT / IN / ORDER BY

15. What can I build with this?
Data Pipelines
Transform, aggregate, and move events in real-time
Low-latency ETL
Convert and write streams to file systems, DBMS, K-V stores, indexes, …
Convert appearing files into streams
Stream & Batch Analytics
Run analytical queries over bounded and unbounded data
Query and compare historic and real-time data
Data Preparation for Live Dashboards
Compute and update data to visualize in real-time

16. The New York Taxi Rides Data Set
The New York City Taxi & Limousine Commission provides a public data set about taxi rides in New York City
We can derive a streaming table from the data
Table: TaxiRides
rideId: BIGINT // ID of the taxi ride
isStart: BOOLEAN // flag for pick-up (true) or drop-off (false) event
lon: DOUBLE // longitude of pick-up or drop-off location
lat: DOUBLE // latitude of pick-up or drop-off location
rowtime: TIMESTAMP // time of pick-up or drop-off event

17. Identify popular pick-up / drop-off locations
Compute every 5 minutes for each location the number of departing and arriving taxis of the last 15 minutes.
SELECT cell, isStart, HOP_END(rowtime, INTERVAL '5' MINUTE, INTERVAL '15' MINUTE) AS hopEnd, COUNT(*) AS cnt FROM (SELECT rowtime, isStart, toCellId(lon, lat) AS cell FROM TaxiRides WHERE isInNYC(lon, lat)) GROUP BY cell, HOP(rowtime, INTERVAL '5' MINUTE, INTERVAL '15' MINUTE)

18. Average ride duration per pick-up location
Join start ride and end ride events on rideId and compute average ride duration per pick-up location.
SELECT pickUpCell, AVG(TIMESTAMPDIFF(MINUTE, e.rowtime, s.rowtime) AS avgDuration FROM (SELECT rideId, rowtime, toCellId(lon, lat) AS pickUpCell FROM TaxiRides WHERE isStart) s JOIN (SELECT rideId, rowtime WHERE NOT isStart) e ON s.rideId = e.rideId AND e.rowtime BETWEEN s.rowtime AND s.rowtime + INTERVAL '1' HOUR GROUP BY pickUpCell

19. Building a Dashboard Elastic Search Kafka SELECT cell, isStart,
HOP_END(rowtime, INTERVAL '5' MINUTE, INTERVAL '15' MINUTE) AS hopEnd,
COUNT(*) AS cnt
FROM (SELECT rowtime, isStart, toCellId(lon, lat) AS cell FROM TaxiRides
WHERE isInNYC(lon, lat))
GROUP BY cell,
HOP(rowtime, INTERVAL '5' MINUTE, INTERVAL '15' MINUTE)
Elastic Search
Kafka

20. Flink SQL in Production @ UBER

21. Uber's business runs in Real-Time

22. Challenges Operation Infrastructure Productivity
> 1 trillion real-time messages / day
At-least-once processing
Exactly-once processing
99.99% SLA on availability
99.99% SLA on subsecond latency
Operation
> 1000 streaming jobs
Multiple DCs
Productivity
Target audience
Operation people
Data scientists
Engineers
Integrations
Logging
Backend services
Storage systems
Data management
Monitoring

23. Stream processing @ Uber
Apache Samza (Since Jul. 2015)
Scalable
At-least-once message processing 
Managed state
Fault tolerance
Apache Flink (Since May, 2017)
All of above
Exactly-once stateful computation
Accurate
Unified stream & batch processing with SQL

24. Lifecycle of building a streaming job

25. Writing the job Business Logics Input Output Testing Debugging
Java/Scala
Streaming/batch
Duplicate code

26. Running the job Resource estimation Deployment Monitoring & Alerts
Logging
Maintenance
Manual process
Hard to scale beyond a few jobs

27. Job from idea to production takes days

28. How can we improve efficiency as a platform?

29. SELECT AVG(…) FROM eats_order WHERE …
 Flink SQL to be savior
SELECT AVG(…) FROM eats_order WHERE …

30. SELECT AVG(…) FROM eats_order WHERE …
Connectors
SELECT AVG(…) FROM eats_order WHERE …
HTTP
Pinot

31. UI & Backend services To make it self-service
SQL composition & validation
Connectors management

32. UI & Backend services To make it self-service
Job compilation and generation
Resource estimation
Analyze input
Analyze query
Test deployment
Kafka input rate
Hive metastore data
YARN containers
CPU
Heap memory
SELECT * FROM ...

33. UI & Backend services To make it self-service Job deployment Promote
Sandbox
Functional correctness
Play around with SQL
Staging
System generated estimate
Production like load
Promote
Production
Managed

34. UI & Backend services
To make it self-service
Job management

35. UI & Backend services To support Uber scale
Monitoring and alert automation
Auto-scaling
Job recovery
DC failover
Watchdog

36. AthenaX Uber's Self-service stream and batch processing platform SQL
Connectors
UI & backend services

37. Business use cases

38. Real-time Machine Learning - UberEats ETD
SELECT restaurant_id, AVG(etd) AS avg_etd FROM restaurant_stream GROUP BY TUMBLE(proctime, INTERVAL '5' MINUTE), restaurant_id

39. Powering Restaurant Manager
Better Data -> Better Food -> Better Business = A Winning Recipe
Eats restaurant manager blog

40. AthenaX wins SQL abstraction with Flink E2E Self service
Non-engineers to use stream processing
E2E Self service
Job from idea to production take minutes/hours
Centralized place to track streaming dataflows
Minimal human intervention, and scale operationally to ~ 1000 jobs in production

41. AthenaX Open Source
Uber engineering blog
Open source repository

42. Thank you!
@fhueske
@ApacheFlink
Available on O’Reilly Early Release!