Go Stream Matvey Arye, Princeton/Cloudflare Albert Strasheim, Cloudflare

Awesome CDN service for websites big & small Millions of request a second peak 24 data centers across the globe

Data Analysis – Customer facing analytics – System health monitoring – Security monitoring => Need global view

Functionality Calculate aggregate functions on fast, big data Aggregate across nodes (across datacenters) Data stored at different time granularities

Storm & Rainbird

Basic Design Requirements 1.Reliability – Exactly-once semantics 2.High Data Volumes

Our Environment Source Storage Source Stream processing

Basic Programming Model Op Storage Op Storage Op Storage Op

Existing Systems S4 The reliability model is not consistent Storm Exactly-once-semantics requires batching Reliability only inside the stream processing system What if a source goes down? The DB?

The Need For End-to-End Reliability Source Stream Proccessing Storage When source comes back up where does it start sending data from? If using something like Storm, need additional reliability mechanisms

The Takeaway Need end-to-end reliability - Or - Multiple reliability mechanisms Reliability of stream processing not enough

Design of Reliability Avoid queuing because destination has failed – Rely on storage at the edges – Minimize replication Minimize edge cases No specialized hardware

Big Design Decisions End-to-end reliability Only transient operator state

Recovering From Failure Source I am starting a stream with you. What have you already seen from me? Storage I’ve seen Source Okie dokie. Here is all the new stuff.

Tracking what you have seen Store identifier for all items Store one identifier for highest number

Tracking what you have seen Store identifier for all items The answer to what have I seen is huge Requires lots of storage for IDs Store one identifier for highest number Parallel processing of ordered data is tricky

Tension between Parallelization High Volume Data Ordering Reliability

Go Makes This Easier Language from Google written for concurrency Goroutine I run code Goroutine I run code Goroutine I run code Goroutine I run code Channels send data between Go routines Most synchronization is done by passing data

Goroutine Scheduling Channels are FIFO queues that have a maximum capacity So goroutine can be in 4 states: 1.Executing Code 2.Waiting for a thread to execute code 3.Blocking to receive data from a channel 4.Blocking to send data to a channel Scheduler optimizes assignment of goroutines to threads.

Efficient Ordering Under The Hood Source distributes items to workers in a specific order Reading from each worker: 1.Read one tuple off the count channel. Assign count to X 2.Read X tuples of the result channel Count of output tuples for each input Actual result tuples Input tuple

Intuition behind design Multiple output channels allows each worker to write independently. Count channel tells reader how many tuples to expect. Does not block except when result needed to satisfy ordering. Judicious blocking allows scheduler to use blocking as a signal for which worker to schedule.

Throughput does not suffer

The Big Picture - Reliability Source provide monotonically increasing ids – per stream Stream processor preserves ordering – per source-stream Central DB maintains mapping of: Source-stream => highest ID processed

Functionality of Stream Processor Compression, serialization Partitioning for distributed sinks Bucketing – Take individual records and construct aggregates Across source nodes Across time – adjustable granularity Batching – Submitting many records at once to the DB Bucketing and batching all done with transient state

Where to get the code Stable Bleeding Edge

Data Model Streaming OLAP-like cubes Useful summaries of high-volume data

Cube Dimensions 01:01:00 foo.com/rfoo.com/qbar.com/nbar.com/m 01:01:01 Time URL 27

Cube Aggregates (Count, Max) bar.com/m 01:01:01 28

Updating A Cube Request #1 bar.com/m 01:01:00 Latency: 90 ms Request #1 bar.com/m 01:01:00 Latency: 90 ms (0,0) 01:01:00 foo.com/rfoo.com/qbar.com/nbar.com/m 01:01:01 Time URL 29

Map Request To Cell Request #1 bar.com/m 01:01:00 Latency: 90 ms Request #1 bar.com/m 01:01:00 Latency: 90 ms (0,0) 01:01:00 foo.com/rfoo.com/qbar.com/nbar.com/m 01:01:01 Time URL 30

Update The Aggregates Request #1 bar.com/m 01:01:00 Latency: 90 ms Request #1 bar.com/m 01:01:00 Latency: 90 ms (1,90) (0,0) 01:01:00 foo.com/rfoo.com/qbar.com/nbar.com/m 01:01:01 Time URL 31

Update In-Place Request #2 bar.com/m 01:01:00 Latency: 50 ms Request #2 bar.com/m 01:01:00 Latency: 50 ms (2,90) (0,0) 01:01:00 foo.com/rfoo.com/qbar.com/nbar.com/m 01:01:01 Time URL 32

Cube Slice 01:01:00 foo.com/rfoo.com/qbar.com/nbar.com/m 01:01:01 Time URL … 01:01:58 01:01:59 Slice 33

Cube Rollup 01:01:0 0 foo.com/rfoo.com/qbar.com/nbar.com/m Time URL URL: bar.com/* Time: 01:01:01 URL: foo.com/* Time: 01:01:01 34

Rich Structure (5,90) (3,75) (8,199) (21,40) D A C 01:01:59 01:01:00 foo.com/rfoo.com/r foo.com/qfoo.com/q bar.com/nbar.com/n bar.com/m 01:01:01 01:01:58 B … E CellURLTime Abar.com/*01:01:01 B* Cfoo.com/*01:01:01 Dfoo.com/r01:01:* Efoo.com/*01:01:* 35

Key Property 2 types of rollups 1.Across Dimensions 2.Across Sources We use the same aggregation function for both Powerful conceptual constraints Semantic properties preserved when changing the granularity of reporting

Where to get the code Stable Bleeding Edge