SEDA: An Architecture for Scalable, Well-Conditioned Internet Services Matt Welsh, David Culler, and Eric Brewer Computer Science Division University of California, Berkeley Presented By: Linh Nguyen

Agenda Motivations and background SEDA Evaluations Concurrencies Goals Stages Queues Dynamic Resource Schedulers Evaluations Efficient methods to solve problems using factored representation Goal: each variable has a value that satisfies all constraints on variable

Motivations High-volume Internet Services The Slashdot effect Yahoo: 1.2 billions page views per day AOL: 10 billions hits per day Now: Google at 3.5 billion searches per day and increasing The Slashdot effect Peak loads can be 100-fold larger than normal In 2009, Google mistook the surge in Michael Jackson-related searches for an attack Increasingly dynamic Dynamic content: Google Maps, Social Networks Service logic changes frequently: Providers want to add functionalities, improvements, etc Services are hosted on general purpose facilities

Thread-based concurrency EECS 492 Fall 2004 28 Sep 04 Thread-based concurrency Each request consumes a thread to process it Synchronization to protect shared resources OS switches between threads Can be achieved in two ways: Thread-per-task model Bounded thread pools M. Wellman

Characteristics of thread-based concurrency Easy to program, since all resources are virtualized. However, applications are not able to manage resources Associated overheads: Cache misses Scheduling overhead Lock contention When number of threads is large, throughput is degraded Bounded thread pools lead to: Unfairness to clients Large waiting times

Event-based concurrency EECS 492 Fall 2004 28 Sep 04 Event-based concurrency Main thread processes incoming events and drives the executions of FSMs Each FSM represents a single request or flow of execution Scheduler controls the execution of each FSM M. Wellman

Characteristics of event-driven concurrency Little degradation in throughput Assume non-blocking I/O, which may not be supported Event scheduling is tied to application logic (when to process incoming event, in what order to process the FSMs) -> harder to modularize

Solutions? Support massive concurrency Simplify building well-conditioned services Enable introspection Support self-tuning resource management Staged event-driven architecture (SEDA)

SEDA Draws from both thread-based (ease of programming) and event-based (extensive concurrency) models Each service is a network of stages Stages connected by event queues

Stage Small number of threads per stage Dynamic controllers automatically tune the number of threads allocated based on demand Event handler provides core logic, processes a batch of multiple events, and implements resource-management policies

Queue (thanks Norman Hutchinson of UBC) Queues are finite An enqueue behavior may fail Block on full queue -> backpressure Drop rejected events -> load shedding Queue introduces explicit execution boundary Threads may only execute within a single stage Performance isolation, modularity, independent load management Explicit event delivery support inspection Trace flow of events through application Monitor queue lengths to detect bottleneck

Dynamic Resource Controllers Batching controller Adjusts batching factor to stabilize throughput Large factor: higher throughput Small factor: Lower response time Thread pool controller - Determines the degree of concurrency Add thread if queue length > threshold Remove idle threads

Evaluations “closed-loop” server: Haboob, comparisons with Apache and Flash “open-loop” server: Gnutella

Haboob performance analysis Jain Fairness Measurement 𝑓(𝑥): fairness in term of clients and their associated requests 𝑁 : number of clients 𝑋 𝑖 : number of requests for each of N clients 𝑓(𝑥) = 1: server is equally fair to all clients

Haboob performance analysis (cont.)

Observations Haboob achieves higher throughput than Apache and Flash as the number of clients reach 64 Apache and Flash exhibit high frequencies of low response times Long tail in Apache and Flash Haboob (SEDA) yields predictable performance, while Apache is very unfair

Gnutella Non-traditional Internet services An user searches for and downloads files from other users Five types of messages: ping, pong, query, queryhits, push.

Gnutella Performance Higher packet latencies for 100 and 200 packets/sec since no threads were added For 400 and 1000 packets/sec two threads were added

Summaries Achieved its goals: Support massive concurrency: Combines event-driven and thread-based Enable introspection to handle peak load: Event queue allows for inspection of request streams Dynamic resource management: Feedback-control with resource controllers Simplify task of building well-conditioned services: Abstract scheduling and resource management from programmers; easier modularization.

Discussions Decisions on how to break down an application into stages Scalability? A retrospective on SEDA Ousterhoust’s why threads are a bad idea? Brewer’s why events are a bad idea?