1 SEDA: An Architecture for Well- Conditioned, Scalable Internet Services Matt Welsh, David Culler, and Eric Brewer Computer Science Division University of California, Berkley

2 An Application for:  servicing requests for Internet services  servicing a load (demand) that is never constant  providing a service that scales to load  keep in mind these issues of scalability:  load fluctuates, sometimes wildly  you only want to allot the resources servicing the load requires  every system has a limit! Scale responsibly!  don’t over commit the system SEDA

3 Design Goals:  service requests for BOTH static content AND dynamic content  demand for dynamic content is increasing  work to deliver static content is predictable  retrieve the static content from disk or cache  send it back on the network SEDA

4 Design Goals:  work to deliver dynamic content is unknown  build content on the fly  how many I/Os to retrieve the content?  layout, insert, and format content  may require substantial computation  posing queries of a database and incorporating results  all the more reason to scale to load! SEDA

5 Design Goals:  adaptive service logic  different levels of load require different service strategies for optimum response  load is determined by BOTH the number of requests for service and the work the server must do to answer them  platform independence by adapting to resources available  load is load, no matter what your system is SEDA

6 Consider (just) Using Threads:  the ‘model’ of concurrency  two ways to just use threads for servicing internet requests:  unbounded thread allocation  bounded thread pool (Apache)  hint: remember !!  (somewhat different reasons though!) SEDA Threads!

7 Consider (just) Using Threads:  unbounded thread allocation  issues thread per request  too many use up all the memory  scheduler ‘thrashes’ between them, CPU spends all its time in context switches  bounded thread pool  works great until every thread is busy  requests that come in after suffer unpredictable waits --> ‘unfairness’ SEDA

8 Consider (just) Using Threads:  transparent resource virtualization (!)  fancy term for the virtual environment threads/processes run in  threads believe they have everything to themselves, unaware of constraint to share resources  t. r. v. = delusions of grandeur!  no participation in system resource management decisions, no indication of resource availability to adapt its own service logic SEDA

9 DON’T Consider Using Threads! SEDA  threaded server throughput degradation  as the number of threads spawned by the system rises, the ability of the system to do work declines  throughput goes down, latency goes up

10 Consistent Throughput is KEY:  well-conditioned service:  sustain throughput by acting like a pipeline  break down servicing of requests into stages  each stage knows its limits and does NOT over provision  so maximum throughput is a constant no matter the varying degree of load  if load exceeds capacity of the pipeline, requests get queued and wait their turn. So latency goes up, but throughput remains the same. SEDA

11 Opaque Resource Allocation:  incorporate Event-Driven Design in the pipeline  remember -- only one thread!  no context switch overhead  paradigm allows for adaptive scheduling decisions  adaptive scheduling and responsible resource management are the keys to maintaining control and not over-committing resources to account for unpredictable spikes in load SEDA

12 Back to the argument:  Event-driven Programming vs. Using Threads  the paper recommends event-driven design as the key to effective scalability, yet it certainly doesn’t make it sound as easy as Ousterhout  event handlers as finite state machines  scheduling and ordering of events ‘all in the hands’ of the application developer  certainly looks at the more complex side of event-driven programming SEDA

13 Anatomy of SEDA:  Staged Event Driven Architecture  a network of stages  one big event becomes series of smaller events  improves modularity and design  event queues  managed separately from event handler  dynamic resource controllers (it’s alive!)  allow apps to adjust dynamically to load  culmination is a managed pipeline SEDA

14 Anatomy of SEDA:  a queue before each stage  stages queue events for other stages  note the modularity  biggest advantage is that each queue can be managed individually SEDA

15 Anatomy of a Stage:  event handler  code is provided by application developer  incoming event queue  for portions of requests handled by each stage  thread pool  threads dynamically allocated to meet load on the stage  controller  oversees stage operation, responds to changes in load, locally and globally SEDA

16 Anatomy of a Stage:  thread pool controller manages the allocation of threads. Number is determined by length of queue  batching controller determines the number of events to process in each invocation of the handler SEDA

17 Asynchronous I/O:  SEDA provides two asynchronous I/O primitives  asynchronous = non-blocking  asynchronous socket I/O  intervenes between sockets and application  asynchronous file I/O  intervenes to handle file I/O for application  different in implementation, but design of each uses event-driven semantics SEDA

18 Performance:  througput is sustained! SEDA

19 Summary:  load is unpredictable  most efficient use of resources is to scale to load without over committing  system resources must be managed dynamically and responsibly  staged network design culminates in a well- conditioned pipeline that manages itself  event-driven design for appropriate scalability  threads are used in stages when possible SEDA