1. SEDA: An Architecture for Well-Conditioned, Scalable Internet Services
Presented by
Kristen Carlson Accardi

2. CS533 - Concepts of Operating Systems
Programming for Internet Services requires different design considerations
Internet places huge demands on providers of services
High availability and scalability are new requirements
Load patterns are bursty
Services have become increasingly complex
Dynamic web content
Security
Interaction with other services
Services are being hosted on General Purpose platforms that were not designed to optimally perform these services
Providers of Internet services have vastly different requirements than other applications. They must support massive amounts of currency. Providers such as yahoo, google etc. will get 10s of billions of hits per day. In addition, the complexity of these services are increasing rapidly. Now, you need to provide database access, plus security, plus dynamic web content. General purpose OSs are designed to abstract resources, and often don’t have a way of guaranteeing access to resources, or allowing applications to specifically control access to resources. These things are necessary in order to scale to this demand.
CS533 - Concepts of Operating Systems

3. CS533 - Concepts of Operating Systems
SEDA Goals
Provide a programming framework for massively concurrent Internet Service applications
Handle load peaks gracefully (must not appear “down”)
adapt to change
Reduce overhead of concurrent processing
SEDA was introduced around 2000 by Matt Welsh as his PHD thesis project. It continues to be developed, and has a Java package known as “sandstorm” that has active users (and you can join the development team if you want, it’s on sourceforge). It is also being used as a base system for future work on tuning resource management for specific apps. Work on SEDA nbio was a foundation for the new java nio package.
CS533 - Concepts of Operating Systems

4. CS533 - Concepts of Operating Systems
What is SEDA?
A design framework
“Staged Event-Driven Architecture”
Combines threads and event based programming models
Applications are constructed as a network of “stages”
Each stage has in incoming event queue. The event queue is what is used to handle the load.
The unique thing about SEDA is combination of threads and events. It is designed to be a general purpose framework for Internet Service applications, and to shield programmers from needing to worry about resource scheduling and allocation. User’s of the framework will focus simply on the application specific logic. They implement an event handler that is associated with a queue. Applications consist of a network of stages, and each stage is separated by input queues. To move to the next stage, an event handler will enqueue an event onto another stage’s input queue.
CS533 - Concepts of Operating Systems

5. Thread model alone is insufficient
Usual thread per request model doesn’t scale due to concurrency overhead
Lock contention
Context switching
Memory footprint
Bounding number of threads causes long wait, which leads to the perception that a service is not available
Doesn’t work for bursty loads
Each time you create a thread, there is overhead associated with that. You may not need to allocate a new address space for the thread, but the thread has it’s own local memory, code space etc. Just the OS structure to hold the thread state alone will take up memory, not to mention the memory allocated for that threads use. Plus, now the thread must be managed. It must be scheduled, activated, destroyed. Not to mention, all threads will likely need to access global data, which requires locking of some type. With massively concurrent applications (think many many many threads), the lock contention will be huge. In addition, the latency between when a thread gets rescheduled after being swapped out could be huge when you have many many threads. Some applications will try to solve this issue by bounding the number of threads that can be created. This means that under heavy load, there may not be enough threads to handle all requests, so connections will be ignored, creating the impression that a service is not available. This is a huge no-no. Imagine our dismay if Google was “down”! If you try to bound based on the upper limit of what you think your traffic is going to be, then you are most of the time over provisioned. But, how can you predict an upper limit?
CS533 - Concepts of Operating Systems

6. Event model alone is insufficient
Event driven programming does scale better than thread based programming
Programming is complex due to the having to keep a lot of state
Ordering issues also increase complexity
Event driven models are not extensible usually, and they are also very difficult to do modularly.
In event driven programming, each task is represented as a finite state machine. Most of the time, event based programs have a few threads, and will keep state for each event that has occurred. The problem with the event model is that it’s hard to implement. And, once you implement it, it’s hard to maintain it, and hard to add features to it. It’s difficult to manage exceptions too.
CS533 - Concepts of Operating Systems

7. SEDA combines threads & events to address these issues
Supports massive concurrency by making use of event-driven execution when possible
Simplifies programming of “well-conditioned” services
“Well conditioned” means handles load gracefully
Shields the programmer from needing to handle scheduling and resource management
Allows applications to adapt to changing loads
Supports self-tuning resource management by allowing resource management parameters to adjust dynamically
As I mentioned, SEDA is a combination of thread based programming and event based programming. Because it is a software framework, it attempts to provide an easier way of getting the concurrency that event driven programming provides, with the easier programming model of the threads. The most powerful thing about SEDA is the ability to adapt to changing loads. It is able to do this because it decouples the event handling from the resource management & scheduling. By monitoring the input and output queues, it can self-tune and adjust dynamically.
CS533 - Concepts of Operating Systems

8. CS533 - Concepts of Operating Systems
Anatomy of a “stage”
Event Queue
Event Handler
Thread pool
Managed by a “controller” which affects scheduling and allocation of threads
Applications are designed as a network of stages
A stage has 3 basic components: An event queue, an event handler and a thread pool. Each stage is managed by a controller. An application programmer would provide the event handler. A SEDA based application would be made of a network of stages, separated by queues.
CS533 - Concepts of Operating Systems

9. Advantages of the stage
Pool of threads limits number of threads per stage to some maximum
Controller can tune number of threads used dynamically based on load
Separation of the event handler from the queuing/thread ops allows the stage to control scheduling, making application programmers able to focus on application specific logic.
Debugging can be simplified because you can easily trace messages hitting input/output queues in each stage
CS533 - Concepts of Operating Systems

10. CS533 - Concepts of Operating Systems
Types of controllers
Thread pool controllers
Adjusts number of threads executing within each stage by sampling the input queue.
Can add up to some max number threads based on event thresholds
Batching controllers
Controls the number of events processed by the event handler at a time.
The controller must trade off throughput w/latency. It monitors the output queue rate to detect degradation in throughput.
You could spend a lot of time messing with controller algorithms & adding new controllers!
There are 2 types of controllers that are discussed in the paper, although theoretically you can add additional controllers. Thread pool controllers control the number of threads per stage. The controller will monitor the input queue to determine whether or not new threads should be added/deleted based on the incoming rate of events. Some threshold can be set to trigger allocation of additional threads. It will also set an upper bound on the number of threads that can be allocated. The batching controller will control how many events are sent to the event handler at a time. It’s usually faster to process a bunch of events at once than one at a time. Your cache is fresh this way. Unfortunately, since this is a staged architecture, the longer one stage is executing, the more latency between stages, so there is a trade off to be made. The controller can monitor the rate at which packets are being output to the next stage to determine if throughput is dropping off due to latency. If it is, then it can decrease the number of events sent to the handler.
These controllers are an area where people are doing research now to determine optimal algorithms for resource management under internet loads.
CS533 - Concepts of Operating Systems

11. CS533 - Concepts of Operating Systems
SEDA implementations
Sandstorm – the SEDA implementation
Written in Java
Designed to be an internet services platform.
Provides AIO
Stages implemented on top of AIO
Haboob – a web server
Written in Java, yet outperforms many C web servers
GNUtella packet router
As I mentioned, a prototype of SEDA was implemented in Java and is still being used today. Some applications were written on top of SEDA in order to provide some initial performance numbers for SEDA based architectures. The upshot is that even though SEDA was implemented in Java, due to it’s design the applications are able to outperform many C-based applications. Note that all of these applications have the characteristic that they have to handle a lot of packet based IO, and the packets also tend to be smaller (minimum sized for GNUtella) or contain a lot of control packets, so it has a lot of overhead in these applications. This overhead would cause tons of additional latency on threads based applications.
CS533 - Concepts of Operating Systems

12. CS533 - Concepts of Operating Systems
Conclusions
Design requirements are (should be) different for Internet services
Handling load variations gracefully should be a requirement
Event queues combined with dynamic resource controllers allow scheduling & resource management to be tuned for a specific service
The author wants you to take away from this paper that application programming for internet services is vastly different. Having the power to dynamically adjust to changing loads allows applications to be tuned for a specific service without requiring the programmer to write their own resource management software for every particular application.
CS533 - Concepts of Operating Systems