Rapid Development of High Performance Servers Khaled ElMeleegy Alan Cox Willy Zwaenepoel

Goals Maximize server utilization and decrease its latency. –Server process doesn’t block! Robust under heavy loads. Relatively easy to develop.

Key Idea Get the best of the two worlds. Event Driven Multi-Threaded Programming level: Implementation level: Multi-threaded serversEvent driven servers Lazy threaded servers

1-KQueue Kernel event notification mechanism. Supported by FreeBSD Kernel. Events queued in a kernel queue, and delivered to user by kernel calls. Events usually correspond to AIO completions, or socket reads/writes. Background

2-AIO Asynchronous I/O mainly used for disk reads/writes. Issue disk request and return immediately without blocking for results. On completion an event is sent to KQueue.

3-Servers Architectures: a- Multi-threaded Servers  Kernel thread per request executing the request handler.  A thread blocking on I/O doesn’t block the server.  Ex: Apache, MySql. Request Handler A B C Function Calls

Multi-threaded Servers (Cont.) Advantages: + Relatively easy to develop. Disadvantages: - Not good performance. Thread scheduling. Context switching. - Doesn’t scale. Under heavy load, many threads would be created exhausting kernel’s memory, crashing the server.

 Single thread (event loop).  AIO operations.  I/O completions / Requests (events) put in Event Queue.  Continuations.  Ex:Flash web server (slightly different architecture). + High performance. + Scalable/Robust. - Hard to write. b- Event Driven Servers Event Loop A1 B1 C1 A2 B2 C2 Events From Kernel “Blocking” operation Event Queue (KQueue)

Lazy Threading Similar design to event driven but no continuations. Threads yield control on fixed points (I/O).  If request handler doesn’t “block” –Run handler to completion then use same thread to handle next event.  Else –Suspend current thread, create another user thread to handle next event.

Wrappers / Thread Suspension IO_Wrapper(..) { uc = save_user_context(..); /* save the current user thread context to be able to resume it later */ aio_func(..); /* non-blocking I/O call */ stack=allocate_stack(); switch_thread(event_loop,stack); /* suspend current thread and create a new thread running the event loop using the newly allocated stack */. } Wrappers around “blocking” operations to create the blocking illusion. Uses AIO. Saves current stack instead of unwinding it. Creates a user thread by jumping to the event loop and using a new stack.

Thread Resumption Event_loop() {. e = get_event(..); /* read new event from the kernel’s kqueue */ if e is completion of aio_func(..) run_thread(uc); /* dispose current thread and resume the IO_Wrapper from where it has stopped */. } Restores registers and jumps back to wrapper as if blocking has finished. Current thread is disposed as it’s useless. Wrapper’s thread eventually returns to the event loop after handling its request.

ServLib Library for fast server development. Provides an infrastructure for building lazy-threaded servers. Hides all the lazy threading details from the server developer. Used to upgrade existing servers, or build new servers.

How to use ServLib Extract the event handler code from the old server/some library or write your own. Initialize ServLib with : –request handler function. –listening port number. Change all blocking calls in the request handler to use the wrappers exported by ServLib.

Experimentation Implementing then comparing the performance of 3 versions of a database server (from sqlite library). And of a similar 3 versions of a web server (from mini_httpd). –A multi-threaded version, –An event driven version and –A lazy threaded version. Use FreeBSD 4.6

Summary Yes NoHigh Performance No YesThread / request? Yes (user) NoYes (kernel) >1 thread NoN/AYesArbitrary thread yield YesNoYes“Easy” to build Yes NoScalable Lazy- Threaded Event Driven Multi- Threaded

Questions?