Top Half / Bottom Half Processing David Ferry, Chris Gill, Brian Kocoloski CSE 422S - Operating Systems Organization Washington University in St. Louis St. Louis, MO 63130

Bottom Halves Defer Work Modern interrupt handlers are split into top half (fast) and bottom half (slow) components Top half: Does minimum work to service hardware Sets up future execution of bottom half Clears interrupt line Bottom half: Performs deferred processing Example: Network card – top half clears card buffer while bottom half processes and routes packets CSE 422S – Operating Systems Organization

Advantages of Bottom Halves 1. Allow the kernel to run time-sensitive work before your bottom half 2. Allow your deferred work to itself be pre-emptible, because it runs with interrupts enabled Some interrupt handles run with all interrupts disabled (IRQF_DISABLED) Others only disable the current interrupt line, which could still be problematic All other devices that use that interrupt line are blocked e.g., devices on PCI buses can share interrupt lines CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Interrupt context Note: there is a key distinction between interrupt context and execution with interrupts disabled Top halves run in interrupt context and with interrupts disabled Bottom halves can run in interrupt context, but run with interrupts enabled Why is this distinction important? Disabling interrupts prevents the kernel from running higher priority work – i.e., it creates non-determinism from the kernel’s perspective CSE 422S – Operating Systems Organization

When does bottom half work run? At some point in the future Bottom halves gives the kernel the opportunity to decide whether or not now is a good time to do this work That means: The kernel could decide to run your bottom half immediately The kernel could decide it is higher priority work to do You thus have no guarantee on when deferred work will finish CSE 422S – Operating Systems Organization

Bottom Half Mechanisms Softirqs (interrupt context) Statically defined Same softirq handler can execute concurrently on different processors, so may be more difficult to program correctly Offer most concurrency at the expense of being harder to program - locking must be done by driver Tasklets (interrupt context) Should be your default choice over softirqs Tasklet handlers of same type do not execute concurrently Work Queues (process context) Defers work to generic kernel threads (kworker) It’s ok for the work function to sleep, block, etc. Replaces having to make your own deferred kernel thread CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Softirqs Most complex bottom half mechanism Only used in a small set of situations When the work to do is substantial and can benefit from concurrency i.e., when you want to run your bottom half work concurrently on multiple cores Rarely used directly by programmers Defined at compile time, code can’t register dynamically Need to watch out for data races (can execute concurrently) Stored in a fixed-sized array of pointers to “functors” (struct with action handler function pointer as a member CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization When do softirqs run? Consider the example of a network interrupt handler Some packets come in, the top-half copies them to DRAM, lowers the interrupt, and then schedules the bottom-half softirq to run (NET_RX_SOFTIRQ) Q: When will the bottom half run? A: it depends Return from hardware interrupt code path (i.e., immediately) In the ksoftirqd kernel thread (i.e, once the kernel schedules a kernel thread to execute it) In any code that explicitly checks for the existence of softirq work CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Tasklets Tasklets may be easier to use than (non-tasklet) softirqs Can be registered dynamically (by code) or at compile time Tasklets are (implemented atop) softirqs Can register as HI_SOFTIRQ (run at high priority) or as TASKLET_SOFTIRQ (run at normal priority) Tasklet handlers of same type can’t run concurrently Can share data (across processors) w/out locking (still can’t block) But, don’t need to worry about data races within a handler type CSE 422S – Operating Systems Organization

Review: when do Softirqs (and therefore, tasklets) run? 3 ways the kernel checks for pending softirqs (LKD pp. 138): In the return from hardware interrupt code path In the ksoftirqd kernel thread In any code that explicitly checks for and executes softirqs, such as the networking subsystem CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Work Queues Work is deferred to run in kernel threads Execute in process context (not in interrupt context) So, can sleep, lock, block, etc. Thus, any work that must sleep/block/etc. should run there Data structure for work queues One is defined for each type of worker thread Per CPU: work items, a thread task structure, and a spin lock Work structure packages up data and processing Work is implemented as pointer to function that is run While work list isn’t empty, thread calls each entry’s function When work list is empty, thread goes to sleep This approach is suitable for many deferred work situations CSE 422S – Operating Systems Organization

Which bottom half should you use? Does your work absolutely need to do operations that may block? E.g., large memory allocations If so, use work queues Is your work very time sensitive, or does it need concurrency? If so, use softirqs You need to modify the kernel source to use this operation – i.e., no kernel modules Otherwise, use tasklets CSE 422S – Operating Systems Organization