The Structuring of Systems Using Upcalls David D. Clark (Presented by John McCall)

A Quick Review of Layers Many designs are organized in terms of multiple layers of abstraction By convention, we talk about layers like they’re stacked on top of each other The most basic layer is on the bottom

Example (Libraries)

Example (Protocols)

Implementing the Design We can assign a module to each layer What happens if a module fails? –Optimally, the failure is detected –Dependent modules may be compromised –But everything else should be unaffected

Layers as Processes Natural solution: give each module its own process –Different address spaces, so no corruption –But now all inter-module communication is asynchronous –In practice, this is really inefficient Even in LRPC: the data is copied too often Even in URPC: there are too many asynchronous handoffs

Layers as Modules (1) So, put the modules in a single address space, as demonstrated in “Swift”, their research O.S. For safety, they wrote Swift in CLU: –Strong compile-time typing –Dynamic bounds checking –Garbage collection

Layers as Modules (2) Modules share the same space, so calls between them are just procedure calls Modules have to be multithreaded: –Assigning a single thread to each module and asynchronously handing off at boundaries is slow –So a single thread follows a job through the layers –So each layer has to handle multiple jobs at once Is there a discipline about who calls whom?

Downwards Control Flow Normally, we think of control flowing from the top down (downcalls) –The application calls a library –The library calls the system Just a normal procedure call: the lower level executes for a while, then returns to the upper level

Upwards Control Flow Sometimes, it’s more natural for control to flow upwards –One job sends a signal to another –The system gets a network packet Just like a normal procedure call: the upper level executes for a while, then returns to the lower level

Duality (1) Upcalls aren’t required: –The upper level can poll the lower level, asking if any interesting asynchronous events have occurred –But this is an asynchronous handoff from the low-level event to the polling job –And there’s inherent overhead in polling for multiple events at once

Duality (2) Downcalls aren’t required either! –A system with only upcalls would handle asynchronous events efficiently –But lower layers would have to poll to see if higher layers have any requests So, they’re dual notions –When should we use upcalls? –When should we use downcalls? –Are there inherent dangers with upcalls?

Downcall-based Sockets downcall: create and bind a socket downcall: receive data if possible downcall: send data if possible downcall: tell me when I can send or receive more data on any of these sockets

Upcall-based Sockets downcall: create and bind a socket –We also need to tell the socket layer which procedures to use as upcalls upcall: some data just arrived upcall: do you have any data to send? downcall: call me back if you can, I have data to send

Advantages of Upcalls (1) The lower level can ask the upper level for “advice” –Easier to implement useful optimizations like “piggybacking” –Gives us natural flow control for sockets No polling overhead/bottleneck –The higher level receives direct notification of changes

Advantages of Upcalls (2) Delays some thread design decisions –What thread do we make an upcall in? A single thread for each client? A pool of threads for each client? A pool of threads shared across clients? –Any of these decisions may be appropriate, depending on the module –Polling code tends to structurally force an early decision that’s difficult to change later

Danger: Upward Dependency Lower levels are often shared among different upper levels (“clients”) Clients should be insulated from failures in other clients Solution: –Separate shared data into global and client- specific variables –Never upcall while holding a lock on global data –Never upcall in a thread you’re afraid to kill

Danger: Indirect Recursion (1) What if a layer downcalls in an upcall? –Downcall might change data in unexpected ways –Or it might just deadlock, if it tries to grab a lock held during the upcall No one solution, but three good options Possible solution #1: never hold locks during upcalls, reevaluate everything on return –Leads to clumsy and inefficient programs –Definitely a last-resort solution

Danger: Indirect Recursion (2) Possible solution #2: prohibit most/all recursive downcalls –Seems very restrictive, but it usually makes sense –Most upcalls ask simple questions and should return quickly, anyway –We can change the interface to make this okay Possible solution #3: the lower level enqueues recursive requests for later –This adds a lot of complexity to the lower level

Conclusions Calls naturally flow both up and down layers of abstraction –But upcalls pose some subtle dangers you have to compensate for Modules which need to be protected from each other can still share an address space –But you need a language with strong typing