The Synchronous Data Center Tian Yang Robert Gifford Andreas Haeberlen Linh Thi Xuan Phan Department of Computer and Information Science University of Pennsylvania HotOS XVII (May 14, 2019)

If trains were asynchronous… Station clocks would be at most loosely synchronized Congestion would appear at unpredictable times Station stops could take an arbitrary amount of time Trains would have arbitrary delays and would often be lost entirely HotOS XVII (May 14, 2019)

The Asynchrony assumption System designers typically assume that: Clocks are at most loosely synchronized Network latencies are unpredictable Packets are often dropped in the network We don’t know much about node speeds This is often a good idea! Sometimes we really don’t know Example: System with multiple administrative domains Nicely conservative assumption If the system works in this model, it almost certainly will work on the actual hardware This is the “default”; rarely questioned HotOS XVII (May 14, 2019)

Asynchrony can be expensive But: No time bounds can be given on anything This makes many things very difficult! Example: Congestion control Example: Fault detection Example: Consistency Example: Fighting tail latencies HotOS XVII (May 14, 2019)

It doesn’t have to be that way! The train network is not asynchronous Single administrative domain (like a data center!) Carefully scheduled; speeds and timings are (mostly) known Not all distributed systems are, either! Example: Cyber-physical systems Clocks are closely in sync Network traffic is scheduled; hard latency bounds are known No congestion losses! (And transmission losses are rare) Node speeds and execution times are known exactly CPS are mostly synchronous (out of necessity)! HotOS XVII (May 14, 2019)

So what? Synchrony helps in two ways: How does that help us? Hard latency bounds -> we know how long we need to wait! Absence of a message at a particular time means something How does that help us? No (surprising) congestion anymore Fault detection would be much easier Consistency would be easier to get Long latency tails would disappear Many algorithms become simpler, or even trivial (“boring”) Workloads with timing requirements can be supported Example: Inflate airbag when sensors detect a collision HotOS XVII (May 14, 2019)

Could a data center be synchronous? At first glance, absolutely not! Some objections: Network is shared, so packet delays are unpredictable! Who knows how long anything takes under Linux? Clocks can’t be synchronized closely enough! Our claim: But: Fastpass (SIGCOMM’14) Real-time operating systems Spanner (OSDI’12) Most of the asynchrony in today’s data centers is avoidable! HotOS XVII (May 14, 2019)

Outline Goal: Synchronous data center How could it be done? Network layer Synchronized clocks Building blocks Hardware Software Scheduling NEXT HotOS XVII (May 14, 2019)

The How: Network layer Why is latency so unpredictable? Cross-traffic and queueing! Inspiration: Fastpass (SIGCOMM’14) Machines must ask an ‘arbiter’ for permission before sending Arbiter schedules packets (at >2Tbit/s on eight cores!) Result: (almost) no queueing in the network! No attempt to control end-to-end timing But we see no reason why this couldn’t be added! HotOS XVII (May 14, 2019)

The How: Synchronized clocks Why are clocks so hard to synchronize? Hard to do in the wide area, or via NTP (with cross-traffic) But it can be done: DTP (SIGCOMM’16) achieves nanosecond-precision … with some help from the hardware Google Spanner (OSDI’12) keeps different data centers to within ~4ms … with some help from atomic clocks Having predictable network latencies should help, too! Figure 6(a) from the DTP paper Figure 6 from the Spanner paper HotOS XVII (May 14, 2019)

The How: Building blocks Async Async Tmax Sync Sync Ordering Fault detection HotOS XVII (May 14, 2019)

The How: Software Why is software timing so unpredictable? Reason #1: Hardware features (caches, etc.) Not as bad as it seems: +/- 2% is possible (TDR, OSDI’14) Emerging features, such as Intel’s CAT, should help Meltdown/Spectre will probably accelerate this trend Reason #2: OS structure Linux & friends are not designed for timing stability Idea from CPS: Use elements from RT-OSes … but it will require deep structural changes! No small “synchrony patch” for Linux! HotOS XVII (May 14, 2019)

The How: Fault tolerance What if things (inevitably) break? Could disrupt the careful synchronous “choreography”! Challenge #1: Telling when it breaks Actually easier with synchrony! Challenge #2: Doing something about it How to reconfigure while maintaining timing guarantees? Idea from CPS: Use mode-change protocols! System can operate in different “modes”, based on observed faults Transition from one mode to another via precomputed protocols Result: Timing is maintained during the transition HotOS XVII (May 14, 2019)

The How: Scheduling Can you schedule an entire data center? Surprisingly, we are getting pretty good at it! Sparrow (SOSP’13) can schedule 100ms tasks on 10,000s of cores Idea from CPS: Compositional scheduling Schedule smaller entities (nodes? pods?) in detail Abstract and aggregate, then schedule next-larger entity Repeat until entire system is scheduled Dispatching can be done locally; so can most updates HotOS XVII (May 14, 2019)

Summary Synchronous data centers seem possible! Reasons to be optimistic: Fastpass, DTP, RTOSes, … There are interesting benefits to be had! Asynchrony creates or amplifies challenges like fault detection, congestion control, consistency, tail latencies, load balancing, performance debugging, algorithmic complexity, … These problems could become simpler, or go away entirely! But much work remains to be done! Not much existing work on DC-scale synchronous systems Can we adapt some ideas from cyber-physical systems? Questions? HotOS XVII (May 14, 2019)