CS514: Intermediate Course in Operating Systems Professor Ken Birman Ben Atkin: TA Lecture 24: Nov. 16

Solving real problems with real-time protocols When does real-time matter? –Air traffic application: we want rapid response when events occur –Telecommunications application: switch requires real-time reactions to events that occur Two categories of real-time –We want an action to be predictably fast –We want an action to occur before a deadline passes

Predictability If this is our goal… –Any well-behaved mechanism may be adequate –But we should be careful about uncommon disruptive cases For example, cost of failure handling is often overlooked Risk is that an infrequent scenario will be very costly when it occurs

Predictability: Examples Probabilistic multicast protocol –Very predictable if our desired latencies are larger than the expected convergence –Much less so if we seek latencies that bring us close to the expected latency of the protocol itself Rule of thumb? –Real-time doesn’t mean “as fast as possible” – more often it means “slow and steady” !

Mixing issues Telephone networks need a mixture of properties –Real-time response –High performance –Stable behavior even when failures and recoveries occur Can we use our tools to solve such a problem?

Friedman’s SS7 experiment Used Horus to emulate a telephone switching system Idea is to control a telephone switch that handles 800 telephone numbers in software Horus runs the “800 number database” on a cluster of processors next to the switch

IN coprocessor example SS7 switch SS7 switch SS7 switch SS7 switch

IN coprocessor example SS7 switch SS7 switch SS7 switch SS7 switch coprocessor

Role of coprocessor A simple database Basically –Switch does a query How should I route a call to from ? Reply: use output line 6 –Time limit of 100ms on transaction Also runs a background protocol to update the database as things change, on a separate network…

Goals for coprocessor Right now, people use hardware fault-tolerant machines for this –E.g. Stratus “pair and a spare” –Mimics one computer but tolerates hardware failures –Performance an issue…

Goals for coprocessor What we want –Scalability: ability to use a cluster of machines for the same task, with better performance when we use more nodes –Fault-tolerance: a crash or recovery shouldn’t disrupt the system –Real-time response: must satisfy the 100ms limit at all times Desired: “7 to 9-nines availability” Downtime: any period when a series of requests might all be rejected

IN coprocessor example SS7 switch Query Element (QE) processors do the number lookup (in- memory database). Goals: scalable memory without loss of processing performance as number of nodes is increased Switch itself asks for help when remote number call is sensed External adaptor (EA) processors run the query protocol EA Primary backup scheme adapted (using small Horus process groups) to provide fault-tolerance with real-time guarantees

Options? A simple scheme: –Organize nodes as groups of 2 processes –Use virtual synchrony multicast For query For response Also for updates and membership tracking –A bit like our ATC example…

IN coprocessor example SS7 switch EA Step 1: Switch sees incoming request

IN coprocessor example SS7 switch EA Step 2: Switch waits while EA procs. multicast request to group of query elements (“partitioned” database)

IN coprocessor example SS7 switch Think EA Step 3: The query elements do the query in duplicate

IN coprocessor example SS7 switch EA Step 4: They reply to the group of EA processes

IN coprocessor example SS7 switch EA Step 6: EA processes reply to switch, which routes call

Results? Terrible performance! –Solution has 2 Horus multicasts on each critical path –Experience: about 600 queries per second but no more Also: slow to handle failures –Freezes for as long as 6 seconds Performance doesn’t improve much with scale either

Next try? Consider taking Horus off the critical path Idea is to continue using Horus –It manages groups –And we use it for updates to the database and for partitioning the QE set But no multicasts on critical path –Instead use a hand-coded scheme

Roy’s hand-coded scheme Queue up a set of requests from an EA to a QE Periodically, sweep the set into a message and send as a batch Process, also as a batch Send the batch of replies back to EA

Clever twists? Split into a primary and secondary EA for each request –Secondary steps in if no reply seen in 50ms –Batch size calculated so that 50ms should be “long enough” Hand optimized I/O and batching code…

Results? Able to sustain 22,000 emulated telephone calls per second Able to guarantee response within 100ms and no more than 3% of calls are dropped (randomly) Performance is not hurt by a single failure or recovery while switch is running Can put database in memory: memory size increases with number of nodes in cluster

Keys to success Horus is doing the hard work of configuration management –But configuration is only “read” by code on critical path –Horus is not really in the performance- critical section of code Also: need enough buffering space to keep running while a failure is sensed and reported

Coprocessors galore SS7 thinks of the scalable cluster as a coprocessor But coprocessor thinks of Horus as a sort of coprocessor –It sits off to one side –Reports membership changes –But “interface” is really just a shared memory segment

Same problem with Totem or CASD? Can’t use these technologies with 100ms timeout! The basic delivery latency already exceeds 100ms Could probably tune either protocol to this setup... but Friedman can probably double his performance too, by tuning Horus to the setup Conclusion is that real-time should be understood from needs of application, not a specific theory

Other settings with a strong temporal element Load balancing –Idea is to track load of a set of machines –Can do this at an access point or in the client –Then want to rebalance by issuing requests preferentially to less loaded servers

Load-balancing with an external adaptor EA

Load-balancing on client Load summary Picks a lightly- loaded machine

Load balancing in farms Akamai widely cited –They download the rarely-changing content from customer web sites –Distribute this to their own web farm –Then use a hacked DNS to redirect web accesses to a close-by, less-loaded machine Real-time aspects? –The data on which this is based needs to be fresh or we’ll send to the wrong server

Real-time in industry Very common in factory settings –At time t start the assembly line –Planning: from time t 0 to t 1 produce MIPS CPU chips on fab-unit 16… –If the pressure rises too quickly, reduce the temperature Often, we use real-time operating systems in support of such applications

Robotics, embedded systems Many emerging applications involve coordination of action by many components E.g. robots that cooperate to construct something Demand for real-time embedded systems technology will be widespread in industry Little is understood about networks in such settings… a big opportunity

Future directions in real- time Expect GPS time sources to be common within five years Real-time tools like periodic process groups will also be readily available (members take actions in a temporally coordinated way) Increasing focus on predictable high performance rather than provable worst-case performance Increasing use of probabilistic techniques

Future Directions David Tennenhouse (MIT, then DARPA ITO, then MCI): –Get real –Get small –Get moving!

Conclusions? Protocols like pbcast are potentially appealing in a subset of applications that are naturally probabilistic to begin with, and where we may have knowledge of expected load levels, etc. More traditional virtual synchrony protocols with strong consistency properties make more sense in standard networking settings Many ways to combine temporal+logical props.

Ending on a thought question Distributed systems depend on many layers of software, hardware, and many assumptions New wave of embedded systems will demand real-time solutions! Are such systems ultimately probabilistic, or ultimately deterministic? Do current reliable systems converge towards deterministic behavior or converge towards chaotic behaviors?