Ensemble and Beyond Presentation to David Tennenhouse, DARPA ITO Ken Birman Dept. of Computer Science Cornell University

Quick Timeline Cornell has developed 3 generations of reliable group communication technology –Isis Toolkit: –Horus System: –Ensemble System: Today starting a major shift in emphasis –Spinglass Project: 1999-

Questions to consider Have these projects been successful? What is the future of Ensemble if we move to a new and different focus? Nature of the new opportunity we now perceive

Timeline Isis Horus Ensemble Introduced reliability into group computing Virtual synchrony execution model Fairly elaborate, monolithic, but adequate speed Many transition successes New York, Swiss Stock Exchanges French Air Traffic Control console system Southwestern Bell Telephone network mgt. Hiper-D (next generation AEGIS)

Virtual Synchrony Model crash G 0 ={p,q} G 1 ={p,q,r,s} G 2 ={q,r,s} G 3 ={q,r,s,t} pqrstpqrst r, s request to join r,s added; state xfer t added, state xfer t requests to join p fails

Why a “model”? Models can be reduced to theory – we can prove the properties of the model, and can decide if a protocol achieves it Enables rigorous application-level reasoning Otherwise, the application must guess at possible misbehaviors and somehow overcome them

French ATC system (simplified) Controllers Air Traffic Database (flight plans, etc) X.500 Directory Radar Onboard

A center contains... Perhaps 50 “teams” of 3-5 controllers each Each team supported by workstation cluster Cluster-style database server has flight plan information Radar server distributes real-time updates Connections to other control centers (40 or so in all of Europe, for example)

Process groups arise here: Cluster of servers running critical database server programs Cluster of controller workstations support ATC by teams of controllers Radar must send updates to the relevant group of control consoles Flight plan updates must be distributed to the “downstream” control centers

Use of our model? French government knows requirements for safety in ATC application With our model, we can reduce their need to a formal set of statements This lets us establish that our solution will really be safe in their setting Contrast with usual ad-hoc methodologies...

Timeline Isis Horus Ensemble Simpler, faster group communication system Uses a modular layered architecture. Layers are “compiled,” headers compressed for speed Supports dynamic adaptation and real-time apps Partitionable version of virtual synchrony Transitioned primarily through Stratus Computer Phoenix system Basis of Stratus f.tol. Proposal to OMG

Layered Microprotocols in Horus Interface to Horus is extremely flexible Horus manages group abstraction group semantics (membership, actions, events) defined by stack of modules encrypt vsync filter sign ftol Ensemble stacks plug-and-play modules to give design flexibility to developer

Processes Communicate Through Identical Multicast Protocol Stacks encrypt vsync ftol encrypt vsync ftol encrypt vsync ftol

Superimposed Groups in Application With Multiple Subsystems encrypt vsync ftol encrypt vsync ftol encrypt vsync ftol encrypt vsync ftol encrypt vsync ftol encrypt vsync ftol Magenta group for video communication Orange for control and coordination

Timeline Isis Horus Ensemble Horus-like stacking architecture, equally fast Includes an innovative group-key mechanism for secure group multicast and key management Uses high level language and can be formally proved correct, an unexpected and major success Many early transition successes SC-21, Quorum via collaboration with BBN Nortel, STC: potential commercial users Discussions with MS (COM+), Sun (RMI.next): could be basis of standards.

Proving Ensemble Correct Unlike Isis and Horus, Ensemble is coded in a language with strong semantics (ML) So we took a spec. of virtual synchrony from MIT’s IOA group (Nancy Lynch) And are actually able to prove that our code implements the spec. and that the spec captures the virtual synchrony property!

What Next? Continue some work with Ensemble –Keep it alive, support and extend it –Play an active role in transition –Assist standards efforts But shift in focus to a completely new effort –Emphasize adaptive behavior, extreme scalability, robustness against local disruption –Fits “Intrinisically Survivable Systems” initiative

Throughput Stability: Achilles Heel of Group Multicast When scaled to even modest environments, overheads of virtual synchrony become a problem –One serious challenge involves management of group membership information –But multicast throughput also becomes unstable with high data rates, large system size, too. Stability of protocols like SRM unknown

Stock Exchange Problem: Vsync. multicast is too “fragile” Most members are healthy…. … but one is slow

Figure 1: Multicast throughput in an 8-member group perturbed by transient failures Ideal Actual

Bimodal Multicast in Spinglass A new family of protocols with stable throughput, extremely scalable, fixed and low overhead per process and per message Gives tunable probabilistic guarantees Includes a membership protocol and a multicast protocol Requires some very weak QoS assumptions

Start by using unreliable multicast to rapidly distribute the message. But some messages may not get through, and some processes may be faulty. So initial state involves partial distribution of multicast(s)

Periodically (e.g. every 100ms) each process sends a digest describing its state to some randomly selected group member. The digest identifies messages. It doesn’t include them.

Recipient checks the gossip digest against its own history and solicits a copy of any missing message from the process that sent the gossip

Processes respond to solicitations received during a round of gossip by retransmitting the requested message. The round lasts much longer than a typical RPC time.

Figure 5: Graphs of analytical results

Spinglass: Summary of objectives Radically different approach yields stable, scalable protocols with steady throughput Small footprint, tunable to match conditions Completely asynchronous, hence demands new style of application development But opens the door to a new lightweight reliability technology supporting large autonomous environments that adapt