Transis 1 Fault Tolerant Video-On-Demand Services Tal Anker, Danny Dolev, Idit Keidar, The Transis Project

Transis 2 VoD Service VoD: Full VCR control 1 video stream per client Client C 1 VoD Service provider Requests Video Stream Movies disk(s)

Transis 3 High Availability Multiple servers –at different sites Fault tolerance: –servers can crash Managing the load: –new servers can be brought up / down –load should be re-distributed “on the fly”  migration of clients

Transis 4 The challenges Low overhead Transparency –How do clients know whom to connect to?  “abstract” service –Clients should be unaware of migration server VoD Service Client C 1 Client C 2 server Client C 1 Client C 2 server VoD Service server Failed Server

Transis 5 Buffer Management and Flow Control Overcome jitter, message re-ordering and migration periods Re-fill buffers quickly after migration –avoid buffer overflow Minimize buffers –minimize pre-fetch bandwidth Dynamically adjust transmission rate to client capabilities –Re-negotiation of QoS

Transis 6 Features of our solution Use group communication in the control plane –connection establishment –fault tolerance and migration Flow control explicitly handles migration Low overhead –~1/1000 of the bandwidth –Negligible memory and CPU overhead Commodity hardware and publicly available network technologies

Transis 7 Environment Implementation –UDP/IP over 10 Mbit/s switched ethernet –Transis –Sun Sparc and BSDI PC’s as video servers –Win NT machines as video clients –MPEG1 & 2 hardware decoders Machine and Network Failures

Transis 8 Implementing the abstract service Use group communication –clients communicate with a well known group name (logical entity) –unaware of the number and identity of the servers in the group Servers periodically share information about clients (every 1/2sec) If a server crashes (or is overloaded), another server transparently takes over

Transis 9 Group Communication Reliable Group Multicast (Group Abstraction) Message Ordering Dynamic Reconfiguration Membership with Strong Semantics (Virtual Synchrony) Systems: Transis, Horus, Ensemble, Totem, Newtop, RMP, ISIS, Psync, Relacs

Transis 1010 The group layout of the VoD service

Transis 11 Transis Allows Simple Design Group abstraction for connection establishment and transparent migration Reliable group multicast allows servers to consistently share information Membership services detects conditions for migration Reliable messages for control –Server takes ~2500 C++ code lines –Client takes ~4000 C code lines (excluding GUI and display)

Transis 1212 VoD Client: Multi Threaded Design Client buffer accounts for jitter period when migration occurs Obviously, Flow-Control and Buffer management were needed

Transis 1313 Flow Control Feedback based flow-control (sparse): –FC messages are sent to the logical server (session group) –Clients determines the changes in the flow:

Transis 1414 Emergency Flow Control When the server receives an emergency message: –The server change the fps rate: fps = latest-known-fps + emergency quantity The emergency quantity decays every second (by a factor) –While the quantity is above zero, the server ignores FC messages from the client

Transis 1515 Performance Measurements On HUJI Network (LAN) Servers at TAU and clients at HUJI (WAN) The measurements show the system is robust and support our transparency claims

Transis 1616 Software Buffers

Transis 1717 Hardware Buffers

Transis 1818 Skipped Frames on LAN

Transis 1919 Skipped Frames on WAN

Transis 2020 Summary Scalable VoD service Load balancing Tolerating machine and network failure All the above are achieved practically for free: –~1/1000 of the total bandwidth –Negligible memory and CPU overhead

Transis 2121 Thanks to... Gregory Chockler The other members of the Transis project