Hasan Bulut hbulut@cs.indiana.edu Scaling and Fault Tolerance for Distributed Messages in a Service and Streaming Architecture Hasan Bulut hbulut@cs.indiana.edu

Motivation Videoconferencing systems; AccessGrid, VRVS, H.323 based systems (i.e. Polycom) Streaming media; client-server architecture. Client has control over the stream. Annotation systems; use streams in local files or obtained from capture device directly We would like to build an integrated system based on messaging middleware where streams from each world can be accessed from another one. This will also extend and bring new application areas. 11/7/2018

Research Issues I Session Management Developing a XML-based control framework where XML messages are also used for information exchange. How much is this framework flexible to achieve an integrated collaboration system. Extending the framework to cellular clients Metadata Description and Management Describing collaboration session metadata in XML format Utilizing WS-Context which is a fault tolerant metadata repository and can be shared among all services and clients in the system. Investigating the impact of this on the design and architecture of the streaming service Session Management Developing a XML-based control framework where XML messages are also used for information exchange. How much is this framework flexible to achieve an integrated collaboration system. Extending the framework to cellular clients Metadata Description and Management Describing collaboration session metadata in XML format Utilizing WS-Context which is a fault tolerant metadata repository and can be shared among all services and clients in the system. Investigating the impact of this on the design and architecture of the streaming service 11/7/2018

Research Issues II Multiple Streams Issues Application Scenarios Data format independent generic streaming framework Services required within messaging middleware to archive and replay of streams achieve instant replay of streams achieve synchronization among streams generated at geographically large area Increase fault tolerance Jitter introduced by archiving and replay service How much delay for LAN and WAN clients Application Scenarios Proposing and developing an annotation system where it can receive streams from real-time live videoconferencing sessions with instant replay capability Increase fault tolerance so that late joining or broken clients can receive the missed parts of the stream 11/7/2018

GlobalMMCS Prototype System 11/7/2018

XML Based General Session Protocol (XGSP) XGSP is a conference control framework. The goal of XGSP is to integrate heterogeneous systems into one collaboration system. It can be viewed as a common A/V signaling protocol designed to support interactions between different A/V collaboration endpoints. Enables different A/V endpoints to collaborate in the same collaboration session. Includes three components; user session management, application session management and floor control. 11/7/2018

Global Multimedia Collaboration System (GlobalMMCS) A prototype system to verify and refine XGSP conference control framework. The current prototype of GlobalMMCS includes: A XGSP media server H.323, SIP gateways and Real Servers for A/V clients XGSP A/V Session Server The web server A prototype system to verify and refine XGSP conference control framework. The current prototype of GlobalMMCS includes: A XGSP media server provides services for bridging multicast and unicast, video-switching, video-mixing and audio-mixing to H.323, SIP as well as AG endpoints. H.323, SIP gateways and Real Servers for A/V clients XGSP A/V Session Server manages real-time A/V sessions, receives messages from gateways and the web server, and performs appropriate actions on the media server. The web server provides an easy-to-use web interface for users to join multimedia sessions and for administrators to perform administrative tasks. 11/7/2018

Extending GlobalMMCS to Mobile Clients 11/7/2018

Services Built Within Messaging Middleware Time Services Jitter Reduction Service Replication Scheme (Repository Redundancy) 11/7/2018

Time Services NB-NTP Time Service High Resolution Timing Service An implementation of Network Time Protocol (RFC 1305) NTP is used to synchronize timekeeping among a set of distributed time servers and clients. Entities generating events in the system should utilize Time Service to timestamp the events. High Resolution Timing Service Implemented for Windows, Linux and Solaris Gives 2-3 usec resolution It defines the architectures, algorithms, entities and protocols used by NTP. Provides sleep(long duration) API to enable threads sleep under 10 msec. 11/7/2018

Test Result The first offset value is -139895 ms, which shows how much the clock in that machine is ahead of the real time. The change of offsets is between (-3) - (2) ms. 11/7/2018

Jitter Reduction Services Buffering Service Time-order events. Time Differential Service Releases events preserving the time spacing between events. Can achieve msec resolution NB Buffering Service Implemented as NB Buffering Service Can be used both by NB nodes or NB clients The goal is to time-order events. It releases events based on 3 parameters. If any of these criteria is satisfied it releases the events. Buffer size in KB Number of events in the buffer The duration of release Above parameters are configurable Delay introduced by the buffer service can vary based on the above parameter values. High Resolution Clock Implemented for Linux, Solaris and Windows platforms. Gives 2-3 usec resolution Provides sleep(long duration) API to enable threads sleep under 10 msec. Calls to sleep function with an input < 10 msec does not work in Java. 11/7/2018

Jitter Reduction Service Test Results 11/7/2018

Repository Redundancy Extended NB Reliable Delivery Scheme to ensure that reliable delivery guarantees are satisfied in the presence of repository failures. Each repository functions autonomously and makes decisions independently. A repository can recover from any other repository as long as the missing event exists in that repository. steering repository: A publisher or subscriber to a reliable-topic can interact with exactly one repository. The repository operates in the active mode for steered clients and in the passive mode for clients that it does not steer. With this scheme, If there are N available repositories, reliable delivery guarantees will be met even if N-1 repositories fail. Each repository keeps a table for any other repository to map the sequence numbers with that repository. steering repository: A publisher or subscriber to a reliable-topic can interact with exactly one repository. The repository operates in the active mode for steered clients and in the passive mode for clients that it does not steer. If steering repository fails, client can choose another one as a steering repository. Archival notification: sequence number of the stored event at a repository is received by other repositories. Acknowledgements and syncs: from a client is received by every repository. 11/7/2018

Repository Redundancy Test Results P1 Publisher S1 Subscriber (S1: Measuring client) Topology C Topology D Topology F Topology E 11/7/2018

Repository Redundancy Test Results 11/7/2018

Generic Streaming Framework and Metadata Management Generic Archiving and Replay Session Recorders Session Players GlobalMMCS Recording and Replay 11/7/2018

Metadata Management We use WS-Context service as a metadata repository. WS-Context provides us distributed and fault tolerant metadata repository which can be shared among every entity in the system. Updates in metadata are published to interested entities by WS-Context service. Two levels of management: session level and intra-session level Sessions level management is to keep track of sessions Intra-session level management is to keep track of streams in the session 11/7/2018

Metadata Management (Session level) 11/7/2018

GlobalMMCS Session Management 11/7/2018

Archive and Replay Session Management 11/7/2018

Generic Archiving and Replay Framework A generic framework for recording and replay of any type of streaming event or data. Instant replay of streams: Real-time (live) streams can be replayed, paused and rewound while streams are being recorded. Stream linkage: Multiple streams are linked together to construct a session. A collaboration session can be recorded and replayed within this framework. Examples; Anabas GlobalMMCS eSports System Generic delivery mechanism independent of media type and format. Not only for multimedia content but also for other content types as well, such as text and image. Based on messaging middleware to achieve generic framework. Anabas – Uses JMS events to transport data such as whiteboard, shared display, audio, etc. GlobalMMCS – Uses NaradaBrokering RTP Events to transport audio and video data. eSports System – Uses RTP Events and NB Events for recording and replay 11/7/2018

Uniform Event Type For Generic Framework Received events are wrapped inside NaradaBrokering native events (NBEvent) with some event specific information. Received event is placed to the payload of the NBEvent. NBEvent also contains timestamp information and event type. Received event is placed to the payload of the NBEvent as is to preserve original data and related information. NBEvent also contains timestamp information to timespace original events during replay and event type to initiate appropriate player for that event type. 11/7/2018

Session Recorders Control message 11/7/2018 Types of topic recorders changes based on the transport type rather than data format. Based on the event type, session recorder initiates appropriate topic recorder to handle the topic. Accumulate events for 1 sec or until 20KB before constructing NBEvent. This is to reduce number of database access. Control message 11/7/2018

Session Players The primary purpose of session player is to simulate clients in the original session. Supports instant replay of real-time live streams that are being recorded. Session players support replay, pause, rewind and fast forward operations. When one of those operations is requested, it is applied to all of the topics (streams) in that session. To achieve this; Each recorded topic (or stream) is mapped to a new topic and events of the same original topic are released to the mapped topic. Utilizes Time Differential Service to timespace events 11/7/2018

Session Players Control message 11/7/2018 Events in a session are synchronized among each other regardless of which topic they are from. Timestamp service is used to let client know what the timestamp of the last entry in the repository for that session. It is also used to let client know what the current timestamp played is (especially for RTP Event topics.) Control message 11/7/2018

GlobalMMCS Session Recording and Replay 11/7/2018

eSports System – Capabilities provided to eSports System Archive and replay of NaradaBrokering native events Archive and replay of GlobalMMCS sessions Instant replay Utilizing WS-Context Service Transporting messages through NaradaBrokering messaging middleware Archive and replay of NaradaBrokering native events NB events can be used to transport audio, video and images as well as text messages Archive and replay of GlobalMMCS sessions Streams from H.323 clients, AccessGrid sessions, capture devices. Instant replay Play, pause, rewind and forward capability over real-time live streams Utilizing WS-Context Service a distributed and fault tolerant metadata repository which can be accessed through WSDL interfaces Transporting messages through NaradaBrokering messaging middleware capable of software multicasting, one replay session can be received by many clients subscribed to the same replay session topics 11/7/2018

eSports System and Streaming Services 11/7/2018

eSports System Interface (Recording) 11/7/2018

eSports System Interface (Replay) 11/7/2018

eSports System – Taking snapshots from Video Players 11/7/2018

Performance Results (Test Setup) LAN Setup : gf4.ucs.indiana.edu WAN Setup (FSU): vlab2.scs.fsu.edu WAN Setup (USCD): synseis.geongrid.org 11/7/2018

Performance Tests (LAN Results) Transport delay: 3513.7 - 3510.8 = 2.9 msec 11/7/2018

Performance Tests (WAN – FSU Results) Transport delay: 3502.4 - 3484.8 = 17.6 msec 11/7/2018

Performance Tests (WAN – UCSD Results) Transport delay: 3462.2 - 3426.8 = 35.4 msec 11/7/2018

Contribution Proposed and implemented a scalable and fault tolerant services-based architecture that integrates videoconferencing systems, streaming media and annotation systems. XML based control and messaging framework for collaboration systems. Architecture allows cellular clients to receive real-time video conferencing streams. Using a shared fault tolerant metadata repository (WS-Context Service) to keep session and stream information. A data format independent generic streaming framework for archive and replay of streams built on top messaging middleware systems with the help of service oriented architecture and Web Services technologies 11/7/2018

Contribution Instant replay of real-time live streams Allows annotation systems to annotate videoconferencing streams Following services are introduced to messaging middleware to increase the quality-of-service of streams and fault tolerant of the collaboration system; Time Service Jitter Reduction Service Replication Scheme RTSP semantics introduced to messaging systems. 11/7/2018