1. Hasan Bulut hbulut@cs.indiana.edu
Scaling and Fault Tolerance for Distributed Messages in a Service and Streaming Architecture
Hasan Bulut

2. Motivation
Videoconferencing systems; AccessGrid, VRVS, H.323 based systems (i.e. Polycom)
Streaming media; client-server architecture. Client has control over the stream.
Annotation systems; use streams in local files or obtained from capture device directly
We would like to build an integrated system based on messaging middleware where streams from each world can be accessed from another one. This will also extend and bring new application areas.
11/7/2018

3. Research Issues I Session Management
Developing a XML-based control framework where XML messages are also used for information exchange.
How much is this framework flexible to achieve an integrated collaboration system.
Extending the framework to cellular clients
Metadata Description and Management
Describing collaboration session metadata in XML format
Utilizing WS-Context which is a fault tolerant metadata repository and can be shared among all services and clients in the system.
Investigating the impact of this on the design and architecture of the streaming service
Session Management
Developing a XML-based control framework where XML messages are also used for information exchange.
How much is this framework flexible to achieve an integrated collaboration system.
Extending the framework to cellular clients
Metadata Description and Management
Describing collaboration session metadata in XML format
Utilizing WS-Context which is a fault tolerant metadata repository and can be shared among all services and clients in the system.
Investigating the impact of this on the design and architecture of the streaming service
11/7/2018

4. Research Issues II Multiple Streams Issues Application Scenarios
Data format independent generic streaming framework
Services required within messaging middleware to
archive and replay of streams
achieve instant replay of streams
achieve synchronization among streams generated at geographically large area
Increase fault tolerance
Jitter introduced by archiving and replay service
How much delay for LAN and WAN clients
Application Scenarios
Proposing and developing an annotation system where it can receive streams from real-time live videoconferencing sessions with instant replay capability
Increase fault tolerance so that late joining or broken clients can receive the missed parts of the stream
11/7/2018

5. GlobalMMCS Prototype System
11/7/2018

6. XML Based General Session Protocol (XGSP)
XGSP is a conference control framework.
The goal of XGSP is to integrate heterogeneous systems into one collaboration system.
It can be viewed as a common A/V signaling protocol designed to support interactions between different A/V collaboration endpoints.
Enables different A/V endpoints to collaborate in the same collaboration session.
Includes three components; user session management, application session management and floor control.
11/7/2018

7. Global Multimedia Collaboration System (GlobalMMCS)
A prototype system to verify and refine XGSP conference control framework.
The current prototype of GlobalMMCS includes:
A XGSP media server
H.323, SIP gateways and Real Servers for A/V clients
XGSP A/V Session Server
The web server
A prototype system to verify and refine XGSP conference control framework.
The current prototype of GlobalMMCS includes:
A XGSP media server
provides services for bridging multicast and unicast, video-switching, video-mixing and audio-mixing to H.323, SIP as well as AG endpoints.
H.323, SIP gateways and Real Servers for A/V clients
XGSP A/V Session Server
manages real-time A/V sessions, receives messages from gateways and the web server, and performs appropriate actions on the media server.
The web server
provides an easy-to-use web interface for users to join multimedia sessions and for administrators to perform administrative tasks.
11/7/2018

8. Extending GlobalMMCS to Mobile Clients
11/7/2018

9. Services Built Within Messaging Middleware
Time Services
Jitter Reduction Service
Replication Scheme (Repository Redundancy)
11/7/2018

10. Time Services NB-NTP Time Service High Resolution Timing Service
An implementation of Network Time Protocol (RFC 1305)
NTP is used to synchronize timekeeping among a set of distributed time servers and clients.
Entities generating events in the system should utilize Time Service to timestamp the events.
High Resolution Timing Service
Implemented for Windows, Linux and Solaris
Gives 2-3 usec resolution
It defines the architectures, algorithms, entities and protocols used by NTP.
Provides sleep(long duration) API to enable threads sleep under 10 msec.
11/7/2018

11. Test Result
The first offset value is ms, which shows how much the clock in that machine is ahead of the real time.
The change of offsets is between (-3) - (2) ms.
11/7/2018

12. Jitter Reduction Services
Buffering Service
Time-order events.
Time Differential Service
Releases events preserving the time spacing between events.
Can achieve msec resolution
NB Buffering Service
Implemented as NB Buffering Service
Can be used both by NB nodes or NB clients
The goal is to time-order events.
It releases events based on 3 parameters. If any of these criteria is satisfied it releases the events.
Buffer size in KB
Number of events in the buffer
The duration of release
Above parameters are configurable
Delay introduced by the buffer service can vary based on the above parameter values.
High Resolution Clock
Implemented for Linux, Solaris and Windows platforms.
Gives 2-3 usec resolution
Provides sleep(long duration) API to enable threads sleep under 10 msec.
Calls to sleep function with an input < 10 msec does not work in Java.
11/7/2018

13. Jitter Reduction Service Test Results
11/7/2018

14. Repository Redundancy
Extended NB Reliable Delivery Scheme to ensure that reliable delivery guarantees are satisfied in the presence of repository failures.
Each repository functions autonomously and makes decisions independently.
A repository can recover from any other repository as long as the missing event exists in that repository.
steering repository: A publisher or subscriber to a reliable-topic can interact with exactly one repository.
The repository operates in the active mode for steered clients and in the passive mode for clients that it does not steer.
With this scheme, If there are N available repositories, reliable delivery guarantees will be met even if N-1 repositories fail.
Each repository keeps a table for any other repository to map the sequence numbers with that repository.
steering repository: A publisher or subscriber to a reliable-topic can interact with exactly one repository.
The repository operates in the active mode for steered clients and in the passive mode for clients that it does not steer.
If steering repository fails, client can choose another one as a steering repository.
Archival notification: sequence number of the stored event at a repository is received by other repositories.
Acknowledgements and syncs: from a client is received by every repository.
11/7/2018

15. Repository Redundancy Test Results
P1 Publisher
S1 Subscriber
(S1: Measuring client)
Topology C
Topology D
Topology F
Topology E
11/7/2018

16. Repository Redundancy Test Results
11/7/2018

17. Generic Streaming Framework and Metadata Management
Generic Archiving and Replay
Session Recorders
Session Players
GlobalMMCS Recording and Replay
11/7/2018

18. Metadata Management
We use WS-Context service as a metadata repository.
WS-Context provides us distributed and fault tolerant metadata repository which can be shared among every entity in the system.
Updates in metadata are published to interested entities by WS-Context service.
Two levels of management: session level and intra-session level
Sessions level management is to keep track of sessions
Intra-session level management is to keep track of streams in the session
11/7/2018

19. Metadata Management (Session level)
11/7/2018

20. GlobalMMCS Session Management
11/7/2018

21. Archive and Replay Session Management
11/7/2018

22. Generic Archiving and Replay Framework
A generic framework for recording and replay of any type of streaming event or data.
Instant replay of streams: Real-time (live) streams can be replayed, paused and rewound while streams are being recorded.
Stream linkage: Multiple streams are linked together to construct a session.
A collaboration session can be recorded and replayed within this framework. Examples;
Anabas
GlobalMMCS
eSports System
Generic delivery mechanism independent of media type and format.
Not only for multimedia content but also for other content types as well, such as text and image.
Based on messaging middleware to achieve generic framework.
Anabas – Uses JMS events to transport data such as whiteboard, shared display, audio, etc.
GlobalMMCS – Uses NaradaBrokering RTP Events to transport audio and video data.
eSports System – Uses RTP Events and NB Events for recording and replay
11/7/2018

23. Uniform Event Type For Generic Framework
Received events are wrapped inside NaradaBrokering native events (NBEvent) with some event specific information.
Received event is placed to the payload of the NBEvent.
NBEvent also contains timestamp information and event type.
Received event is placed to the payload of the NBEvent as is to preserve original data and related information.
NBEvent also contains timestamp information to timespace original events during replay and event type to initiate appropriate player for that event type.
11/7/2018

24. Session Recorders Control message 11/7/2018
Types of topic recorders changes based on the transport type rather than data format.
Based on the event type, session recorder initiates appropriate topic recorder to handle the topic.
Accumulate events for 1 sec or until 20KB before constructing NBEvent. This is to reduce number of database access.
Control message
11/7/2018

25. Session Players
The primary purpose of session player is to simulate clients in the original session.
Supports instant replay of real-time live streams that are being recorded.
Session players support replay, pause, rewind and fast forward operations. When one of those operations is requested, it is applied to all of the topics (streams) in that session.
To achieve this;
Each recorded topic (or stream) is mapped to a new topic and events of the same original topic are released to the mapped topic.
Utilizes Time Differential Service to timespace events
11/7/2018

26. Session Players Control message 11/7/2018
Events in a session are synchronized among each other regardless of which topic they are from.
Timestamp service is used to let client know what the timestamp of the last entry in the repository for that session.
It is also used to let client know what the current timestamp played is (especially for RTP Event topics.)
Control message
11/7/2018

27. GlobalMMCS Session Recording and Replay
11/7/2018

28. eSports System – Capabilities provided to eSports System
Archive and replay of NaradaBrokering native events
Archive and replay of GlobalMMCS sessions
Instant replay
Utilizing WS-Context Service
Transporting messages through NaradaBrokering messaging middleware
Archive and replay of NaradaBrokering native events
NB events can be used to transport audio, video and images as well as text messages
Archive and replay of GlobalMMCS sessions
Streams from H.323 clients, AccessGrid sessions, capture devices.
Instant replay
Play, pause, rewind and forward capability over real-time live streams
Utilizing WS-Context Service
a distributed and fault tolerant metadata repository which can be accessed through WSDL interfaces
Transporting messages through NaradaBrokering messaging middleware
capable of software multicasting, one replay session can be received by many clients subscribed to the same replay session topics
11/7/2018

29. eSports System and Streaming Services
11/7/2018

30. eSports System Interface (Recording)
11/7/2018

31. eSports System Interface (Replay)
11/7/2018

32. eSports System – Taking snapshots from Video Players
11/7/2018

33. Performance Results (Test Setup)
LAN Setup : gf4.ucs.indiana.edu
WAN Setup (FSU): vlab2.scs.fsu.edu
WAN Setup (USCD): synseis.geongrid.org
11/7/2018

34. Performance Tests (LAN Results)
Transport delay: = 2.9 msec
11/7/2018

35. Performance Tests (WAN – FSU Results)
Transport delay: = 17.6 msec
11/7/2018

36. Performance Tests (WAN – UCSD Results)
Transport delay: = 35.4 msec
11/7/2018

37. Contribution
Proposed and implemented a scalable and fault tolerant services-based architecture that integrates videoconferencing systems, streaming media and annotation systems.
XML based control and messaging framework for collaboration systems.
Architecture allows cellular clients to receive real-time video conferencing streams.
Using a shared fault tolerant metadata repository (WS-Context Service) to keep session and stream information.
A data format independent generic streaming framework for archive and replay of streams built on top messaging middleware systems with the help of service oriented architecture and Web Services technologies
11/7/2018

38. Contribution Instant replay of real-time live streams
Allows annotation systems to annotate videoconferencing streams
Following services are introduced to messaging middleware to increase the quality-of-service of streams and fault tolerant of the collaboration system;
Time Service
Jitter Reduction Service
Replication Scheme
RTSP semantics introduced to messaging systems.
11/7/2018