Design and Implementation of Audio/Video Collaboration System Based on Publish/subscribe Event Middleware CTS04 San Diego 19 January 2004 PTLIU Laboratory for Community Grids Geoffrey Fox, Wenjun Wu, Ahmet Uyar, Hasan Bulut Computer Science Department Indiana University, Bloomington IN 47404 http://www.globalmmcs.org gcf@indiana.edu 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

A/V Collaboration Systems H323 H.323 is defined as an umbrella standard specifying the components to be used within an H.323-based environment. SIP The Session Initiation Protocol (SIP) defines how to establish, maintain and terminate Internet sessions including multimedia conferences Access Grid enhanced Mbone A/V tools ( VIC, RAT ) Internet 2 network ( Multicast support ) 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

XGSP Collaboration Framework To integrate heterogeneous systems into one collaboration system A unified, scalable, robust “overlay” network is needed to support A/V and data group communication over heterogeneous networking environments. A common A/V signaling protocol has to be designed to support interactions between different A/V collaboration endpoints. Different A/V endpoints should collaborate in the same collaboration session. 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

Group Communication in Collaboration Collaboration applications usually need a group communication service for both multipoint data and control information delivery Centralized conferencing systems usually depend upon a single conference server Distributed conferencing systems use IP multicast Access Grid uses Internet2 multicast for audio/video transmission. Problems: Centralized conferencing systems don’t have good scalability IP multicast has not become ubiquitously available 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

uri="http://www.globalmmcs.org" email="gcf@indiana.edu" Narada Broker Network (P2P) Community For message/events service Broker Broker (P2P) Community Resource Broker Broker Broker Data base (P2P) Community Software multicast Broker (P2P) Community 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

uri="http://www.globalmmcs.org" email="gcf@indiana.edu" Narada Brokering Narada Brokering from the Community Grid Labs is adopted as a general event brokering middleware supports publish-subscribe messaging model with a dynamic collection of brokers provides services for TCP, UDP, Multicast, SSL and raw RTP clients Supports the communication through firewalls and proxies operate either in a client-server mode like JMS or in a completely distributed JXTA-like peer-to-peer mode. 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

uri="http://www.globalmmcs.org" email="gcf@indiana.edu" Narada Brokering Covers the heterogeneity of network transportation and provides unified multipoint transportation API Software multicast Communication over firewalls and proxy boundaries Communication over multiple transports Application level Quality of Service Filter messages to slow (collaborative/real-time) clients Provides robust, scalable and high efficient multipoint transportation services Availability and scalability Efficient routing and bandwidth utilizations 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

A/V Collaboration over publish/subscribe Middleware video streams (VS) VS {v1, v2, … vm } audio streams (AS) AS { a1, a2, … an } A/V endpoints: E1, E2, … En Each endpoint may send a single or multiple video streams, but only send an audio stream Different types of A/V endpoints have different collaboration capabilities. Multicast endpoints are able to receive multiple video and audio streams, display all the video streams in their screens, and mix all the audio streams by themselves Unicast endpoints like Polycom Via Video can only receive and play a single video and audio stream 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

Publish/subscribe of A/V A stream in VS and AS is regarded as a “topic” Each RTP packet from this stream is regarded as an “event” for this topic Only the sender of this stream can “publish” A/V events to this topic Other endpoints need to subscribe to this topic in order to receive the stream data Create mixed video and audio stream topics for unicast endpoints 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

Major Events for each stream topic Event Name Change in the stream NewStreamEvent This stream has been created ByeEvent The stream gives a BYE RTCP packet, indicating it has left the session TimeOutEvent This stream has not send any RTCP packet for a long time, indicating it may have left the session Active-To-Passive The stream has stopped sending RTP packet Passive-To-Active The stream resumed sending RTP packet 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

A/V Collaboration Architecture 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

RTP packets encapsulation RTPLink: RTP Packets ~ Narada Brokering Event Every legacy A/V client needs one corresponding RTPLink to be set up at a broker within broker network. Unicast RTPLink: Integer Topic Numbers for RTP and RTCP Multicast RTPLink: A reflector between NaradaBrokers and multicast groups, encapsulating raw RTP packets from a multicast IP address to RTP events, publishing these events to NaradaBrokers, and forwarding the data it receives from broker network on the same IP address 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

Audio Mixer, Video Mixer, Image Grabber Audio Mixing The audio mixer creates a mixed audio stream from all the audio streams in the session Video Mixing Video mixing makes the unicast users watch the pictures of multiple participants in a meeting through one video stream Video Thumbnail visualize the VS set in the session, embedded into the control panel of each endpoint, which Image grabbers capture video streams and save them as static JPEG files. All the media processing components can be distributed among the pool of the media servers connected to NaradaBrokering infrastructures. 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

H.323, SIP Gateway Servers, A/V Session Server H.323 and SIP gateway transform these protocol specific messages into XGSP signaling messages so that H.323 and SIP A/V endpoints could communicate with the XGSP A/V session server The session server implements session management logics creating/destroying A/V sessions allowing endpoints to join/leave session Allowing users to make audio/video selection, managing A/V application components 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

uri="http://www.globalmmcs.org" email="gcf@indiana.edu" Implementation Java Media Framework (JMF) : Openh323 is basis of H323 Gateway NIST SIP stack is basis of SIP Gateway NaradaBrokering is open source messaging from Indiana 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

uri="http://www.globalmmcs.org" email="gcf@indiana.edu" Demo Pics ( I ) 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

uri="http://www.globalmmcs.org" email="gcf@indiana.edu" Demo Pics ( II ) 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

uri="http://www.globalmmcs.org" email="gcf@indiana.edu" Demo Pics ( III ) 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

Please watch our demo at 5:30 PM ! Thank you! Please watch our demo at 5:30 PM ! 9/20/2018 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"