1. Real-Time Collaborative Environments Colin Perkins http://csperkins.org/

2. Copyright © 2005 University of Glasgow Talk Outline Real-time Collaborative Environments –Definition and Requirements –Example: AccessGrid –History and Current Status –Underlying Technologies and Standards –Capture and Analysis of Real-time Sessions Research Issues and Future Directions …from the perspective of a developer of those applications, and of the relevant technical standards

3. Copyright © 2005 University of Glasgow Original Picture ©1998 UCL Picture from http://www.accessgrid.org/ Real-Time Collaborative Environments Support collaborative work by scientists, industry and others –Both one-to-one and group communication –Widely distributed participants; heterogeneity Provide high-quality audio-visual media Provide sense of community and presence –Venues as a rendezvous point; familiar virtual meeting places Integrate with other computing resources –Visualization, shared state & data repositories, computational resources, applications Secure and trusted infrastructure

4. Copyright © 2005 University of Glasgow Picture from http://www.accessgrid.org/ The AccessGrid Example: A widely used real-time collaborative environment –Supports many-to-many interactive meetings –Typically room based, although not required by the underlying technology Components: –Venue client –Media tools –User experience A framework for initiating real-time collaborative work sessions –Default media tools provided, can be replaced

5. Copyright © 2005 University of Glasgow The AccessGrid: Venue Client Venue client provides the rendezvous point to join a session Maintains state, participant list, links to other venues, etc. Picture from http://www.accessgrid.org/

6. Copyright © 2005 University of Glasgow The AccessGrid: Typical Media Tools Joining a venue starts multicast audio and video tools on the controller display –Audio level and activity indicator –Video thumbnails and statistics A separate display shows the main video windows…

7. Copyright © 2005 University of Glasgow The AccessGrid: User Experience Picture from http://www.accessgrid.org/ Tool development ongoing - the user experience will change with time…

8. Copyright © 2005 University of Glasgow History and Standards Development Long development history Strong open standards Commercial interest 1985197519952005 Mbone toolsAccessGrid vi c vatrat nte wb sdr Grid Computing RTPNVP STST-II SIP H.323 NVP-II H.320 3G phones Video conferencing IP Multicast RFC 741

9. Copyright © 2005 University of Glasgow Current Deployment Status Hundreds of nodes at sites worldwide Large scale multicast video conferences Standard, if primitive, media formats Non-standard web service control infrastructure Tens of millions of mobile phones Voice telephony and multicast streaming Standard, advanced, media formats Standard SIP-based control infrastructure Dedicated video conferencing Wi-Fi and wired SIP-based VoIP PC-based video streaming Mixture of standard and proprietary Others… Largely successful deployment of open standard media coding and transport protocols –With notable exceptions, and continuing patent issues… Largely successful deployment of open standard session setup and control protocols –AccessGrid, although open, is the main exception to the use of standards-based protocols Independent and interoperable implementations available and widely used

10. Copyright © 2005 University of Glasgow Underlying Technologies Complex protocol stack, many options Two phase operation: control then data Media codecs Light weight sessions IP TCPUDP/UDPlite SIPSAPRTP SDP, SDPng Session and call control RTSP DCCP Web/Grid Services STUN ICE/TURN NAT Traversal SIP+SDP Web services RTP + media codecs

11. Copyright © 2005 University of Glasgow Control Protocols: Concepts Many control protocols exist –For telephony and integration with the PSTN –For streaming video, television and radio –For collaborative work in a Grid environment Some common features: –Initial rendezvous point and user location –Media format negotiation –Address and port negotiation –Session metadata Dates and times Additional information Participant scheduling Shared state –Negotiation of security parameters v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000

12. Copyright © 2005 University of Glasgow Control Protocols: Implementations Session Initiation Protocol (SIP) –Commercial video conferencing products –Commercial telephony and voice-over-IP Real-Time Streaming Protocol –Video streaming (e.g. QuickTime) Web services –AccessGrid All evolutions of HTTP designed to support negotiation and control of real-time media sessions Complex, flexible Large toolkits Initial negotiation Ongoing control

13. Copyright © 2005 University of Glasgow Data Transfer Protocols Once negotiation completed, media data transferred Complex due to nature of IP networks –A “best effort” packet delivery service –Performance not guaranteed Loss, delay, reordering, corruption, duplication –Higher level protocols must compensate Application complexity, simple network An application must correct errors, recover media timing, etc., in addition to processing audio/visual data Cannot use TCP, since require accurate timing Main standard is RTP: Real-time Transport Protocol IP Ethernet ADSL PPP Optical Fibre Twisted Pair TCPUDP HTTP FTP SMTPRTP SIP RTSP HTML MIME Media codecs Application programs Wireless

14. Copyright © 2005 University of Glasgow A standard protocol for real-time collaborative environments on IP networks Provides features for: –Timing recovery and synchronisation –Loss detection and recovery –Media format identification –Participant identification and presence –Reception quality statistics but must be implemented in an application! A toolkit for real-time data transfer Payload data Padding Synchronization source (SSRC) identifier Timestamp Sequence NumberPTMPXVCC Contributing source (CSRC) identifiers RTP: Real-time Transport Protocol

15. Copyright © 2005 University of Glasgow Capture and Analysis of Media Sessions Media capture and decoding is complex –One cannot simply dump packets to a file! –Need to understand transport protocol Reconstruct timing, correct errors, log metadata –Need to understand the signalling protocol Metadata needed to find and understand the media Data storage format: –Standard file formats exist, but often lacking in metadata and context –Formats based on the ISO multimedia file format better? Relevant? “RTP Payload Format for ETSI ES 201 108 Distributed Speech Recognition Encoding”, RFC 3557

16. Copyright © 2005 University of Glasgow Underlying Technologies Flexible and sophisticated control protocols –Initial rendezvous and negotiation –Ongoing control during a collaborative work session Capable data transfer protocols; range of media formats “Some assembly required” –Protocols are expected to be tightly coupled to applications –The design rejects much of the traditional protocol layering –Collaborative work environments use a wide range of protocols; often in different ways to commercial products

17. Copyright © 2005 University of Glasgow Research Issues and Future Directions Continual evolution of protocols –New media formats –Extension of the control frameworks Emergency calls (E.911) Presence and messaging NAT traversal, seamless operation over present-day Internet Integration of AccessGrid with standard control protocols –SIP-based architecture? Desirable to move away from “Grid” protocols, to a more standard architecture, to leverage massive commercial development Evolution of IP multicast  peer-to-peer overlays

18. Copyright © 2005 University of Glasgow Summary Real-time Collaborative Environments –Definition and Requirements –Example: AccessGrid –History and Current Status –Underlying Technologies and Standards –Capture and Analysis of Real-time Sessions Research Issues and Future Directions Any Questions?