The Future of Telepresence Alex Eleftheriadis,

2 IMTC 2010/06/15 What is Telepresence? Wikipedia: Telepresence refers to a set of technologies which allow a person to feel as if they were present, to give the appearance that they were present, or to have an effect, at a location other than their true location. Alternatively: Videoconferencing sucks! What can you do to make it better if you have all the money you need?

3 IMTC 2010/06/15 Lessons Learned Elsewhere Computing Networking Communications

4 IMTC 2010/06/15 DEC VAX (1977, 0.5 MIPS) VAX 11/780 (first two cabinet sections), shown with Unibus expansion cabinet (middle cabinet section), two tape drives, two RP05 or RP06 removable pack disk drives, a DECwriter printing terminal, and a VT52 CRT terminal

5 IMTC 2010/06/15 iPhone 3G (2008, 1250 MIPS)

6 IMTC 2010/06/15 Lessons Learned Elsewhere Computing Fast Personal Portable Networking Scalable Ubiquitous Communications Reliable Ubiquitous Mobile

7 IMTC 2010/06/15 Telepresence Today Extremely expensive Not scalable Not portable Not personal Very high quality => Traditional videoconferencing: same, but sacrifices quality for low cost.

8 IMTC 2010/06/15 Quality of Experience Cost TypeDesktopExecutiveRoomTelepresence $100’s$1000’s$10,000’s$100,000’s Quality of Experience 10 hours/month 100 hours/month Telepresence quality At least SD quality per face Total delay under 250ms latency Legacy video conferencing quality

9 IMTC 2010/06/15 Key Videoconferencing System Characteristics Rate Matching Support for heterogeneous endpoints and access networks Personalized Layout Each user selects the resolution and users he/she wants to see Low Delay Interactivity requires <250msec end-to-end Error Resilience Tolerance to packet loss rates >10% Error Localization An error in one user should not affect other users Complexity Lower complexity = Lower cost Low complexity allows to integrate with existing network services (e.g., in cheap routers) => How can we have all that, but with high quality and scalability?

10 IMTC 2010/06/15 H.264 SVC Scalable Video Coding (SVC) Developed by JVT (=Joint Video Team of ITU and ISO) SVC is Amendment 3 to H.264 AVC Most of SVC is in Annex G of H.264 AVC CFP April 2004, Consented in Nov 2007 RTP payload format nearly completed Architecture centered on VidyoRouter™ (“VRU”)

11 IMTC 2010/06/15 Temporal Structure of Non-Scalable Codecs

12 IMTC 2010/06/15 Temporal Scalability

13 IMTC 2010/06/15 Spatial Scalability Multiplex SVC Bitstream PredictionCoding H.264/AVC- compatible base layer PredictionCoding Prediction Coding Scale

14 IMTC 2010/06/15 Temporal + Spatial Scalability

15 IMTC 2010/06/15 Temporal + Spatial Scalability QVGA 30 fps QVGA 30 fps

16 IMTC 2010/06/15 Temporal + Spatial Scalability VGA 15 fps VGA 15 fps

17 IMTC 2010/06/15 What does the VidyoRouter do? 1 Mbps 500 Kbps 150 Kbps VidyoRouter™ High Resolution High Frame Rate Low Resolution Low Frame Rate 2 Mbps High Resolution Medium Frame Rate Medium Resolution Medium Frame Rate High Resolution High Frame Rate

18 IMTC 2010/06/15 Traditional MCU Architecture ENCODE DECODE ENCODE DECODE Quality Loss from Cascaded Encodings High Cost High Delay High Cost High Delay DECODE ENCODE COMPOSE

19 IMTC 2010/06/15 VidyoRouter™ vs. MCU VidyoRouter is simply an application layer router Nearly zero-delay (~20 msec), allows interactive multi-point sessions Eliminates video quality loss due to transcoding Rate matching and personal layout are simple routing decisions Error localization and robustness up to 20% packet loss rates Computing power that rides the Intel ® curve Endpoint processing of the video from different participants makes the VidyoRouter highly scalable Similar complexity to other network appliances

20 IMTC 2010/06/15 Enhancement Layer Base Layer Significant Impact Conventional Coding Scalable Coding Minor or No Impact SVC+VR Eliminates Error Resilience Problem

21 IMTC 2010/06/15 Temporal Scalability

22 IMTC 2010/06/15 Concluding Remarks Videoconferencing and telepresence will remain “small”, unless our technology enables scaling with high quality and low cost. Video coding scalability and the video router architecture allow: Replacement of the MCU with a much simpler, scalable device – the Video Routing Unit (VRU) Very high error resilience (=high quality user experience) Co-existence of lower-end systems with telepresence systems Use of standard (Intel) hardware components – riding the Intel curve These concepts finally solve 30-year-old packet video problems, and are the key ingredients for making telepresence “personal”.

23 IMTC 2010/06/15 IMTC SuperOp! Room

24 IMTC 2010/06/15 IMTC SuperOp! Room

25 IMTC 2010/06/15 Concluding Remarks Videoconferencing and telepresence will remain “small”, unless our technology enables scaling with high quality and low cost. Video coding scalability and the video router architecture allow: Replacement of the MCU with a much simpler, scalable device Very high error resilience (=high quality user experience) Use of standard (Intel) hardware components – riding the Intel curve These concepts finally solve 30-year-old packet video problems, and are the key ingredients for making telepresence “personal”. Can I get the same benefit from simulcasting? Or without two spatial layers? Or ….? Partially. If you use part of the recipe, you get only part of the flavor. Wait … So, the future of telepresence is … Vidyo?

Thank You!