1 3D Tele-Collaboration over Internet2 Herman Towles, UNC-CH representing members of the National Tele-Immersion Initiative (NTII) ITP 2002 Juan-les-Pins, France 06 December D Tele-Collaboration over Internet2 Herman Towles, UNC-CH representing members of the National Tele-Immersion Initiative (NTII) ITP 2002 Juan-les-Pins, France 06 December 2002

2 NTII Collaborators & Co-authors University of North Carolina at Chapel HillUniversity of North Carolina at Chapel Hill –Wei-Chao Chen, Ruigang Yang, Sang-Uok Kum, and Henry Fuchs University of PennsylvaniaUniversity of Pennsylvania –Nikhil Kelshikar, Jane Mulligan, and Kostas Daniilidis Brown UniversityBrown University –Loring Holden, Bob Zeleznik, and Andy Van Dam Advanced Network & ServicesAdvanced Network & Services –Amela Sadagic and Jaron Lanier University of North Carolina at Chapel HillUniversity of North Carolina at Chapel Hill –Wei-Chao Chen, Ruigang Yang, Sang-Uok Kum, and Henry Fuchs University of PennsylvaniaUniversity of Pennsylvania –Nikhil Kelshikar, Jane Mulligan, and Kostas Daniilidis Brown UniversityBrown University –Loring Holden, Bob Zeleznik, and Andy Van Dam Advanced Network & ServicesAdvanced Network & Services –Amela Sadagic and Jaron Lanier

3 Clear Motivation to Provide Higher ResolutionHigher Resolution Larger, more immersive Field-of-ViewLarger, more immersive Field-of-View Participants at Accurate Geometric ScaleParticipants at Accurate Geometric Scale Eye ContactEye Contact Spatialized Audio (Group settings)Spatialized Audio (Group settings) More Natural Human-Computer InterfacesMore Natural Human-Computer Interfaces Higher ResolutionHigher Resolution Larger, more immersive Field-of-ViewLarger, more immersive Field-of-View Participants at Accurate Geometric ScaleParticipants at Accurate Geometric Scale Eye ContactEye Contact Spatialized Audio (Group settings)Spatialized Audio (Group settings) More Natural Human-Computer InterfacesMore Natural Human-Computer Interfaces

4 Related Work Improved Resolution & FOVImproved Resolution & FOV –Access Grid – Childers et al., 2000 –Commerical, multi-channel extensions of ‘1-camera to 1-display’ Gaze-AwarenessGaze-Awareness –MONJUnoCHIE System – Aoki et al., 1998 –Blue-C Project - Kunz and Spagno, –VIRTUE Project – Cooke, Kauff, Schreer et al., D Reconstruction/New Novel Views3D Reconstruction/New Novel Views –CMU’s Virtualized Reality Project – Narayanan, Kanade, 1998 –Visual Hull Methods – Matusik, McMillan et al, 2000 –VIRTUE Project – Cooke, Kauff, Schreer et al., Human Computer InterfacesHuman Computer Interfaces –T-I Data Exploration (TIDE) – Leigh, DeFanti et al., 1999 –VisualGlove Project - Constanzo, Iannizzotto, 2002 Improved Resolution & FOVImproved Resolution & FOV –Access Grid – Childers et al., 2000 –Commerical, multi-channel extensions of ‘1-camera to 1-display’ Gaze-AwarenessGaze-Awareness –MONJUnoCHIE System – Aoki et al., 1998 –Blue-C Project - Kunz and Spagno, –VIRTUE Project – Cooke, Kauff, Schreer et al., D Reconstruction/New Novel Views3D Reconstruction/New Novel Views –CMU’s Virtualized Reality Project – Narayanan, Kanade, 1998 –Visual Hull Methods – Matusik, McMillan et al, 2000 –VIRTUE Project – Cooke, Kauff, Schreer et al., Human Computer InterfacesHuman Computer Interfaces –T-I Data Exploration (TIDE) – Leigh, DeFanti et al., 1999 –VisualGlove Project - Constanzo, Iannizzotto, 2002

5 XTP: ‘Xtreme Tele-Presence UNC ‘Office of the Future’ Andrei State 1998

6 Research Snapshots

7 Presentation Outline Motivation and Related WorkMotivation and Related Work NTII Tele-Collaboration TestbedNTII Tele-Collaboration Testbed –Acquisition and 3D Reconstruction –Collaborative Graphics & User Interfaces –Rendering & Display –Network ResultsResults Future ChallengesFuture Challenges Motivation and Related WorkMotivation and Related Work NTII Tele-Collaboration TestbedNTII Tele-Collaboration Testbed –Acquisition and 3D Reconstruction –Collaborative Graphics & User Interfaces –Rendering & Display –Network ResultsResults Future ChallengesFuture Challenges

8 Scene Acquisition & Reconstruction Foreground: Real-Time Stereo AlgorithmForeground: Real-Time Stereo Algorithm –Frame Rate: 2-3 fps (550MHz Quad-CPU) - REAL-TIME! –Volume: 1 cubic meter –Resolution: 320x240 (15K-25K foreground points) Background: Scanning Laser RangefinderBackground: Scanning Laser Rangefinder –Frame Rate: 1 frame in minutes - OFFLINE! –Volume: Room-size –Resolution: More data than you can handle! Composite Live Foreground & Static Background Foreground: Real-Time Stereo AlgorithmForeground: Real-Time Stereo Algorithm –Frame Rate: 2-3 fps (550MHz Quad-CPU) - REAL-TIME! –Volume: 1 cubic meter –Resolution: 320x240 (15K-25K foreground points) Background: Scanning Laser RangefinderBackground: Scanning Laser Rangefinder –Frame Rate: 1 frame in minutes - OFFLINE! –Volume: Room-size –Resolution: More data than you can handle! Composite Live Foreground & Static Background

9 Real-Time Foreground Acquisition Trinocular Stereo Reconstruction AlgorithmTrinocular Stereo Reconstruction Algorithm –After background segmentation, find corresponding pixels in each image using MNCC method –3D ray intersection yields pixel depth –Median filter the disparity map to reduce outliers Produce 320x240 Depth Maps (1/z, R,G,B)Produce 320x240 Depth Maps (1/z, R,G,B) Trinocular Stereo Reconstruction AlgorithmTrinocular Stereo Reconstruction Algorithm –After background segmentation, find corresponding pixels in each image using MNCC method –3D ray intersection yields pixel depth –Median filter the disparity map to reduce outliers Produce 320x240 Depth Maps (1/z, R,G,B)Produce 320x240 Depth Maps (1/z, R,G,B) = Images courtesy of UPenn GRASP Lab +

10 UNC Acquisition Array Five Dell 6350 Quad-Processor Servers Seven Sony Digital 1394 Cameras – Five Trinocular Views

11 Stereo Processing Sequence Camera Views Disparity Maps 3 Views of Combined Point Clouds Images courtesy of UPenn GRASP Lab

12 Collaborative Graphics & User I/F

13 Shared 3D Objects Scene Graph SharingScene Graph Sharing –Distributed, Common Scene Graph Dataset –Local Changes, Shared Automatically with Remote Nodes Object Manipulation with 2D & 3D PointersObject Manipulation with 2D & 3D Pointers –3D Virtual Laser Pointing Device –Embedded magnetic tracker –Laser beam rendered as part of Scene Graph –One event/behavior button Scene Graph SharingScene Graph Sharing –Distributed, Common Scene Graph Dataset –Local Changes, Shared Automatically with Remote Nodes Object Manipulation with 2D & 3D PointersObject Manipulation with 2D & 3D Pointers –3D Virtual Laser Pointing Device –Embedded magnetic tracker –Laser beam rendered as part of Scene Graph –One event/behavior button

14 Rendering System Overview Rendering System Overview

15 3D Stereo Display Passive Stereo & Circular PolarizationPassive Stereo & Circular Polarization –Custom Filters on Projectors –Lightweight Glasses –Silvered Display Surface Front ProjectionFront Projection –Usable in any office/room –Ceiling-mounted Configurations Two Projector StereoTwo Projector Stereo –100% Duty Cycle –Brighter & No flicker –Permits multi-PC Rendering Passive Stereo & Circular PolarizationPassive Stereo & Circular Polarization –Custom Filters on Projectors –Lightweight Glasses –Silvered Display Surface Front ProjectionFront Projection –Usable in any office/room –Ceiling-mounted Configurations Two Projector StereoTwo Projector Stereo –100% Duty Cycle –Brighter & No flicker –Permits multi-PC Rendering

16 View-Dependent Rendering HiBall  6DOF TrackerHiBall  6DOF Tracker –3D Position & Orientation –Accurate, Low latency & noise –Headband-mounted Sensor –HiBall to Eyeball Calibration PC Network ServerPC Network Server HiBall  6DOF TrackerHiBall  6DOF Tracker –3D Position & Orientation –Accurate, Low latency & noise –Headband-mounted Sensor –HiBall to Eyeball Calibration PC Network ServerPC Network Server

17 Rendering Configurations One PC Configuration (Linux)One PC Configuration (Linux) –Dual-channel NVIDIA graphics Three PC Configuration (Linux)Three PC Configuration (Linux) –Separate left & right-eye rendering PCs w/NVIDIA graphics –One PC used as network interface, multicasts depth map stream to rendering PCs Performance – 933MHz PCs & GeForce2Performance – 933MHz PCs & GeForce2 –Interactive Display Rates of fps –Asynchronous updates of 3D Reconstruction (2-3Hz) & Scene Graph (20Hz) Newest Rendering Configuration 10-20XNewest Rendering Configuration 10-20X –2.4GHz, GeForce4, Multi-Threaded, VAR Arrays One PC Configuration (Linux)One PC Configuration (Linux) –Dual-channel NVIDIA graphics Three PC Configuration (Linux)Three PC Configuration (Linux) –Separate left & right-eye rendering PCs w/NVIDIA graphics –One PC used as network interface, multicasts depth map stream to rendering PCs Performance – 933MHz PCs & GeForce2Performance – 933MHz PCs & GeForce2 –Interactive Display Rates of fps –Asynchronous updates of 3D Reconstruction (2-3Hz) & Scene Graph (20Hz) Newest Rendering Configuration 10-20XNewest Rendering Configuration 10-20X –2.4GHz, GeForce4, Multi-Threaded, VAR Arrays

18 Network Considerations All Tests over Internet2All Tests over Internet2 Data Rates of ~20-75 Mbps from Armonk, NY and Philadelphia into Chapel HillData Rates of ~20-75 Mbps from Armonk, NY and Philadelphia into Chapel Hill –320 x 240 Resolution –Up to 5 Reconstruction Views per site –Frame Rates 2-3 fps TCP/IPTCP/IP Latency of 2-3 seconds typicalLatency of 2-3 seconds typical All Tests over Internet2All Tests over Internet2 Data Rates of ~20-75 Mbps from Armonk, NY and Philadelphia into Chapel HillData Rates of ~20-75 Mbps from Armonk, NY and Philadelphia into Chapel Hill –320 x 240 Resolution –Up to 5 Reconstruction Views per site –Frame Rates 2-3 fps TCP/IPTCP/IP Latency of 2-3 seconds typicalLatency of 2-3 seconds typical

19 Presentation Outline Motivation and Related WorkMotivation and Related Work NTII Tele-Collaboration TestbedNTII Tele-Collaboration Testbed –Acquisition and 3D Reconstruction –Collaborative Graphics & User Interfaces –Rendering & Display –Network ResultsResults Future ChallengesFuture Challenges Motivation and Related WorkMotivation and Related Work NTII Tele-Collaboration TestbedNTII Tele-Collaboration Testbed –Acquisition and 3D Reconstruction –Collaborative Graphics & User Interfaces –Rendering & Display –Network ResultsResults Future ChallengesFuture Challenges

20 Results ‘Roll the Tape’

21 Summary ‘One-on-One’ 3D Tele-Immersion Testbed‘One-on-One’ 3D Tele-Immersion Testbed Life-size, view-dependent, passive stereo displayLife-size, view-dependent, passive stereo display Interact with shared 3D Objects using a virtual laser pointerInteract with shared 3D Objects using a virtual laser pointer Half-Duplex Operation todayHalf-Duplex Operation today Operation over Internet2 between Chapel Hill, Philadelphia and ArmonkOperation over Internet2 between Chapel Hill, Philadelphia and Armonk Audio over H.323 or POTSAudio over H.323 or POTS ‘One-on-One’ 3D Tele-Immersion Testbed‘One-on-One’ 3D Tele-Immersion Testbed Life-size, view-dependent, passive stereo displayLife-size, view-dependent, passive stereo display Interact with shared 3D Objects using a virtual laser pointerInteract with shared 3D Objects using a virtual laser pointer Half-Duplex Operation todayHalf-Duplex Operation today Operation over Internet2 between Chapel Hill, Philadelphia and ArmonkOperation over Internet2 between Chapel Hill, Philadelphia and Armonk Audio over H.323 or POTSAudio over H.323 or POTS

22 Future Challenges Improved 3D Reconstruction QualityImproved 3D Reconstruction Quality –Larger Working Volume, Faster Frame Rates – 60 cameras –Fewer Reconstruction Errors (using structured light and adaptive correlation kernels) Reduce System Latency and Susceptibility to Network CongestionReduce System Latency and Susceptibility to Network Congestion –Pipelined architecture –Shunt Protocol (between TCP/UDP and IP layers) that allows multiple flows to do coordinated congestion control Full Duplex OperationFull Duplex Operation Unobtrusive OperationUnobtrusive Operation –No headmounts, No eyeglasses! Improved 3D Reconstruction QualityImproved 3D Reconstruction Quality –Larger Working Volume, Faster Frame Rates – 60 cameras –Fewer Reconstruction Errors (using structured light and adaptive correlation kernels) Reduce System Latency and Susceptibility to Network CongestionReduce System Latency and Susceptibility to Network Congestion –Pipelined architecture –Shunt Protocol (between TCP/UDP and IP layers) that allows multiple flows to do coordinated congestion control Full Duplex OperationFull Duplex Operation Unobtrusive OperationUnobtrusive Operation –No headmounts, No eyeglasses!

23 Thank You Research funded by Advanced Network and Services, Inc. and National Science Foundation (USA) Research funded by Advanced Network and Services, Inc. and National Science Foundation (USA)

24 UPenn Acquisition Array Fifteen Sony Digital 1394 Cameras – Five Trinocular Views

25 System Overview

26 Past Experiments Past Experiments UPenn Philadelphia Advanced Armonk, NY UNC Chapel Hill 3D Data + 2D Images 2D Video + Audio UNC Chapel Hill 3D Data + 2D Images 2D Video + Audio Advanced Armonk, NY Scene Bus Data With Collaboration w/o Collaboration