1 Combining Approximate Geometry with VDTM – A Hybrid Approach to 3D Video Teleconferencing Celso Kurashima 2, Ruigang Yang 1, Anselmo Lastra 1 1 Department of Computer Science University of North Carolina at Chapel Hill 2 Laboratório de Sistemas Integráveis - LSI Escola Politécnica da Universidade de São Paulo Fortaleza, October 8th, 2002 Combining Approximate Geometry with VDTM – A Hybrid Approach to 3D Video Teleconferencing Celso Kurashima 2, Ruigang Yang 1, Anselmo Lastra 1 1 Department of Computer Science University of North Carolina at Chapel Hill 2 Laboratório de Sistemas Integráveis - LSI Escola Politécnica da Universidade de São Paulo Fortaleza, October 8th, 2002 SIBGRAPI 2002

2 Introduction

3 Introduction Video Conference 2D vs. 3D Video Conference 2D vs. 3D 1 camera Many cameras Fixed viewpont Free viewpoint No eye contact Eye contact Standard Video Computer Vision & Computer Graphics

4 Outline 3D Video Teleconference System3D Video Teleconference System Geometry ExtractionGeometry Extraction Rendering Images with VDTMRendering Images with VDTM ResultsResults ConclusionConclusion 3D Video Teleconference System3D Video Teleconference System Geometry ExtractionGeometry Extraction Rendering Images with VDTMRendering Images with VDTM ResultsResults ConclusionConclusion

5 3D Video Teleconference System (1/3) Top ViewTop View (A)(B) (A)(B)

6 3D-Video Teleconference System (2/3)

7 3D-Video Teleconference System (3/3) How does it work?How does it work? 1 st : Create a geometry proxy of the person1 st : Create a geometry proxy of the person 2 nd : Texture map images onto the proxy with VDTM : an IBR method developed by Debevec et al. (’96, ’98)2 nd : Texture map images onto the proxy with VDTM : an IBR method developed by Debevec et al. (’96, ’98) How does it work?How does it work? 1 st : Create a geometry proxy of the person1 st : Create a geometry proxy of the person 2 nd : Texture map images onto the proxy with VDTM : an IBR method developed by Debevec et al. (’96, ’98)2 nd : Texture map images onto the proxy with VDTM : an IBR method developed by Debevec et al. (’96, ’98)

8 Geometry Proxy Extraction (1/5) The proxy : a simple geometric representation of the personThe proxy : a simple geometric representation of the person Mesh of trianglesMesh of triangles Map textures onto the triangles facesMap textures onto the triangles faces Extraction: a pair of camerasExtraction: a pair of cameras May be two of texture camerasMay be two of texture cameras The proxy : a simple geometric representation of the personThe proxy : a simple geometric representation of the person Mesh of trianglesMesh of triangles Map textures onto the triangles facesMap textures onto the triangles faces Extraction: a pair of camerasExtraction: a pair of cameras May be two of texture camerasMay be two of texture cameras

9 Geometry Proxy Extraction (2/5) Algorithm- Plane + Parallax (Kumar, 94) method:Algorithm- Plane + Parallax (Kumar, 94) method: –Robust Plane Fitting –Stereo Feature tracking Algorithm- Plane + Parallax (Kumar, 94) method:Algorithm- Plane + Parallax (Kumar, 94) method: –Robust Plane Fitting –Stereo Feature tracking

10 Geometry Proxy Extraction (3/5) Robust Plane FittingRobust Plane Fitting –Segmentation –Points on the Silhouette –Matching –Fit Plane –Distance –Std. Dev. –Remove distant points –Repeat Robust Plane FittingRobust Plane Fitting –Segmentation –Points on the Silhouette –Matching –Fit Plane –Distance –Std. Dev. –Remove distant points –Repeat

11 Geometry Proxy Extraction (4/5) Stereo Feature Tracking KLT tracker (Kanade-Lucas-Tomasi, ‘91, ‘94)

12 Geometry Proxy Extraction (5/5) Triangulation & Proxy formation + =

13 Rendering Images with VDTM – View Dependent Texture Mapping (1/2) VDTM requires a good spatial geometry of the objects (Debevec98)VDTM requires a good spatial geometry of the objects (Debevec98) Our system [ Buehler et al (2001) and Heigl et al (1999) ]Our system [ Buehler et al (2001) and Heigl et al (1999) ] Spatial geometry == proxy Textures == live images (from cameras) Textures are mapped onto the proxy according to the viewpointTextures are mapped onto the proxy according to the viewpoint VDTM requires a good spatial geometry of the objects (Debevec98)VDTM requires a good spatial geometry of the objects (Debevec98) Our system [ Buehler et al (2001) and Heigl et al (1999) ]Our system [ Buehler et al (2001) and Heigl et al (1999) ] Spatial geometry == proxy Textures == live images (from cameras) Textures are mapped onto the proxy according to the viewpointTextures are mapped onto the proxy according to the viewpoint

14 Rendering Images with VDTM – View Dependent Texture Mapping (2/2) Virtual Camera at Viewpoint DVirtual Camera at Viewpoint D Texture from cameras C i mapped onto the triangles facesTexture from cameras C i mapped onto the triangles faces Blending weights in vertex VBlending weights in vertex V Angles  i, used to compute the weights valuesAngles  i, used to compute the weights values  i = exp(-  i 2 /2.  2 ) Virtual Camera at Viewpoint DVirtual Camera at Viewpoint D Texture from cameras C i mapped onto the triangles facesTexture from cameras C i mapped onto the triangles faces Blending weights in vertex VBlending weights in vertex V Angles  i, used to compute the weights valuesAngles  i, used to compute the weights values  i = exp(-  i 2 /2.  2 )

15 Results (1/3) Geometry Proxy & Image rendering with VDTMGeometry Proxy & Image rendering with VDTM Geometry ProxyGeometry Proxy Image rendered with 2 texture camerasImage rendered with 2 texture cameras Image rendered with 4 texture camerasImage rendered with 4 texture cameras

16 Results (2/3) Cameras: Firewire IEEE 1394 SONYCameras: Firewire IEEE 1394 SONY –Frame size: 320x240 pixels Geometry/Renderer PCGeometry/Renderer PC –Processor: Intel Pentium4, 2.2 GHz –Graphics card: nVidia GeForce3 Video Frame Rate: 3-4 fpsVideo Frame Rate: 3-4 fps Cameras: Firewire IEEE 1394 SONYCameras: Firewire IEEE 1394 SONY –Frame size: 320x240 pixels Geometry/Renderer PCGeometry/Renderer PC –Processor: Intel Pentium4, 2.2 GHz –Graphics card: nVidia GeForce3 Video Frame Rate: 3-4 fpsVideo Frame Rate: 3-4 fps

17 Results (3/3) 3D Video Teleconferencing3D Video TeleconferencingMovie Movie

18 Conclusions A hybrid system for 3D Video TeleconferencingA hybrid system for 3D Video Teleconferencing Fast geometry proxy extraction, using a robust plane fitting method and stereo feature tracking, combined with view- dependent texture mappingFast geometry proxy extraction, using a robust plane fitting method and stereo feature tracking, combined with view- dependent texture mapping Real-time demonstration with personal computers and commodity graphics cardReal-time demonstration with personal computers and commodity graphics card A hybrid system for 3D Video TeleconferencingA hybrid system for 3D Video Teleconferencing Fast geometry proxy extraction, using a robust plane fitting method and stereo feature tracking, combined with view- dependent texture mappingFast geometry proxy extraction, using a robust plane fitting method and stereo feature tracking, combined with view- dependent texture mapping Real-time demonstration with personal computers and commodity graphics cardReal-time demonstration with personal computers and commodity graphics card

19 Future Work Vision-based head-tracking for viewpoint controlVision-based head-tracking for viewpoint control Increase of the frame rate with faster segmentationIncrease of the frame rate with faster segmentation Vision-based head-tracking for viewpoint controlVision-based head-tracking for viewpoint control Increase of the frame rate with faster segmentationIncrease of the frame rate with faster segmentation

20 Acknowledgements Herman TowlesHerman Towles Office of the Future (OOTF) group at UNC-CHOffice of the Future (OOTF) group at UNC-CH U.S. Dept. of Energy and Sandia National Labs.U.S. Dept. of Energy and Sandia National Labs. U.S. National Science FoundationU.S. National Science Foundation NEC/CPDIANEC/CPDIA Herman TowlesHerman Towles Office of the Future (OOTF) group at UNC-CHOffice of the Future (OOTF) group at UNC-CH U.S. Dept. of Energy and Sandia National Labs.U.S. Dept. of Energy and Sandia National Labs. U.S. National Science FoundationU.S. National Science Foundation NEC/CPDIANEC/CPDIA

21 Thank you!