Coding Approaches for End-to-End 3D TV Systems Seo, DongMahn 24th Jan., 2006 Anthony Vetro, Wojciech Matusik, Hanspeter Pfister, Jun Xin Mitsubishi Electric Research Laboratories, Cambridge

Contents Introduction System Overview Multi-View Video Coding Results Concluding Remarks 24 January 2005

Introduction Three-dimensional TV The next revolution in the history of television. Not viable in the early 1960’s until recently Requires capturing and displaying multi-view video of real-life dynamic scenes High processing and bandwidth requirements 24 January 2005

Introduction (cont.) Prototype 3D TV system End-to-End 3D TV Except for broadcasting over a digital channel All aspects of 3D TV : multi-view video acquisition, compression, and 3D display Distributed Architecture Distributed clusters of PCs To handle the large processing and bandwidth requirements of multi-view video. Real-Time Performance Acquired and displayed in real-time with only a minimal amount of lag 24 January 2005

Introduction (cont.) Scalability Projection-based 3D display completely scalable in the number of acquired, transmitted, and displayed views Projection-based 3D display array of 16 projectors to high-resolution display with 16×1024 ×768 pixels A lenticular screen provides auto-stereocopic images with horizontal parallax and 16 views Computational alignment image alignment and intensity adjustment of 3D display completely automatic using a camera in the loop 24 January 2005

System Overview All PCs 3 GHz Pentium 4 processors, 2 GB of RAM, and run WindowsXP 24 January 2005

System Overview (cont.) Acquisition Some systems 2D array of lenslets optical fibers in combination with a Fresnel lens in front of a high-definition camera to capture multiple views simultaneously multiple views per lenslet both horizontal and vertical directions limited resolution of the camera sensor (fixed at HDTV) very limited number of lenslets to acquire high-resolution multi-view video an array of synchronized cameras camera with cluster of PCs Stanford multi-camera array : up to 128 cameras this paper 16 high-resolution (1300×1030) cameras, progressive video at 12 frames per second pair of cameras are connected by IEEE 1394 bus to one of eight producer PCs. 24 January 2005

System Overview (cont.) to guarantee good image quality on the display side to interpolate dense virtual views from the sparse video data impossible without a scene model such as per-pixel depth maps a prior model of the acquired objects densely-spaced linear array of 17 cameras roughly perpendicular to a common camera plane the output views on the display correspond to the acquired views of the cameras impossible to align multiple cameras precisely using lightfield rendering on the display side to synthesize new views acquire or compute per-pixel depth maps to improve the view-interpolation quality 24 January 2005

System Overview (cont.) 3D Display 24 January 2005

System Overview (cont.) Parallax and Lenticular Displays 24 January 2005

System Overview (cont.) Multi-Projector Implementation 24 January 2005

System Overview (cont.) 24 January 2005

System Overview (cont.) View Interpolation one possible implementation of our 3D TV system one-to-one mapping of cameras to projectors without view-interpolation very simple and scales well not very flexible cameras and projectors need to be equally spaced, which is hard to achieve in practice cannot handle the case when the number of cameras and projectors is not the same not provide the ability to interactively control the viewpoint, a feature that has been termed free-viewpoint video 24 January 2005

System Overview (cont.) performance of our lightfield rendering implementation completely independent of the total number of transmitted views each virtual output view required only a small number of source frames maximum bandwidth on the network between consumers limited, which is important to provide scalability 24 January 2005

Multi-View Video Coding 3D TV broadcasting transmitted all views to multiple users simultaneously transmitting 16 uncompressed video streams with 1280×720 resolution (4:2:0 format) at 30 frames per second 5.3Gb/sec bandwidth efficient multi-view video coding to make 3D TV attractive to broadcasters and network operators simulcast temporally encode the individual video streams our current prototype system full camera resolution (1300×1030) encoded in real-time with MPEG-2 and decoded on the producer PCs advantage : commercially available codecs could be used coding efficiency is generally lower than with other multi-view coding approaches 24 January 2005

Multi-View Video Coding (cont.) Another approach to multi-view video compression :: European ATTEST project to reduce the data to a single view with per-pixel depth map compressed in real-time and broadcast as an MPEG-2 enhancement layer receiver side : stereo or multi-view images are generated using image-based rendering difficult to generate high-quality output occlusions or high disparity in the scene single view cannot capture view-dependent appearance effects such as reflections and specular highlights MPEG H.264/AVC 24 January 2005

Multi-View Video Coding (cont.) additional issues to consider to achieve optimal compression efficiency cameras are not expected to be perfectly aligned illumination and color between views is not consistent due to intrinsic parameters of each camera sampling theory minimal number of views required for transmission and rendering lower bound for the minimum number of samples required for lightfield/lumigraph rendering related to camera resolution and scene depth complexity the adaptive dropping of segments in the multi-view video entire frames or blocks interpolated or synthesized at the receiving end have strong potential to improve the coding efficiency in the rate-distortion sense 24 January 2005

Result top : viewer side of the front-projection display bottom : corresponding images of the camera array 24 January 2005

Concluding Remarks implemented the first real-time end-to-end 3D TV system with enough views and resolution to provide a truly immersive 3D experience 24 January 2005

Thank you! 24 January 2005