1. Coding Approaches for End-to-End 3D TV Systems
Seo, DongMahn
24th Jan., 2006
Anthony Vetro, Wojciech Matusik, Hanspeter Pfister, Jun Xin
Mitsubishi Electric Research Laboratories, Cambridge

2. Contents Introduction System Overview Multi-View Video Coding Results
Concluding Remarks
24 January 2005

3. Introduction Three-dimensional TV
The next revolution in the history of television.
Not viable in the early 1960’s until recently
Requires capturing and displaying multi-view video of real-life dynamic scenes
High processing and bandwidth requirements
24 January 2005

4. Introduction (cont.) Prototype 3D TV system End-to-End 3D TV
Except for broadcasting over a digital channel
All aspects of 3D TV : multi-view video acquisition, compression, and 3D display
Distributed Architecture
Distributed clusters of PCs
To handle the large processing and bandwidth requirements of multi-view video.
Real-Time Performance
Acquired and displayed in real-time with only a minimal amount of lag
24 January 2005

5. Introduction (cont.) Scalability Projection-based 3D display
completely scalable in the number of acquired, transmitted, and displayed views
Projection-based 3D display
array of 16 projectors
to high-resolution display with 16×1024 ×768 pixels
A lenticular screen provides auto-stereocopic images with horizontal parallax and 16 views
Computational alignment
image alignment and intensity adjustment of 3D display
completely automatic using a camera in the loop
24 January 2005

6. System Overview All PCs
3 GHz Pentium 4 processors, 2 GB of RAM, and run WindowsXP
24 January 2005

7. System Overview (cont.)
Acquisition
Some systems
2D array of lenslets
optical fibers in combination
with a Fresnel lens in front of a high-definition camera to capture multiple views simultaneously
multiple views per lenslet both horizontal and vertical directions
limited resolution of the camera sensor (fixed at HDTV)
very limited number of lenslets
to acquire high-resolution multi-view video
an array of synchronized cameras
camera with cluster of PCs
Stanford multi-camera array : up to 128 cameras
this paper
16 high-resolution (1300×1030) cameras, progressive video at 12 frames per second
pair of cameras are connected by IEEE 1394 bus to one of eight producer PCs.
24 January 2005

8. System Overview (cont.)
to guarantee good image quality on the display side
to interpolate dense virtual views from the sparse video data
impossible without
a scene model such as per-pixel depth maps
a prior model of the acquired objects
densely-spaced linear array of 17 cameras
roughly perpendicular to a common camera plane
the output views on the display correspond to the acquired views of the cameras
impossible to align multiple cameras precisely
using lightfield rendering on the display side to synthesize new views
acquire or compute per-pixel depth maps to improve the view-interpolation quality
24 January 2005

9. System Overview (cont.)
3D Display
24 January 2005

10. System Overview (cont.)
Parallax and Lenticular Displays
24 January 2005

11. System Overview (cont.)
Multi-Projector Implementation
24 January 2005

12. System Overview (cont.)
24 January 2005

13. System Overview (cont.)
View Interpolation
one possible implementation of our 3D TV system
one-to-one mapping of cameras to projectors
without view-interpolation
very simple and scales well
not very flexible
cameras and projectors need to be equally spaced, which is hard to achieve in practice
cannot handle the case when the number of cameras and projectors is not the same
not provide the ability to interactively control the viewpoint, a feature that has been termed free-viewpoint video
24 January 2005

14. System Overview (cont.)
performance of our lightfield rendering implementation
completely independent of the total number of transmitted views
each virtual output view
required only a small number of source frames
maximum bandwidth on the network between consumers
limited, which is important to provide scalability
24 January 2005

15. Multi-View Video Coding
3D TV broadcasting
transmitted all views to multiple users simultaneously
transmitting 16 uncompressed video streams
with 1280×720 resolution (4:2:0 format) at 30 frames per second
5.3Gb/sec bandwidth
efficient multi-view video coding
to make 3D TV attractive to broadcasters and network operators
simulcast
temporally encode the individual video streams
our current prototype system
full camera resolution (1300×1030) encoded in real-time with MPEG-2 and decoded on the producer PCs
advantage : commercially available codecs could be used
coding efficiency is generally lower than with other multi-view coding approaches
24 January 2005

16. Multi-View Video Coding (cont.)
Another approach to multi-view video compression :: European ATTEST project
to reduce the data to a single view with per-pixel depth map
compressed in real-time and broadcast as an MPEG-2 enhancement layer
receiver side : stereo or multi-view images are generated
using image-based rendering
difficult to generate high-quality output
occlusions or high disparity in the scene
single view cannot capture view-dependent appearance effects
such as reflections and specular highlights
MPEG
H.264/AVC
24 January 2005

17. Multi-View Video Coding (cont.)
additional issues to consider to achieve optimal compression efficiency
cameras are not expected to be perfectly aligned
illumination and color between views is not consistent due to intrinsic parameters of each camera
sampling theory
minimal number of views required for transmission and rendering
lower bound for the minimum number of samples required for lightfield/lumigraph rendering
related to camera resolution and scene depth complexity
the adaptive dropping of segments in the multi-view video
entire frames or blocks
interpolated or synthesized at the receiving end
have strong potential to improve the coding efficiency in the rate-distortion sense
24 January 2005

18. Result top : viewer side of the front-projection display
bottom : corresponding images of the camera array
24 January 2005

19. Concluding Remarks
implemented the first real-time end-to-end 3D TV system
with enough views and resolution to provide a truly immersive 3D experience
24 January 2005

20. Thank you!
24 January 2005