1. 2010/10/13VCLAB, National Tsing Hua University, Taiwan1 Multiview Video Kai-Chao Yang

2. 2010/10/13VCLAB, National Tsing Hua University, Taiwan2 Outline History of Video Coding Standards Definition of Multiview Video Applications of Multiview Video Concept of Stereo Video Multiview TV/Video System  Multiview Content  Capture  Correction  Coding  Display

3. 2010/10/13VCLAB, National Tsing Hua University, Taiwan3 History of Video Coding Standards H.264/MPEG4 AVCH.262/MPEG-2 19841986198819901992199419961998200020022004200620082010 H.261H.263H.263+H.263++ MPEG-1MPEG-4 v2/visual SVCMVC ISO MPEG JVT ITU-T VCEG FRExt

4. 2010/10/13VCLAB, National Tsing Hua University, Taiwan4 Definition of Multiview Video Multiview video  Multiple cameras are used to simultaneously acquire various viewpoints of a scene. Multiview video coding  Encoding of sequences captured simultaneously from multiple cameras using a single video stream.

5. 2010/10/13VCLAB, National Tsing Hua University, Taiwan5 Applications of Multiview Videos Free viewpoint TV  The viewers can experience the free viewpoint navigation within the range covered by the cameras  http://www.youtube.com/watch?v=0yP_J6M4fiU http://www.youtube.com/watch?v=0yP_J6M4fiU Three-dimensional TV Immersive teleconference

6. 2010/10/13VCLAB, National Tsing Hua University, Taiwan6 Examples (1/4) Entertainments  3D game, FVV, … Medicine  3D microscope, 3D endoscope, … Education  3D model, 3D classroom, … Surveillance  Nurse, object recognition and tracking, … Simulation  Flight simulation, …

7. 2010/10/13VCLAB, National Tsing Hua University, Taiwan7 Examples (2/4) 3D microscope and 3D endoscope http://www.njneurosurgeons.com/news/2010/04/19/new -jerseys-first-3d-endoscopic-surgery-performed-by-ninj- faculty.html

8. 2010/10/13VCLAB, National Tsing Hua University, Taiwan8 Examples (3/4) Pedestrian Tracking Kyungnam Kim and Larry S. Davis, "Multi-Camera Tracking and Segmentation of Occluded People on Ground Plane using Search-Guided Particle Filtering", European Conference on Computer Vision (ECCV), LNCS, 2006.

9. 2010/10/13VCLAB, National Tsing Hua University, Taiwan9 Examples (4/4) Flight Simulation

10. 2010/10/13VCLAB, National Tsing Hua University, Taiwan10 Concept of Stereo Image/Video 3D information: Determination of relative depths in a perceived scene  Stereopsis  Accommodation of the eyeball  Occlusion  Linear perspective  Vertical position  … 2D knowledge

11. 2010/10/13VCLAB, National Tsing Hua University, Taiwan11 Concept of Stereo Video (3D Video) Stereopsis  visual perception leading to the sensation of depth from the two slightly different projections of the world onto the retinas of the two eyes

12. 2010/10/13VCLAB, National Tsing Hua University, Taiwan12 Concept of Stereo Video

13. 2010/10/13VCLAB, National Tsing Hua University, Taiwan13 Multiview TV System ContentDeliveryDisplayEncodingDecoding

14. 2010/10/13VCLAB, National Tsing Hua University, Taiwan14 Multiview Content

15. 2010/10/13VCLAB, National Tsing Hua University, Taiwan15 Multiview Content Camera capture  Stereo cameras  Camera array Computer generated  Objects is defined  Depth is inherent Conversion from 2D video  Detecting objects  Assigning depth  Filling occluded parts

16. 2010/10/13VCLAB, National Tsing Hua University, Taiwan16 Camera Capture Stanford multi-camera array (128 video cameras) Google street view camera (Image only) Panasonic Full-HD 3D camera

17. 2010/10/13VCLAB, National Tsing Hua University, Taiwan17 2D to 3D Usage of 2D depth information

18. 2010/10/13VCLAB, National Tsing Hua University, Taiwan18 Correction Rectification of misalignment Normalization of colors

19. 2010/10/13VCLAB, National Tsing Hua University, Taiwan19 Multiview Video Coding

20. 2010/10/13VCLAB, National Tsing Hua University, Taiwan20 Representation Formats (Full-Resolution) Full-Resolution Stereo and Multiview  Double data rate for stereo videos  N-fold data rate for N-view videos Efficient compression is the key issue  MVC extension of H.264/AVC

21. 2010/10/13VCLAB, National Tsing Hua University, Taiwan21 Representation Formats (Stereo Interleaving) Stereo Interleaving  A multiplex of the two views into a single sequence Spatial multiplexing  The left and right views are sub-sampled and interleaved into a single frame Temporal multiplexing  The left and right views are interleaved as alternating frames Advantages  Compatible with existing codecs and delivery infrastructure Drawbacks  Loss of spatial or temporal resolution R L R L

22. 2010/10/13VCLAB, National Tsing Hua University, Taiwan22 Representation Formats (Depth- based) Depth-based Format  2D video + depth map Advantages  Backward compatibility with the older coding standards  Supporting both stereo and multiview displays  Depth is adjustable Drawbacks  Limited depth range  Occlusions  Other views have to be synthesized 2D view Synthesized view

23. 2010/10/13VCLAB, National Tsing Hua University, Taiwan23 Multiview Video Coding Single view Multi-view  Independent encoding Low coding complexity but also low coding efficiency  Inter-view prediction Exploiting both spatial and temporal redundancy IPBBBBBBP IPBBBBBBP IPBBBBBBP

24. 2010/10/13VCLAB, National Tsing Hua University, Taiwan24 Multiview Video Coding Multiview camera P B B B P B B B I P B B B P B B B B P B B B P B B B P P B B B P B B B I P B B B P B B B B P B B B P B B B P P B B B P B B B I P B B B P B B B B P B B B P B B B P B IBP P PB P P

25. 2010/10/13VCLAB, National Tsing Hua University, Taiwan25 View Generation View 1View 2synthesis

26. 2010/10/13VCLAB, National Tsing Hua University, Taiwan26 View Generation View synthesis  Disparity-based e.g. view1  view3/2  view2 object(0,0)  object(3,0)  object(6,0)  Depth-based 2D + depth map http://www.imec.be/ScientificReport/SR2007/html/1384302.html DEPTH MAPS EXTRACTION FROM MULTI-VIEW VIDEOS http://www.youtube.com/watch?v=KtRSbey1sKM

27. 2010/10/13VCLAB, National Tsing Hua University, Taiwan27 View Generation

28. 2010/10/13VCLAB, National Tsing Hua University, Taiwan28 Display (3D space) Volumetric video Holographic video http://www.youtube.com/watch?v=kIDgC2no1uo

29. 2010/10/13VCLAB, National Tsing Hua University, Taiwan29 Display Free viewpoint video   http://www.youtube.com/watch?v=vyhz8KgW49 E http://www.youtube.com/watch?v=vyhz8KgW49 E  http://www.youtube.com/watch?v=GSumx0Zs2 XA http://www.youtube.com/watch?v=GSumx0Zs2 XA

30. 2010/10/13VCLAB, National Tsing Hua University, Taiwan30 Display with 3D Glasses Stereo video with glasses  Left view and right view Passive and active

31. 2010/10/13VCLAB, National Tsing Hua University, Taiwan31 Display with 3D Glasses Anaglyph, polarized, and shutter glasses http://www2.ciw.com.cn/h/2562/357866-17902.html

32. 2010/10/13VCLAB, National Tsing Hua University, Taiwan32 Display Lenticular panel and Parallax barrier

33. 2010/10/13VCLAB, National Tsing Hua University, Taiwan33 Display Stereo view display without glasses A1 A2 Pixel 2 Pixel 1

34. 2010/10/13VCLAB, National Tsing Hua University, Taiwan34 Display Multiview display without glasses A2 A1 B1 C1 B2 C2 Pixel 2 Pixel 1

35. 2010/10/13VCLAB, National Tsing Hua University, Taiwan35 Conclusions An overview of multiview video from capture to display. Issues  Display Reduction of brightness Reduction of refresh rate Reduction of temporal of spatial resolution  Coding efficiency  Decoding delay  2D  3D  View synthesis  …

36. 2010/10/13VCLAB, National Tsing Hua University, Taiwan36 Reference A. Vetro, “Representation and Coding Formats for Stereo and Multiview Video,” Tech. Rep. TR2010-011, April, 2010. S. Gaël, “Depth Map Estimation and Use for 3DTV,” Tech. Rep. n  0379, February, 2010. Y.-S. Ho and K.-J. Oh, “Overview of Multi-view Video Coding,” IWSSIP, 2007. 許精益 and 黃乙白 “3D 立體顯示技術之發展與研 究,” 光學工程第 98 期, 96 年 6 月.