1. 3D Stereo Video Coding Heejune AHN Embedded Communications Laboratory Seoul National Univ. of Technology Fall 2013 Last updated 2013. 9. 31

2. Heejune AHN: Image and Video Compressionp. 2 Outline Stereo video display Stereo video principle Multi-View Coding H.264 amendment HEVC extension

3. Heejune AHN: Image and Video Compressionp. 3 3D display Stereo (existing) Anaglyphic (passive red-cyan glasses) Polarized (passive polarized glasses) Shutter/Synchronized (alternative frame sequencing) Auto stereographic ( No glasses) Hologram

4. Heejune AHN: Image and Video Compressionp. 4 Anaglyph

5. Heejune AHN: Image and Video Compressionp. 5 Stereo Display Polarization type micro-polarizer (patterned retarder) polarized glass Active retarder ½ ver. resol. full ver. resol.

6. Heejune AHN: Image and Video Compressionp. 6 Active Shutter glasses type 2 times frame rate (i.e. 120 Hz) synchronization cross-talk due to incomplete sync. eye fatigue more expensive glasses than polarized L R

7. Heejune AHN: Image and Video Compressionp. 7 No glasses Lenti-cular - Array of lenslets in front of the pixel raster; relies on refraction of light through lenslets. Parallax barrier - A barrier mask is placed in front of the pixel raster to control the light emitted to certain viewing positions Limitation on distance

8. Heejune AHN: Image and Video Compressionp. 8 Display Industry trends 3D TV

9. Heejune AHN: Image and Video Compressionp. 9 TV market trends

10. Heejune AHN: Image and Video Compressionp. 10 The Principle of Stereo Vision Disparity between two eyes

11. Heejune AHN: Image and Video Compressionp. 11 Human can understands 3D without disparity.

12. Heejune AHN: Image and Video Compressionp. 12 Disparity D is in reverse proportional to depth (Z).

13. Heejune AHN: Image and Video Compressionp. 13 Stereo Video Coding Frame Compatible Stereo Spatial: side-by-side, top-bottom, line/row interleaved Time: views interleaved as alternating frames or fields +: Leverage existing infrastructure (2D coding) - : Reduced-resolution, coding in-efficiency Currently in TV broadcast service Multiview Video Coding (MVC) H.264 amendment (2011) 3D Video and Free Viewpoint Video (FVV) Exploits correlation among views + : Full-resolution Adopted as format for 3D Blue-ray Disc

14. Heejune AHN: Image and Video Compressionp. 14 Frame-compatible Stereo Video Top-bottom, side-by-side, line-row interleaved Example

15. Heejune AHN: Image and Video Compressionp. 15 H.264/MVC statistical dependence in temporal and inter-views temporal, spatial, plus inter-view prediction Backwards compatible to H.264 single view profiles

16. Heejune AHN: Image and Video Compressionp. 16 MVD Coding Multi-View(2D)+Depth Coding HEVC extension Better for auto-stereoscopic with DIBR (Depth Information Based Rendering) Depth maps : distance from the cameras/view points inverse proportional to disparity

17. Heejune AHN: Image and Video Compressionp. 17 Block diagram MV capturing, coding, decoding, and rendering

18. Heejune AHN: Image and Video Compressionp. 18 Coding algorithm 1 base + N-1 dependent views N associate depth maps

19. Heejune AHN: Image and Video Compressionp. 19 Interview Motion prediction time view

20. Heejune AHN: Image and Video Compressionp. 20 Interview residual signal prediction Predict ‘residual’ of a current block using a coded residual block in a reference view Determined disparity using the estimated depth maps (same depth as for inter-view motion parameter prediction) Disparity-compensated prediction of the current residual Bilinear filtering for sub-sample interpolation use after motion compensation residual signal depthmap

21. Heejune AHN: Image and Video Compressionp. 21 Depth map coding Depth map properties 1. Sharp edges representing object borders 2. Large areas of slowly varying values representing object areas 3. Edges in depth maps are correlated with edges in video pictures Depth map coding ideas 1. Representation of depth edges 2. Partition block into two regions with constant sample values 3. Prediction based on co-located texture block 4. Optional transform coding of residual

22. Heejune AHN: Image and Video Compressionp. 22 Non-rectangular block partitions Approximating the signal of a depth block by a model that partitions the area into two non-rectangular regions Each region is represented by a constant value

23. Heejune AHN: Image and Video Compressionp. 23 Motion parameter inheritance Get parameter info for depth map from those from view data