3D EXTENSION of HEVC: Multi-View plus Depth Parashar Nayana Karunakar Student Id: Department of Electrical Engineering

High-Efficiency Video Coding (HEVC) is the newest video coding standard of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). [1][14] The Joint Collaborative Team on 3D Video Coding Extension Development was created to develop 3D video coding technology more advanced than the current multiview video coding (MVC) features of H.264. The standards for which these 3D video coding extension technologies will provide such enhanced capabilities may include H.262, H.264 and the High Efficiency Video Coding (HEVC).[12]

HEVC – ENCODER BLOCK DIAGRAM Encoder block diagram H.264 [18] Fig 1. – Typical HEVC Encoder[1]

What is 3D/Multi-View/Stereo Video Multiple Cameras views of the same scene are captured – Multiple View Video(MVV) Efficient Compression techniques are essential as MVV contains vast amount of data both during storage and transmission. Inter-view statistical dependencies are exploited for combined temporal/inter-view prediction. When color video and an associated per sample depth map are considered, we get Multi-view video plus depth representation. Fig 2. – Test Sequence –Balloons with depth map [3]

Fig 3. – Overview of the system structure and data format for the transmission of 3D video [12]

Encoding and decoding of each view of a multi-view data set separately,referred to as simulcast coding, can be done with any video codec including H.264/AVC and HEVC. This would be simple but inefficient as inter-view statistical dependencies are not exploited. In order to exploit all the statistical dependencies within multi-view dataset, inter-view prediction has to be combined with temporal prediction. As seen in fig 3b., In MVC, one of the views is conventionally coded in conformance to the HEVC codec. For coding the other views, in addition to previously coded pictures of the same view already coded co-located pictures of other views can also be used as reference pictures. In Multi –view video plus depth (MVD) format, only a few views are actually coded. Based on the transmitted videos and depth maps, additional views can be rendered.. Fig 4a. – Simulcast coding structure with hierarchical B pictures for temporal prediction(black arrows)[2] Fig 4b. – Multi-view coding structure with hierarchical B pictures for both temporal (black arrows) and inter-view prediction(red arrows) [2]

Fig 5. – Block Diagram of a 3D Video Codec[4]

The basic structure of the 3D video codec[11] is shown in the block diagram of Figure 4. In principle, each component signal is coded using an HEVC-based codec[11]. The resulting bit stream packets, or more accurately, the resulting Network Abstraction Layer (NAL) units, are multiplexed to form the 3D video bit stream. The base or independent view is coded using an unmodified HEVC codec. The base view sub-stream can be directly decoded using the conventional HEVC decoder. For coding the dependent views and the depth data, modified HEVC codec are used, which are extended by including additional coding tools and inter-component prediction techniques that employ already coded data inside the same access unit as indicated by the red arrows in Figure 4 For enabling an optional discarding of depth data from the bit stream, e.g., for supporting the decoding of a stereo video suitable for conventional stereo displays, the inter-component prediction can be configured in a way that video pictures can be decoded independently of the depth data..

The video pictures and, when present, the depth maps are coded access unit by access unit, as it is illustrated in Figure 6. An access unit includes all video pictures and depth maps that correspond to the same time instant. NAL units containing camera parameters may be additionally associated with an access unit. The video pictures and depth maps corresponding to a particular camera position are indicated by a view identifier (viewId). All video pictures and depth maps that belong to the same camera position are associated with the same value of viewId. Inside an access unit, the video picture and, when present, the associated depth map with viewId equal to 0 are coded first, followed by the video picture and depth map with viewId equal to 1, etc. For ordering the reconstructed video pictures and depth map after decoding, each value of viewId is associated with another identifier called view order index (VOI). The view order index is a signed integer values, which specifies the ordering of the coded views from left to right. Fig 6. - Access units structure and coding order of view components[12]

CODING OF DEPENDENT VIEWS -- Additional tools have been integrated into the HEVC codec, which employ already coded data in other views for efficiently representing a dependent view. These tools include - Disparity-compensated prediction, View synthesis based inter-view prediction, Post processing in-loop filtering, Inter-view motion prediction, Depth-based motion parameter prediction, Inter-view residual prediction, Adjustment of QP of texture based on depth data. CODING OF DEPTH MAPS – There are certain additional tools and also some tools are removed for coding of Depth maps. Some of the differences are -- Depth Maps are coded in 4:0:0 format, Non-linear depth representation is used, Z-near Z-far compensated weighted prediction, Modified motion compensation and motion vector coding ( No interpolation is used i.e. for depth maps, the inter-picture prediction is always performed with full-sample accuracy. Disabling of in-loop filtering ( deblocking filter and SAO), Depth modeling modes ( Four new Intra-prediction modes are used), Motion parameter inheritance.

The aim of the project is to explore and implement 3D extension of HEVC. The project will be carried out in the following steps: Study the MVC used for H.264/AVC [13] and use it as a reference to learn about MVD extension for HEVC.[1][14] Learn about the tools that are different to MVD [11]when compared to standard HEVC codec. Study and compare the MVD extensions of HEVC carried out on HM 5.1[11] by Fraunhoffer HHI [11]and Qualcomm [11] (the two most recent ones). Tabulate the results in terms of PSNR and bit-rate differences. Explore the challenges that might be present while porting the 3D extensions to the latest HM 9.1 reference software [13]and implement it.

LIST OF ACRONYMS: AVC: Advanced video coding DIBR: Depth Image Base Rendering HD: High Definition HHI: Heinrich Hertz Institute HM: HEVC Test Model IEC: International Electrotechnical Commission ISO: International Organization for Standardization ITU-T: International Telecommunication Union-Telecommunication MPEG: Moving picture experts group MC: Motion Compensation MV: Motion Vector MVC: Multi-View Coding MVD: Multi-View plus Depth MVV: Multi-View Video NAL: Network Abstraction Layer PSNR: Peak Signal to Noise Ratio SAO: Sample Adaptive Offset VCEG: Video coding experts group VOI: View Order Index

REFERENCES [1] G.J. Sullivan; J. Ohm; Woo-Jin Han and T.Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Transactions on Circuits and Systems for Video Technology, Volume: 22, Issue: 12, Pages , December 2012.Overview of the High Efficiency Video Coding (HEVC) Standard [2] P. Merkle, A Smolic, K. Müller, and T. Wiegand, “Multi-View video plus depth data representation and coding”. Picture Coding Symposium, 2007 [3] Test Sequences: [4] H. Schwarz etal "3D Video Coding Using Advanced Prediction, Depth Modeling, and Encoder Control Methods", Picture Coding Symposium, May [5] G. Tech, H. Schwarz, K. Müller, and T. Wiegand, "Effects of synthesized View Distortion based 3D Video Coding on the Quality of interpolated and extrapolated Views", IEEE Intl. Conf. on Multimedia and Exposition, Pages , July [6] P. Merkle etal, "3D Video: Depth Coding Based on Inter-component Prediction of Block Partitions", Picture Coding Symposium, May [7] H. Schwarz and T. Wiegand, "Inter-View Prediction of Motion Data in Multiview Video Coding", Picture Coding Symposium, May [8] G. Tech, H. Schwarz, K. Müller, and T. Wiegand, "3D Video Coding using the Synthesized View Distortion Change", Picture Coding Symposium, May [9] M. Winken, H. Schwarz, and T. Wiegand, "Motion Vector Inheritance for High Efficiency 3D Video plus Depth Coding," Picture Coding Symposium, May [10] S. Bosse, H. Schwarz, T. Hinz, and T. Wiegand, "Encoder Control for Renderable Regions in High Efficiency Multiview Video Plus Depth Coding", Picture Coding Symposium, May [11] 3D Extension Software Repository: [12] “Test Model under Consideration for HEVC based 3D video coding”, ISO/IEC JTC1/SC29/WG11 MPEG2011/N12559 February 2012, San Jose, CA, USA [13] HM Software Repository: [14] HEVC Text Specification Draft 9: [15] H.264/AVC reference website -