1. Overview of the Stereo and Multiview Video Coding Extensions of the H
Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard
By Anthony Vetro, Fellow IEEE, Thomas Wiegand, Fellow IEEE, and
Gary J. Sullivan, Fellow IEEE
PROCEEDINGS OF THE IEEE 2011

2. The Emerging MVC Standard for 3D Video Services
Ying Chen, Ye-KuiWang, Kemal Ugur, MiskaM. Hannuksela,
Jani Lainema, andMoncef Gabbouj
EURASIP Journal on Advances in Signal Processing 2009

3. Outline Introduction Multiview Scenarios and Applications
Standardization Requirements
H.264/MPEG-4 AVC Basics
Extending H.264/MPEG-4 AVC for Multiview
Frame-Compatible Stereo Encoding Formats
Conclusion and Future Work

4. Introduction
Joint Video Team of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) standardize an extension of H.264/MPEG-4 that is referred to as Multiview Video Coding (MVC)
MVC provides a compact representation for multiple views of a video scene, stereo-paired video for 3-D viewing
The stereo high profile of the MVC extension was selected by the Blu-Ray Disc Association as the coding format for 3-D video with high-definition resolution
The system level integration of MVC is more challenging as the decoder output may contain more than one view and can consist of any combination of the views with any temporal level
Various “frame compatible” approaches for support of stereo-view video as an alternative to MVC are discussed

5. Multiview Scenarios and Applications

6. Multiview Scenarios and Applications
Free-viewpoint video, the viewpoint can be interactively changed
There exist several candidate views for the viewer, one of them is selected as the target view
Decoder focus on decoding target view
Efficient switching between different view
3-D TV, more than one view is decoded and display simultaneously
Stereoscopic video
Classic stereo systems that require special-purpose glasses
Auto-stereoscopic displays that do not require glasses
3-D video
Multiple actual or rendered views of the scene are presented to the viewer, e.g. using ‘virtual reality’ glasses or an advanced auto-stereoscopic display, so that view changes with head movements and the viewer has the feeling of immersion in the 3-D
Parallel processing of different views and flexible stream adaption

7. Multiview Scenarios and Applications
Teleconference applications
Both interactivity and virtual reality
Rendering of 3-D TV content or view synthesis
Depth information is needed
2-D TV or HDTV application are still dominating the market
MVC content should provide a way for those 2-D decoders to generate a display from an MVC bitstream

8. Standardization Requirements
High compression efficiency
Huge amount of data in MVC
Enable Inter-view prediction
Efficient memory management of decoded pictures
Hierarchical temporal scalability was found to be efficient for MVC
Significant gain compared to independent compression of each view
Random access
Ensure that any image can be accessed, decoded, and displayed by starting the decoder at a random access point and decoding a relatively small quantity of data on which that image may depend
Insertion intra coded pictures
View-switching random access

9. Standardization Requirements
Typical MVC prediction structure
IDR
anchor
back

10. Standardization Requirements
Scalability
The ability of a decoder to access only a portion of a bitstream still being able to generate effective video output – reduce temporal or spatial resolution
Temporal scalability and View-scalability
Adaption of user preference, network bandwidth, decoder complexity
Decoder resource consumption
A number of views are to be decoded and display
Optimal decoder in terms of memory and complexity is very important to make real-time decoding possible
Parallel processing
In 3-D TV, multiple views need to be decoded simultaneously
Reduce computation time to achieve real-time decoding

11. Standardization Requirements
Backward compatibility
A subset of the MVC bitstream corresponding to “base view” needs to be decodable by an ordinary H.264/MPEG-4 AVC decoder, and other data representing other views should be encoded in a way that will not affect base view decoding
Achieving a desired degree quality consistency among views or to select a preferential quality for encoding some views versus others
Convey camera parameters along with the bitstream in order to support intermediate view interpolation at the decoder
MVC share some design principles with SVC, such as backward compatibility with H.264/AVC, temporal scalability, network friendly adaption…
New mechanisms include view scalability, interview prediction structure, coexisting of decoded pictures from multiple dimensions in the decoded picture buffer, multiple representation in the display, parallel decoding at decoder

12. H.264/MPEG-4 AVC BASICS H.264/AVC covers
Video coding layer (VCL) : creates a coded representation of the source content
Network abstraction layer (NAL) : formats these data and provides header information
Network Abstraction Layer (NAL)
A coded H.264/MPEG-4 AVC video data stream is organized into NAL units, which are packets that contain an integer number of bytes
VCL NAL units
Picture content (coded slices or slice data)
Non-VCL NAL units
Parameter sets
Sequence Parameter Set (SPS) : sequence-level header information
Picture Parameter Set (PPS) : infrequently changing picture-level header information
SEI messages
Do not affect the core decoding process
Assist the decoding process or subsequent processing (bitstream manipulation or display)
Type
Payload

13. H.264/MPEG-4 AVC BASICS

14. H.264/MPEG-4 AVC Basics Video coding Layer (VCL)
The set of consecutive NAL units associated with a single coded picture is referred to as an access unit
A set of consecutive access unit with certain properties is referred to as a coded video sequence
A coded video sequence represents an independently decodable part of a video stream and always starts with an instantaneous decoding refresh (IDR) access unit
Video coding Layer (VCL)
Reference picture buffering and the associated buffering memory control
The behavior of the decoded picture buffer (DBP) can be adaptively controlled by memory management control operation (MMCO) commands
The reference picture lists that are used for coding of P or B slices can be arbitrarily constructed from the pictures available in the DPB via reference picture list modification (RPLM) commands

15. Extending H.264/MPEG-4 AVC for Multiview
Bitstream Structure
Compress multiview stream to include a “base view” bitstream, which is coded independently from all other views in a manner compatible with decoders for single-view profile of the standard
There are useful properties of the coded pictures in the H.264/AVC-compliant base view, such as temporal level, which are not indicated in the VCL NAL units of H.264/AVC. To indicate those properties, the prefix NAL unit has been introduced
Coded pictures from different views may use different SPS that contain the view dependency information for inter-view prediction
New syntax elements include
view_id : identifier of each view
temporal_id : temporal scalability hierarchy
priority_id : used for the simple one-path bitstream adaption process
anchor_pic_flag : indicate a picture is an anchor picture or non-anchor picture
idr_flag : indicate a picture is IDR picture or not
inter_view_flag : indicate a decoded picture is used for inter-view reference or not

16. Extending H.264/MPEG-4 AVC for Multiview
Extension NAL unit type
back

17. Extending H.264/MPEG-4 AVC for Multiview
Enabling Inter-View Prediction
Exploit both temporal and spatial redundancy
The flexible reference picture management capabilities that had already been designed into H.264/MPEG-4 AVC
Making the decoded pictures from other views available in the reference picture lists for use by the inter-picture prediction processing
MVC design does not allow the prediction of a picture in one view at a given time using a picture from another view at different time
Inter-view prediction may be used for encoding the non-base view of an IDR picture

18. Extending H.264/MPEG-4 AVC for Multiview
Additional picture type: anchor picture
Similar to IDR that they do not use temporal prediction , do allow inter-view prediction
It is prohibited for any picture that follows anchor picture to use any picture that precedes the anchor picture as a reference for inter-picture prediction
Provides a clean random access point for access to a given view
High-Level Syntax
Three important pieces of information are carried in the SPS extension
View identification
Total number of views
A listing of view identifiers (ex: 0-2-1)
View dependency information
The number of inter-view reference pictures for list0/list1
The views that may be used for predicting a particular view (ex: view1 use view0 and view2 as reference views)
Separate for anchor and non-anchor pictures

19. Extending H.264/MPEG-4 AVC for Multiview
Level index for operation point
Indicator of the resource requirements for a decoder that conforms to a particular level
A specific temporal subset and a set of views including those intended for output and the views that they depend
Multiple level values could be signaled as part of the SPS extension, with each level being associated with a particular operating point
The syntax indicates the number of views that are targeted for output as well as the number of views that would be required for decoding particular operating points
Profiles and Levels
Profiles
Determine the subset of coding tools that must be supported by conforming decoders
Based on the high profile of H.264/MPEG-4 AVC
Multiview high profile
Supports multiple views
Does not support interlace coding tools
Stereo high profile
Two views
Support interlace coding tools

20. Extending H.264/MPEG-4 AVC for Multiview

21. Extending H.264/MPEG-4 AVC for Multiview
Levels
Constrains on the bitstreams produced by MVC encoders, to establish bounds on the necessary decoder resources and complexity
Limit on the amount of frame memory required for the decoding of a bitstream
The maximum throughput in terms of macroblocks per second
Maximum picture size
Overall bitrate
Coding Performance
20%-30% bitrate saving
2-3 dB gains

22. Extending H.264/MPEG-4 AVC for Multiview
SEI Message for Multiview Video
Parallel decoding information SEI message
MVC scalable nesting SEI message
Indicate the scope of views or temporal levels for which the message apply
Reuse the syntax of H.264/AVC SEI messages for a specific set of views and temporal level
View scalability information SEI message
The mapping between each operation point and the required NAL units
Signal profile and level for each operation point which is identified by the view_id
Multiview scene information SEI message Multiview acquisition information SEI message
Signal camera parameters, which are helpful in view interpolation by a renderer
Nonrequired view component SEI message
Indicates that a particular view component is not needed for decoding
Operation point : a subset bitstream which identified by the combination of required view_id
and temporal_id values.

23. Extending H.264/MPEG-4 AVC for Multiview
Parallel coding of multiple views
3-D broadcasting use cases, display need to output many views simultaneously to support head-motion parallax
Use Parallel decoding information SEI message indicates the video is encoded in a way that macroblock in view 1 picture could only use reconstruction values of macroblocks that belong to certain rows in view 0 picture

24. Extending H.264/MPEG-4 AVC for Multiview
View dependency change SEI message
Signal changes in the view dependency structure
Operation point not present SEI message
Indicates operation points that are not present in the bitstream
Useful in streaming and networking scenarios
Base view temporal HRD SEI message
Associated with an IDR access unit
Signal information relevant to the hypothetical reference decoder (HRD) parameters associated with the base view
HRD: a virtual buffering algorithm that can be used to test the behavior of the coded bitstream and its effect on a real decoder

25. Frame-Compatible Stereo Encoding Formats
Frame-compatible formats refer to a class of stereo video formats in which the two stereo views are essentially multiplexed into a single coded frame or sequence of frames
Other common names include stereo interleaving or spatial/temporal multiplexing formats

26. Frame-Compatible Stereo Encoding Formats
Basic Principles
With a frame-compatible format, the left and right views are packed together in the samples of a single video frame
Half of the coded samples represent the left view and other half represent the right view
Each coded view has half the resolution of the full coded frame
Temporal multiplexing
The left and right views would be interleaved as alternating frames or field of a coded sequence
Frame-compatible formats have received considerable attention from the broadcast industry since the coded video can be processed by encoders and decoders that were not specially designed to handle stereo video
Only the final display stage requires some customization for recognizing and properly rendering the video to enable a 3-D viewing experience

27. Frame-Compatible Stereo Encoding Formats
The drawback of representing the stereo signal in this way is that spatial or temporal resolution would be only half of that used for 2-D video with the same encoded resolution
Key additional issue with frame-compatible formats is distinguishing the left and right views
Signaling
The signaling for a complete set of frame-compatible formats has been standardized within the H.264/MPEG-4 AVC standard as SEI messages
Frame packing arrangement (FPA) SEI message was specified in an amendment of the H.264/MPEG-4 AVC standard

28. Conclusion and Future Work
Three-dimensional video has drawn significant attention recently. The efficient representation and compression of stereo and multiview video is a central component of any 3-D or multiview
This paper reviewed the recent extensions H.264/MPEG-4 AVC standard that support 3-D stereo and multiview video
The MVC standard includes stereo and multiview video by enabling inter-view prediction as well as temporal interpicture prediction
Another important development has been the efficient representation, coding, and signaling of frame-compatible stereo video formats
As the market evolves and new types of displays and services are offered, additional new technologies and standards will need to be introduced.
The generation of the large number of views required by autostereoscopic displays would be needed.
Solutions that consider the inclusion of depth map information for this purpose are a significant area of focus for future designs