1. Concepts of Multimedia Processing and Transmission
GMU - IT 481, Spring 2006
1/23/2006
Concepts of Multimedia Processing and Transmission
IT 481, Lecture #7
Dennis McCaughey, Ph.D.
23 October, 2006
(C) Hung Nguyen, 2006

2. Broadcast Environment
IT 481, Fall 2006
08/28/2006

3. H.264 Overview 1997: ITU-T Video Coding Experts Group began work
2001: ISO/MPEG joined the ITU-T and formed a Joint Video Team (JVT) that took over the H.264 project
The JVT objective was the creation of a single video coding standard that would simultaneously result in a new part of the MPEG-4 family of standards and a new ITU-T (H.264) Recommendation
IT 481, Fall 2006
08/28/2006

4. History
IT 481, Fall 2006
08/28/2006

5. H.264 Advantages
Up to 50% savings in bit rate: Compared to H.263+ or MPEG-4 Simple Profile
High quality video: H.264 offers consistently good video quality at high and low bit rates.
Error resilience: H.264 provides the tools necessary to deal with packet loss in packet networks and bit errors in error-prone wireless networks.
Network friendliness: Through the Network Adaptation Layer, H.264 bit streams can be easily transported over different networks
IT 481, Fall 2006
08/28/2006

6. Relationship to Other Standards
Identical specifications have been approved in both ITU-T / VCEG and ISO/IEC / MPEG
In ITU-T / VCEG this is a new & separate standard
ITU-T Recommendation H.264
ITU-T Systems (H.32x) will be modified to support it ! In
ISO/IEC / MPEG this is a new “part” in the MPEG-4 suite
Separate codec design from prior MPEG-4 visual
New part 10 called “Advanced Video Coding” (AVC – similar to “AAC” position in MPEG-2 as separate codec)
MPEG-4 Systems / File Format has been modified to support it
H | MPEG-2 Systems also modified to support it
IETF finalizing RTP payload packetization
IT 481, Fall 2006
08/28/2006

7. Applications Entertainment Video (1-8+ Mbps, higher latency)
Broadcast / Satellite / Cable / DVD / VoD / FS-VDSL / …
DVB/ATSC/SCTE, DVD Forum, DSL Forum
Conversational Services (usu. <1Mbps, low latency)
Circuit Switched
H.320 Conversational H324/M
3GPP Conversational H.324/M
Packet-switched
H.323 Conversational Internet/best effort IP/RTP
3GPP Conversational IP/RTP/SIP
Streaming Services (usu. lower bit rate, higher latency)
3GPP Streaming IP/RTP/RTSP
Streaming IP/RTP/RTSP (without TCP fallback)
Other Services
3GPP Multimedia Messaging Services
IT 481, Fall 2006
08/28/2006

8. Profiles and Levels
Many standards contain different configurations of capabilities – often based in “profiles” & “levels”
A profile is usually a set of algorithmic features
A level is usually a degree of capability (e.g. resolution or speed of decoding)
H.264/AVC has three profiles
Baseline (lower capability plus error resilience, e.g., videoconferencing, mobile video)
Main (high compression quality, e.g., broadcast)
Extended (added features for efficient streaming)
IT 481, Fall 2006
08/28/2006

9. Grouping of Capabilities into Profiles
Three profiles now: Baseline, Main, and Extended
Baseline (e.g., Videoconferencing & Wireless)
I and P picture types (not B)
In-loop deblocking filter
1/4-sample motion compensation
Tree-structured motion segmentation down to 4x4 block size
VLC-based entropy coding (CAVLC)
Some enhanced error resilience features
Flexible macroblock ordering/arbitrary slice ordering
Redundant slices
Note: No support for interlaced video in Baseline
IT 481, Fall 2006
08/28/2006

10. Non-Baseline Profiles
Main Profile (esp. Broadcast/Entertainment)
All Baseline features except enhanced error resilience features
B pictures
Adaptive weighting for B and P picture prediction
Picture and MB-level frame/field switching
CABAC
Note: Main is not exactly a superset of Baseline
Extended Profile (esp. Streaming/Internet)
All Baseline features
More error resilience: Data partitioning
SP/SI switching pictures
Note: Extended is a superset of Baseline (but not of Main)
IT 481, Fall 2006
08/28/2006

11. H.264 Encoder
IT 481, Fall 2006
08/28/2006

12. H.264 Main Stages
Dividing each video frame into blocks of pixels so that processing of the video frame can be conducted at the block level
Exploiting the spatial redundancies that exist within the video frame by coding some of the original blocks through transform, quantization and entropy coding.
Exploiting the temporal dependencies that exist between blocks in successive frames so that only changes between successive frames need to be encoded.
For any given block, a search is performed in the previously coded one or more frames to determine motion vectors that are then used by the encoder and decoder to predict the subject block
Exploiting any remaining spatial redundancies that exist within the video frame by coding residual blocks; i.e. the difference between the original blocks and the corresponding predicted blocks again through transform, quantization and entropy coding.
IT 481, Fall 2006
08/28/2006

13. H.264 Features Enhanced Motion compensation
Multiple block sizes and shapes (MPEG-2 has 0nly 8x8)
Higher resolution ¼ pixel motion estimation
Multiple reference frame selection and bi-directional mode selection
Employs an integer based DCT that does not have the mismatch problem in the inverse transform
Improved entropy coding
IT 481, Fall 2006
08/28/2006

14. Intra Prediction & Coding
Intra coding refers to the case where only spatial redundancies within a video picture are exploited.
The resulting frame is referred to as an I-picture and are typically encoded by directly applying the transform to the different macroblocks in the frame.
Encoded I-pictures are large in size since a large amount of information is usually present in the frame, and no temporal information is used as part of the encoding process.
In order to increase the efficiency of the intra coding process in H.264, spatial correlation between adjacent macroblocks in a given frame is exploited.
Based on the observation that adjacent macroblocks tend to have similar properties.
First step in the encoding process for a given macroblock, is the prediction of the macroblock of interest from the surrounding macroblocks (typically the ones located on top and to the left of the macroblock of interest, since those macroblocks would have already been encoded).
The difference between the actual macroblock and its prediction is then coded, which results in fewer bits to represent the macroblock of interest
IT 481, Fall 2006
08/28/2006

15. H.264 4x4 Intra Prediction Modes
H.264 offers 9 modes for prediction of 4x4 luminance blocks, including DC prediction (Mode 2) and 8 directional modes, labeled 0 thru 8 in Figure 4. This process is illustrated above in which pixels A to M from neighboring blocks have already been encoded and may be used for prediction
IT 481, Fall 2006
08/28/2006

16. Examples Mode 0 (Vertical Prediction) Mode 3 (Diagonal-Down-Left)
a, e, i and m are equal to A,
b, f, j and n are equal to B,
c, g, k and o are equal to C, and
d, h, l and p are equal to D.
Mode 3 (Diagonal-Down-Left)
a is equal to (A+2B+C+2)/4,
b, e are equal to (B+2C+D+2)/4,
c, f, i are equal to (C+2D+E+2)/4,
d, g, j, m are equal to (D+2E+F+2)/4,
h, k, n are equal to (E+2F+G+2)/4,
l, o are equal to (F+2G+H+2)/4, and
p is equal to (G+3H+2)/4.
IT 481, Fall 2006
08/28/2006

17. Other Intra Prediction Modes
8x8 Intra Prediction Modes
Luminance Blocks
Uses all nine prediction modes
Chrominance Blocks
Uses four prediction modes (DC, Vertical, Horizontal and Planar).
16x16 Intra Prediction Modes
For regions with less spatial detail (i.e., flat regions),
Uses four prediction modes (DC, Vertical, Horizontal and Planar) is chosen for the prediction of the entire luminance component of the macroblock
The prediction mode must be encoded for each block.
The mode for each block is efficiently coded by assigning shorter symbols to more likely modes,
The probability of each mode is determined based on the modes used for coding the surrounding blocks
IT 481, Fall 2006
08/28/2006

18. Inter Prediction and Coding
Inter prediction and coding is based on using motion estimation and compensation to take advantage of the temporal redundancies that exist between successive frames, hence.
Motion estimation in H.264 supports most of the key features adopted in earlier video standards, but with improved efficiency.
Supports P-pictures (with single and multiple reference frames) and B-pictures, and a new inter-stream transitional picture called an SP-picture.
The inclusion of SP-pictures in a bit stream enables efficient switching between bit streams with similar content encoded at different bit rates, as well as random access and fast playback modes.
Four main motion estimation features used in H.264:
(1) the use of various block sizes and shapes,
(2) the use of high-precision sub-pixel motion vectors,
(3) the use of multiple reference frames, and
(4) the use of de-blocking filters in the prediction loop
IT 481, Fall 2006
08/28/2006

19. Block Sizes
The availability of smaller motion compensation blocks improves prediction. The small blocks improve the ability of the model to handle fine motion detail and result in better subjective viewing quality by not large blocking artifacts
IT 481, Fall 2006
08/28/2006

20. Tree Structure
H.264 allows a combination of 4x8, 8x4, or 4x4 sub-blocks within an 8x8 sub-block as shown for a 16x16 macroblock.
Zig-zag scan pattern
IT 481, Fall 2006
08/28/2006

21. Motion Estimation Accuracy
The prediction capability of the motion compensation algorithm in H.2 64 is further improved by allowing motion vectors to be determined with higher levels of spatial accuracy than in existing standards.
Quarter-pixel accurate motion compensation is the lowest-accuracy form of motion compensation in H.264
In contrast with prior standards based primarily on half-pixel accuracy, with quarter-pixel accuracy only available in the newest version of MPEG-4.
¼-pixel spatial accuracy can yield as much as 20% in bit rate savings as compared to using integer-pixel spatial accuracy.
IT 481, Fall 2006
08/28/2006

22. Multiple Reference Picture Selection
The H.2 64 standard offers the option of having multiple reference frames in inter picture coding,
Results in better subjective video quality and more efficient coding of the video frame under consideration.
Multiple reference frames might help making the H.264 bit stream error resilient.
There would be additional processing delays and higher memory requirements at both the encoder and decoder.
Using 5 reference frames for prediction can yield 5-10% in bit rate savings as compared to using only one reference frame.
IT 481, Fall 2006
08/28/2006

23. De-Blocking (Loop) Filter
H.264 specifies the use of an adaptive de-blocking filter that operates on the horizontal and vertical block edges within the prediction loop
Removes artifacts caused by block prediction errors. The filtering is generally based on 4x4 block boundaries, in which two pixels on either side of the boundary may be updated using a different filter.
The rules for applying the de-blocking filter are intricate and quite complex,
Its use is optional for each slice (loosely defined as an integer number of macroblocks).
The improvement in subjective quality often more than justifies the increase in complexity.
The de-blocking filter yields a substantial improvement in subjective quality.
IT 481, Fall 2006
08/28/2006

24. IT 481, Fall 2006
08/28/2006

25. Integer Transform
Prediction error blocks resulting from either intra prediction or inter prediction are then transformed using a new integer DCT.
H.264 is unique in that it employs a purely integer spatial transform
An approximation of the DCT as opposed to the usual floating-point DCT specified with rounding-error tolerances as used in earlier standards.
H.264 allows the use of both 4x4 and 8x8 transform block sizes.
The small shape helps reduce blocking and ringing artifacts,
The precise integer specification eliminates any mismatch issues between the encoder and decoder in the inverse transform.
IT 481, Fall 2006
08/28/2006

26. Quantization & transform Coefficient Scanning
The quantization step provides a significant portion of the data compression.
In H.264, the transform coefficients are quantized using scalar quantization.
Fifty-two different quantization step sizes can be chosen on a macroblock basis
Prior standards (H.263 supports thirty-one, for example).
In H.264 the step sizes are increased at a compounding rate of approximately 12.5%, rather than by a constant increment.
The fidelity of chrominance components is improved by using finer quantization step sizes as compared to those used for the luminance.
IT 481, Fall 2006
08/28/2006

27. Entropy Coding
Entropy coding is based on assigning shorter code words to symbols with higher probabilities of occurrence, and longer codewords to symbols with less frequent occurrences.
Parameters to be entropy coded include:
Transform coefficients for the residual data,
Motion vectors and
Other encoder information.
Two types of entropy coding have been adopted:
Variable-Length coding (VLC)
Context-Based Adaptive Binary Arithmetic Coding (CABAC).
IT 481, Fall 2006
08/28/2006

28. Variable Length Encoding
In some video coding standards, symbols and the associated codewords are organized in look-up tables, referred to as VLC tables, which are stored at both the encoder and decoder.
In H.263, a number of VLC tables are used, depending on the type of data under consideration (e.g., transform coefficients, motion vectors).
H.264 offers a single Universal VLC (UVLC) table that is to be used in entropy coding of all symbols in the encoder except for the transform coefficients.
Simple,
Disadvantage, in that a single table is usually derived using a static probability distribution model, which ignores the correlations between the encoder symbols
IT 481, Fall 2006
08/28/2006

29. VLC-Transform Coefficients
In H.264, the transform coefficients are coded using Context Adaptive Variable Length Coding (CAVLC).
CAVLC is designed to take advantage of several characteristics of quantized 4x4 blocks.
First, non-zero coefficients at the end of the zigzag scan are often equal to +/- 1. CAVLC encodes the number of these coefficients (“trailing 1s”) in a compact way.
Second, CAVLC employs run-level coding efficiently to represent the string of zeros in a quantized 4x4 block.
Moreover, the numbers of non-zero coefficients in neighboring blocks are usually correlated.
The number of non-zero coefficients is encoded using a look-up table that depends on the numbers of non-zero coefficients in neighboring blocks.
Finally, the magnitude (level) of non-zero coefficients gets larger near the DC coefficient and get smaller around the high frequency coefficients.
CAVLC takes advantage of this by making the choice of the VLC look-up table for the level adaptive in a way where the choice depends on the recently coded levels
IT 481, Fall 2006
08/28/2006

30. Context-Based Adaptive Binary Arithmetic Coding (CABAC)
Arithmetic coding makes use of a probability model at both the encoder and decoder for all the syntax elements
Transform coefficients
Motion vectors.
A process called context modeling increases the coding efficiency of arithmetic coding,
the underlying probability model is adapted to the changing statistics with a video frame
Context modeling provides estimates of conditional probabilities of the coding symbols.
Suitable context models, allow inter-symbol redundancy to be exploited
Switching between different probability models according to already coded symbols in the neighborhood of the current symbol to encode.
IT 481, Fall 2006
08/28/2006

31. Context Models
Different models are often maintained for each syntax element (e.g., motion vectors and transform coefficients have different models).
If a given symbol is non-binary valued, it will be mapped onto a sequence of binary decisions, so-called bins.
The actual binarization is done according to a given binary tree –
The UVLC binary tree is often used.
Each binary decision is then encoded with the arithmetic encoder using the new probability estimates, which have been updated during the previous context modeling stage.
After encoding of each bin, we adjust upward the probability estimate for the binary symbol that was just encoded.
Hence, the model keeps track of the actual statistics
IT 481, Fall 2006
08/28/2006

32. Example
IT 481, Fall 2006
08/28/2006

33. Reference
“Emerging H.264 Standard: Overview and TMS320C64x Digital Media Platform Implementation” UB Video Inc White Paper, 2002
IT 481, Fall 2006
08/28/2006