An Introduction to H.264/AVC and 3D Video Coding

Outline Video Coding Concepts  basic concept review  image coding structure  video coding structure H.264/AVC Introduction  history  performance comparison H.264/AVC Coding Tools  inter prediction  intra prediction  transform & quantization  de-blocking filter  entropy coding 3D Video Coding  3D video format  multiview video coding Summary with Q&A

Video Coding Concept -basic concept review -image coding structure -video coding structure

The Scope of Image and Video Coding Standardization Only the Syntax and Decoder are standardized:

Images and Video

Needs for Video Compression Without compression  Visual telephony (e.g. 15 frames/s): 325 (pels) x 288 (lines) x 15 (farmes/s) x 1.5 bytes = Mbit/s  Digital TV (ITU-T 601 frames/s): 720 (pels) x 480 (lines) x 30 (farmes/s) x 1.5 bytes = Mbit/s  HDTV (e.g. 1280x720 pels 60 frames/s): Compression results in lower bit rates  Lower transmission and storage cost

RGB vs. YCbCr [1/3]

RGB vs. YCbCr [2/3]

RGB vs. YCbCr [3/3]

Common YCbCr Formats

Subjective View

Block Based Coding [1/2]

Block Based Coding [2/2]

Group of Picture (GOP)

Video Coding Concept -basic concept review -image coding structure -video coding structure

Image Coding Structure

Transform

Quantization S: (3 bits) Quantization: Quantization step-size Q=2: S/2 Quantization Levels (Q): (2 bits) Inverse quantization (x2): Quantization error: Quantization step-size Q=4: S/4 Quantization Levels (Q): (2 bits) Inverse quantization (x4): Quantization error:

Effect of DCT + Quantization

Entropy coding

Video Coding Concept -basic concept review -image coding structure -video coding structure

Temporal Redundancy [1/2] The amount of data to be coded can be reduced significantly

Standard Video Encoder

Block Based Motion Compensation [1/2]

Algorithms for Motion Estimation Full Search  Guarantee find the global minimum SAD  high computational complexity Fast Search  Local minimum SAD  Low computational complexity  Reduce candidate blocks  Reduce matching pixels in candidate blocks

Diamond Search

Video coding structure

H.264/AVC Introduction -History -Performance comparison

History

Joint Video Team

MPEG-2 Has Hit A Wall

MPEG-4 in Comparison

H.26L Provides Focus

MPEG-4 “Adopts” H.264

State of the Art Standards MPEG-2  DVD, DVT, since 1994 MPEG-4  DVR, Digital Still Camera, since 1999  ~1.5x coding gain over MPEG-2 (ASP) MPEG-4 part 10, AVC (H.264)  Mobile video, DVB-H, Blu-ray Disc and etc.  2~3x coding gain over MPEG-2

AVC Profiles

coding tools and profiles

H.264/AVC Introduction -History -Performance comparison

Compare to Other Standard Fair comparisons of H.26L(TML-8.0) versus H.263v3,MPEG-2,and MPEG-4  TML-8.0 at half of the bit rate as MPEG-4 for the same visual fidelity  Source from VCEG-N18.doc (Soptember,2001) Objective evaluation  Average improvement of TML-8.0over MPEG-2 (VM-5) of 5.8 dB PSNR (peak gain 7.2 dB) for equal bandwidths  TML-8.0 average gain of 3.1 dB relative to H (High-Latency Profile) for equal bandwidths (up to 5.2 dB)  Gain of 2.2 dB over MPEG-4 (Advanced Simple Profile) for equal bandwidths (max. 3.6 dB)

Test Sets “Streaming” Test:  Four QCIF sequences coded at 10 Hz and 15 Hz (Foreman, Container, News, Tempete)  Four CIF sequences coded at 15 Hz and 30 Hz (Bus, Flower, Garden, Mobile and Calendar, and Tempete)  With B frame “Real-Time Conversation” Test:  Four QCIF sequences encoded at 10Hz and 15Hz (Akiyo, Foreman, Mother and Daughter, and Silent Voice)  Four CIF sequences encoded at 15Hz and 30Hz (Carphone, Foreman, Paris, and Sean)  Without B frames

Objective evaluation [1/2]

Objective evaluation [2/2]

Subjective evaluation Example: Sequence Mobile, frame 40

Perceptual Test of H.264/AVC High Profile

Objective Performance of H.264/AVC High Profile

Intra mode performance [1/2]  Average gain of H.264 to JPEG: 5.2 dB (luma)  Average gain of H.264 to JPEG2000: 1.12 dB (luma)  Average gain of Motion JPEG2000 to H.264: 1.42 dB (chroma)  The smaller the bit rate, the higher the gain of H.264

Intra mode performance [2/2]

Intra mode performance [chroma]

Intra mode performance [FRExt] a set of 8 photographic monochrome test images  with resolutions from 512x512 up to 2048x3072 samples Average gain of H.264/AVC HP to JPEG2000: 0.5 dB  over the entire test image set and all bit-rates

JPEG2000 vs. H.264 Intra

H.264/AVC Coding Tools -Inter prediction -Intra prediction -Transform and Quantization -De-blocking Filter -Entropy Coding

Basic Coding Structure

Standard Tools Comparison

Motion Compensation

Macro Block Partitions

Example – Frame 1

Example – Frame 2

Example – Residual [no MC]

Example – Residual [16x16]

Example – Residual [8x8]

Example – Residual [4x4]

Example of Variable Block Sizes Large block means  Less bits for MVs  More bits on residuals Small block means  More bits for MVs  Less bits on residuals

Summary Key Features  Enhances motion compensation  Small blocks for transform coding  De-blocking filter 50% bit rate saving against any other standards, by  Better prediction  More computation  More memory Video coding layer is still based on hybrid video coding algorithm, buy with important differences