1. H.264/Advanced Video Coding – A New Standard Song Jiqiang Oct 21, 2003

2. The Scope of Video Coding Standardization  Only restrictions on the Bitstream, Syntax, and Decoder are standardized: – Permits the optimization of encoding – Permits complexity reduction for implementability – Provides no guarantees on quality

3. Chronological Table of Video Coding Standards H.261 (1990) MPEG-1 (1993) H.263 (1995/96) H.263+ (1997/98) H.263++ (2000) H.264 ( MPEG-4 Part 10 ) (2003) MPEG-4 v1 (1998/99) MPEG-4 v2 (1999/00) MPEG-4 v3 (2001) 1990 1992 1994 1996 1998 2000 2002 2003 MPEG-2 (H.262) (1994/95) ISO/IEC MPEG ITU-T VCEG

4. H.261: The Basis of Modern Video Compression  The first widespread practical success  Milestone: The first design establishes the typical hybrid structure that dominates today. – 16  16 marcoblock motion compensation – 8  8 DCT transform – Variable-length entropy coding  Operated at 64-2048 Kbps (p  64Kbps)  Still in use – Low complexity, low latency – Mostly as a backward-compatibility feature – Overtaken by H.263

5. MPEG-1: For Storage  Five parts: System, Visual, Audio, Conformance, Reference Software  Applications: VCD, VOD, Digital Camera  Superior quality to H.261 when operated at higher bit rates (≥ 1 Mbps for CIF 352x288 resolution)  Provides approximately VHS quality between 1-2 Mbps using SIF 352x240/288 resolution  Technical features: Adds bi-directional motion prediction and half-pixel motion to H.261 design  Use is fairly widespread, but mostly overtaken by MPEG-2

6. MPEG-2 / H.262: High Bit Rate, High Quality  MPEG-2 contains 10 parts  MPEG-2 Visual = H.262  Not especially useful below 2 Mbps (range of use normally 2-20 Mbps)  Applications: SDTV (2-5Mbps), DVD (6- 8Mbps), HDTV (20Mbps), VOD  Support for interlaced scan pictures  PSNR, temporal, and spatial scalability  “Profile” and “Level”  10-bit precision video sampling

7. H.263: The Next Generation  Has overtaken H.261 as dominant video- conferencing codec  Superior to H.261 at all bit rates  Wins by a factor of two at very low rates  Four basic options: UMV, SAC, Advanced prediction mode, PB-frame  H.263+ (1998): supports all bit rates, more options  H.263++ (2000): more options, emphasizing on error resilience and scalability

8. MPEG-4: H.263 + Additions + Variable Shape Coding  Goal: Support for interactive multimedia  Visual Object (AO), Audio Object (AO) and AVO  Roughly follows H.263 design and adds all prior features and (most important) shape coding  18 video coding profiles  Includes zero-tree wavelet coding of still textured pictures, segmented coding of shapes, coding of synthetic content  2D & 3D mesh coding, face animation modeling  10-bit and 12-bit video  Contains 9 parts. Part 10 will be H.264

9. A Note on Terminology of H.264  The following terms are used interchangeably: – H.26L – The Work of the JVT or “JVT CODEC” – JM2.x, JM3.x, JM4.x – The Thing Beyond H.26L – The “AVC” or Advanced Video CODEC  Proper Terminology going forward: – MPEG-4 Part 10 (Official MPEG Term) ISO/IEC 14496-10 AVC – H.264 (Official ITU Term)

10. Position of H.264

11. New Features of H.264  Multi-mode, multi-reference MC  Motion vector can point out of image border  1/4-, 1/8-pixel motion vector precision  B-frame prediction weighting  4  4 integer transform  Multi-mode intra-prediction  In-loop de-blocking filter  UVLC (Uniform Variable Length Coding)  NAL (Network Abstraction Layer)  SP-slices

12. Profiles and Levels  Profiles: Baseline, Main, and X – Baseline: Progressive, Videoconferencing & Wireless – Main: esp. Broadcast – X: Mobile network  Baseline profile is the minimum implementation – Without CABAC, 1/8 MC, B-frame, SP-slices  11 levels – Resolution, capability, bit rate, buffer, reference # – Built to match popular international production and emission formats – From QCIF to D-Cinema

13. Basic Marcoblock Coding Structure Entropy Coding Scaling & Inv. Transform Motion- Compensation Control Data Quant. Transf. coeffs Motion Data Intra/Inter Coder Control Decoder Motion Estimation Transform/ Scal./Quant. - Input Video Signal Split into Macroblocks 16x16 pixels Intra-frame Prediction De-blocking Filter Output Video Signal

14. Motion Compensation Entropy Coding Scaling & Inv. Transform Motion- Compensation Control Data Quant. Transf. coeffs Motion Data Intra/Inter Coder Control Decoder Motion Estimation Transform/ Scal./Quant. - Input Video Signal Split into Macroblocks 16x16 pixels Intra-frame Prediction De-blocking Filter Output Video Signal Various block sizes and shapes 8x8 0 4x8 01 01 23 4x4 8x4 1 0 8x8 Types 0 16x16 01 8x16 MB Types 8x8 01 23 16x8 1 0

15. Multiple Reference Frames Entropy Coding Scaling & Inv. Transform Motion- Compensation Control Data Quant. Transf. coeffs Motion Data Intra/Inter Coder Control Decoder Motion Estimation Transform/ Scal./Quant. - Input Video Signal Split into Macroblocks 16x16 pixels Intra-frame Prediction De-blocking Filter Output Video Signal Multiple Reference Frames for Motion Compensation

16. B-frame Prediction Weighting  Playback order: I 0 B 1 B 2 B 3 P 4 B 5 B 6 ……...  Bitstream order: I 0 P 4 B 1 B 3 B 2 P 8 B 5 ……... I 0 B 1 B 2 B 3 P 4 B 5 B 6 Time

17. 4  4 Integer Transform Entropy Coding Scaling & Inv. Transform Motion- Compensation Control Data Quant. Transf. coeffs Motion Data Intra/Inter Coder Control Decoder Motion Estimation Transform/ Scal./Quant. - Input Video Signal Split into Macroblocks 16x16 pixels Intra-frame Prediction De-blocking Filter Output Video Signal  4x4 Block Integer Transform  Main Profile: Adaptive Block Size Transform (8x4,4x8,8x8)  Repeated transform of DC coeffs for 8x8 chroma and 16x16 Intra luma blocks

18. Intra-prediction Modes Entropy Coding Scaling & Inv. Transform Motion- Compensation Control Data Quant. Transf. coeffs Motion Data Intra/Inter Coder Control Decoder Motion Estimation Transform/ Scal./Quant. - Input Video Signal Split into Macroblocks 16x16 pixels Intra-frame Prediction De-blocking Filter Output Video Signal  Directional spatial prediction (9 types for luma, 1 chroma) e.g., Mode 3: diagonal down/right prediction a, f, k, p are predicted by (A + 2Q + I + 2) >> 2 Q A B C D E F G H I a b c d J e f g h K i j k l L m n o p M N O P 1 0 3 4 56 7 8 2- DC

19. In-loop De-blocking Filter Without filter with H.264/AVC De-blocking  Highly compressed decoded inter picture  Significantly reduces prediction residuals

20. SP-Slices  Efficiently switching between two bitstreams  Provides VCR-like functions

21. Comparison

22. Comparison

23. Summary  Video coding is based on hybrid video coding and similar in spirit to other standards but with important differences  New key features are: – Enhanced motion compensation – Small blocks for transform coding – Improved de-blocking filter – Enhanced entropy coding  Substantial bit-rate savings (up to 50%) relative to other standards for the same quality  Enhancement on visual quality seems better than that on PSNR  The complexity of the encoder triples that of the prior ones  The complexity of the decoder doubles that of the prior ones

24. Implementations and Implementability  UB Video (JVT-C148) CIF resolution on 800 MHz laptop – Encode: 49 fps, Decode: 137 fps – Encode+Decode: 36 fps – Better quality than R-D optimized H.263+ Profile 3 (IJKT) while using 25% higher rate and low-delay rate control  Videolocus (JVT-D023) SDTV resolution – 30 fps encode on P4 2 GHz with hardware assist – Decode on P3 1 GHz laptop (no hardware assist) – No B frames, no CABAC (approx baseline)  DGFX – SDTV, HDTV SW Encoders, Decoders and PreProcessing – 10-12 Bit Implementation  Others: HHI, Deutsche Telekom, Broadcom, Nokia, Motorola, NDS/Tandberg, Harmonic, LSI Logic, etc.

25. Questions? Thank you!