H.264/AVC

Evolution of Video coding Standards ITU-T Standard Joint ITU-T/MPEG Standards MPEG 1988 1990 1992 1994 1996 1998 2000 2002 2004 H.261 (Version 1) (Version 2) H.263 H.263+ H.263++ H.262/MPEG-2 H.264/MPEG-4 AVC MPEG-1 MPEG-4

Applications Entertainment Video (1-8+ Mbps, higher latency) Broadcast / Satellite / Cable / DVD / VoD/ FS-VDSL / … DVB/ATSC/SCTE, DVD Forum, DSL Forum Conversational Services (usu. <1Mbps, low latency) H.320 Conversational 3GPP Conversational H.324/M H.323 Conversational Internet/best effort IP/RTP 3GPP Conversational IP/RTP/SIP Streaming Services (usu. lower bit rate, higher latency) 3GPP Streaming IP/RTP/RTSP Streaming IP/RTP/RTSP (without TCP fallback) Other Services 3GPP Multimedia Messaging Services

Structure of H.264/AVC video encoder Control Data Video Coding Layer Coded Macroblock Data Partitioning Coded Slice/Partition Network Abstraction Layer H.320 MP4FF H.323/IP MPEG-2 etc.

High-Level VCL Summary Some new key aspects are: Enhanced motion compensation Small blocks for transform coding Improved de-blocking filter Enhanced entropy coding Substantial bit-rate savings relative to other standards for the same quality

Macroblocks and Slices Macroblocks: 16x16 luma + 8x8 for chroma samples Slice: a set of MBs that can be decoded without use of other slices I slice: intra-prediction (I-MBs) P slice: possibly one inter-prediction signal (I- and P-MBs) B slice: up to two inter-prediction signals (I- and B-MBs) SP slice: P slice to facilitate switching between coded streams SI slice: I slice to to facilitate switching between coded streams The only reason that we may require adjacent slices is for deblocking filter. The main feature of SP-frames is that identical SP-frames can be reconstructed even when different reference frames are used for their prediction. SI-frames are used in conjunction with SP-frames. This property make them useful for applications such as random access, adapt to network condition, and error recovery/ resilience We next talk about ordering of MBs.

Enhanced MC (Inter Prediction) Every macroblock can be split in one of 7 ways for improved motion estimation

Enhanced MC (Inter Prediction) Accuracy of motion compensation = 1/4 pixel Up to 5 reference frames for SDTV size @ L3 Reference B pictures Weighted predictions Motion Estimation을 수행하는 신호에 적응적으로 가중치를 주어 계산한 후 예측신호를 생성

4 modes for 16x16 intra prediction Intra Spatial Prediction using surrounding available samples Intra 4x4 Prediction Modes 4 modes for 16x16 intra prediction Intra 16x16 Prediction Modes

Transform MBs are divided up into 4x4 or 8x8 blocks Transformation matrix T4x4 or T8x8 is applied to residual data of every block For the DC Coefficients, Hadamard transform H4x4 for the 4×4 array of luma DC coefficients in Intra MBs predicted in 16×16 mode. Hadamard transform H2x2 for the 2 × 2 array of chroma DC coefficients (in any macroblock).

Quantization Logarithmic step size control A total of 52 values of Qstep are supported by the standard, indexed by a Quantization Parameter, QP. Qstep doubles in size for every increment of 6 in QP. Can change to any step size at macroblock level Quantization reconstruction is one multiply, one add, one shift The wide range of quantiser step sizes makes it possible for an encoder to control the tradeoff accurately and flexibly between bit rate and quality. The values of QP can be different for luma and chroma.

Adaptive Deblocking Filter Blocking artifacts 4*4 transforms and block-based motion compensation Block edges are typically reconstructed with less accuracy than interior pixels. Improve subjective and objective quality of the decoded picture Result in bit rate savings of around 6~9% Original Frame Reconstructed, QP=36 (no filter) Reconstructed, QP=36 (with filter)

Entropy Coding CAVLC (Context Adaptive Variable Length Coding) Context : already coded information of the neighboring blocks and the coding status of the current block Optimized VLC tables are provided for each context to code the coefficients in different statistical conditions CABAC (Context Adaptive Binary Arithmetic Codes) Use a binary arithmetic coding engine Compression improvement is consequence of Adaptive probability estimation Improved context modeling scheme Exploiting symbol correlations by using contexts Average bit-rate saving over CAVLC 5~15%

Error Resilience Tools Flexible Macroblock Order (FMO) Arbitrary Slide Order (ASO) Redundant Slice (RS) Data Partitioning (DP)

MB to Slice Group Mappings Flexible Macroblock Ordering (FMO) a subset of the macroblocks may contain one or more slices Application example: Error resilience

Arbitrary Slice Ordering (ASO) The decoding order of the slices could be arbitrary Application example: reduce end-end transmission delay in RT app

H.264/AVC의 해상도의 크기

Profiles

Basic Coding Structure

Comparison to Previous Standards The set of test sequences for this comparison consists of four QCIF (10 and 15 Hz) sequence 176 x144 resolution at 10–256 kb/s four CIF (15 Hz and 30 Hz) sequence CIF 352 x 288 resolution at 128–1024 kb/s

Comparision of Standards Feature/Standard MPEG-1 MPEG-2 MPEG-4 part 2 (visual) H.264/MPEG-4 part 10 Macroblock size 16x16 16x16 (frame mode) 16x8 (field mode) Block Size 8x8 16x16, 16x8, 8x8 16x16, 8x16, 16x8, 8x8, 4x8, 8x4, 4x4 Transform 8x8 DCT 8x8 DCT/Wavelet 4x4, 8x8 Int DCT 4x4, 2x2 Hadamard Quantization Scalar quantization with step size of constant increment Vector quantization Scalar quantization with step size of increase at the rate of 12.5% Entropy coding VLC VLC, CAVLC, CABAC Motion Estimation & Compensation Yes Yes, more flexible Up to 16 MVs per MB Playback & Random Access

Comparision of Standards (cont’d..) Feature/Standard MPEG-1 MPEG-2 MPEG-4 part 2 (visual) H.264/MPEG-4 part 10 Pel accuracy Integer, ½-pel Integer, ½-pel, ¼-pel Profiles No 5 8 3 Reference picture one multiple Bidirectional prediction mode forward/backward forward/forward backward/backward Picture Types I, P, B, D I, P, B I, P, B, SP, SI Error robustness Synchronization & concealment Data partitioning, FEC for important packet transmission Synchronization, Data partitioning, Header extension, Reversible VLCs Data partitioning, Parameter setting, Flexible macroblock ordering, Redundant slice, Switched slice Transmission rate Up to 1.5Mbps 2-15Mbps 64kbps - 2Mbps 64kbps -150Mbps Compatibility with previous standards n/a Yes Encoder complexity Low Medium High

References Related group Test software Test sequences Papers MPEG website http://www.mpeg.org JVT website: ftp://ftp.imtc-files.org/jvt-experts www.mpegif.org Test software H.264/AVC JM Software: http://bs.hhi.de/~suehring/tml/download Test sequences http://ise.stanford.edu/video.html http://kbs.cs.tu-berlin.de/~stewe/vceg/sequences.htm http://www.its.bldrdoc.gov/vqeg ftp.tnt.uni-hannover.de/pub/jvt/sequences/ http://trace.eas.asu.edu/yuv/yuv.html Papers Wiegand, T.; Sullivan, G.J.; Bjontegaard, G.; Luthra, A., “Overview of the H.264/AVC Video Coding Standard,” in IEEE transactions on circuits and systems for video technology, Vol. 12, No.7, July. 2003. Sullivan, G.J.; Wiegand, T., "Video Compression - From Concepts to the H.264/AVC Standard," Proceedings of the IEEE , vol.93, no.1, pp.18-31, Jan. 2005 Jorn Ostermann et al., “Video coding with H.264/AVC: Tools, Performance, and Complexity,” in IEEE Circuit and systems magazine, first quarter. 2004. M. Mahdi Ghandi and Mohammad Ghanbari, “The H.264/AVC Video Coding Standard for the Next Generation Multimedia Communication,” in IAEEE Journal