Supplement, Chapters 6 MC Course, 2009

Chapter 6 H.264/MPEG4 Part 10 Macroblock syntax elements Reference picture management 1/8 motion estimation Motion vector prediction Intra prediction Deblocking filter Transform and quantization Direct and weighted predictions Data partition Raw byte sequence payload

H.264 Encoder/Decoder Forward path Reconstruction path 2018/11/23 unfiltered unfiltered 2018/11/23 Chapter 6 2009

H.264 Profiles FRExt High profiles 2018/11/23 Chapter 6 2009

H.264 Slice Types There is minimal inter-dependency between coded slices which can help to limit the propagation of errors 2018/11/23 Chapter 6 2009

Slice Syntax Skipped (not coded) MBs 2018/11/23 Chapter 6 2009

Macroblock Syntax Elements 2018/11/23 Chapter 6 2009

Reference Picture Management Every reconstructed frame can be marked as Unused for reference Short term picture Long term picture None (directly output) Adaptive memory control (Encoder  Decoder) Managing the short and long term picture indexes. Short term picture may be assigned a long term frame index Short/long term picture may be marked as unused for reference Instantaneous Decoder Refresh Picture (IDR) An encoder sends an IDR to clear the reference picture buffer 2018/11/23 Chapter 6 2009

Reference Picture Management (P-slice) Example: Current frame number=250 Number of reference frames=5 2018/11/23 Chapter 6 2009

Macroblock to slice group map types 2018/11/23 Chapter 6 2009

Slice Groups: dispersed 4 groups 2018/11/23 Chapter 6 2009

Slice Groups: Foreground and Background Map 2018/11/23 Chapter 6 2009

Slice Groups Box-out, Raster and Wipe 2018/11/23 Chapter 6 2009

Interpolation of Chroma: eight-sample a = round ([(8-dx)(8-dy)A+dx(8-dy)B+(8-dx)dyC+dxdyD]/64) 2018/11/23 Chapter 6 2009

Motion Vector Prediction leftmost topmost 2018/11/23 Chapter 6 2009

Motion Vector Prediction (2) For transmitted partitions excluding 16  8 and 8  16 partition sizes: MVp= median (MVA , MVB , MVC ) For upper 16  8 partitions: MVp = MVB For lower 16  8 partitions: MVp = MVA For left 8  16 partitions: MVp = MVA For right 8  16 partitions: MVp = MVC For skipped macroblocks: MVp= (as if the block were encoded in 16  16 Inter mode) 2018/11/23 Chapter 6 2009

Intra Prediction Signaling intra prediction modes The choice of intra prediction mode for each 44 block must be signaled to the decoder and this could potentially require a large number of bits. However, intra modes for neighboring 44 blocks are often correlated. Mode 1 Mode 1 Mode 1 probably 2018/11/23 Chapter 6 2009

Intra Prediction Signaling intra prediction modes 1:H 3 Signaling intra prediction modes Calculate the “most probable” prediction mode of E in both the encoder and the decoder Most probable mode Minimum of the prediction modes of A and B Flag: prev_intra4x4_pred_mode 1: Most probable mode = Prediction mode of E If most probable mode  prediction mode of E prev_intra4x4_pred_mode = 0 rem_intra4x4_pred_mode is used and sent to the decoder only 3 bits are needed for 9 modes Most probable mode Prediction mode of E: 0, …, i-1, i, i+1, i+2, …, 8 rem_intra4x4_pred_mode: 0, …, i-1, -, i , i+1, …, 7 2018/11/23 Chapter 6 2009

Deblocking Filter Edge filtering order Boundary strength (bS) 5 1 2 4 3 6 7 8 9 10 11 12 1616 luma 88 chroma 2018/11/23 Chapter 6 2009

Deblocking Filter Filter decision Filter implementation (p2, p1, p0, q0, q1, q2) is filtered only if bS > 0 and |p0-q0| < , |p1-p0| <  , and |q1-q0| ≦ .  and  depend on QPs of block p and q. Filter implementation bS  {1, 2, 3} (p1, p0, q0, q1) → (p’0, q’0) If |p2-p0|<, (p2, p1, p0, q0) → (p’1) (luma only) If |q2-q0|<, (q2, q1, q0, p0) → (q’1) (luma only) bS = 4 If luma block, |p2-p0|< , and |p0-q0|<round(/4) (p2,p1,p0,q0,q1)→p’0, (p2,p1,p0,q0) →p’1, (p3,p2,p1,p0,q0) →p’2 otherwise (p1, p0, q1) → p’0 If luma block, |q2-q0|< , and |p0-q0|<round(/4) (q2,q1,q0,p0,p1)→q’0, (q2,q1,q0,p0) →q’1, (q3,q2,q1,q0,p0) →q’2 otherwise (q1, q0, p1) → q’0 4-tap filter 2018/11/23 Chapter 6 2009

Digital Cosine Transform Y = A X AT X = AT Y A 2018/11/23 Chapter 6 2009

Digital Cosine Transform (2) 2018/11/23 Chapter 6 2009

Digital Cosine Transform (3) scaling factor core To ensure orthogonal, b is modified a = ½, b = , d = ½ (2nd, 4th rows of C, 2nd, 4th columns of CT)2, … d = c/b  0.414 To simplify the implementation: d  0.5 2018/11/23 Chapter 6 2009

Transform and Quantization 4x4 residual transform and quantization Quantization Pre-scaling matrix Transform round( / Qstep) Input matrix (Implemented by right shift) Integer arithmetic using +, - and SHIFT 16-bit arithmetic MF / 2qbits qbits=15+floor(QP/6) 2018/11/23 Chapter 6 2009 Table lookup

Transform and Quantization 2018/11/23 Chapter 6 2009

Transform and Quantization 4x4 rescaling (in inverse quantizer) Table lookup  Z × Qstep × × 64 2floor(QP/6) The output doubles for every (QP/6) Quantized value scaling factor to avoid rounding errors 2018/11/23 Chapter 6 2009

Transform and Quantization A complete process of Transform, quantization, rescaling, and inverse transform 2018/11/23 Chapter 6 2009

Direct Prediction Spatial direct mode Temporal direct mode 2018/11/23 Chapter 6 2009

Weighted Prediction Three types ‘Explicit’ prediction P slice with ‘explicit’ weighted prediction B slice with ‘explicit’ weighted prediction B slice with ‘implicit weighted prediction ‘Explicit’ prediction The weighted factors are determined by the encoder. ‘Implicit’ prediction The weighted factors are calculated based on the relative temporal positions of the reference pictures. 2018/11/23 Chapter 6 2009

Data Partitioned Slices Different partitions can be placed in separate NAL unit. Partition A Slice header, macroblock header. Partition B Coded residual data for I and SI macroblocks. Partition C Coded residual data for inter coded macroblocks. 2018/11/23 Chapter 6 2009

Raw Byte Sequence Payload (RBSP) 2018/11/23 Chapter 6 2009