Multi-Frame Reference in H.264/AVC 卓傳育

Outline Introduction to Multi-Frame Reference in H.264/AVC Multi-Frame Reference Problem Two papers propose to multi-frame reduction and selection. Conclusion

Basic Macroblock Coding Structure

Multiple Reference Frames

Motion Compensation Accuracy

Intra Prediction

ANALYSIS AND REDUCTION OF REFERENCE FRAMES FOR MOTION ESTIMATION IN MPEG-4 AVC/JVT/H.264 Yu-Wen Huang 1, Bing-Yu Hsieh 1, Tu-Chih Wang 1, Shao-Yi Chien 1, Shyh-Yih Ma 2, Chun-Fu Shen 2, and Liang-Gee Chen 1 1. DSP/IC Design Lab., Department of Electrical Engineering, National Taiwan University, 2. Vivotek Incorporation,

Searching steps of intra/inter prediction with multiple reference frames H.264 Proposed Selected modes, inter/intra prediction residues, motion vectors

Statistics of Reference Frames If we can detect that the transformed and quantized coefficients are very close to zero in the first reference frame, we can turn off the matching process from the rest frames.

Statistics of Reference Frames

MV compactness

Statistics of Average MV Compactness If the MV compactness of a MB after ME from previous frame is very small, we should stop searching the rest 4 frames. If the optimal frame or mode of a MB does not change after ME from 5 frames, we classify these MBs as type I. Otherwise, we classify them as type II.

Texture consideration

Summarize of the analysis If 16x16 mode is selected, the optimal reference frame tend to be unchanged. If inter-modes with smaller blocks are selected, searching more frames tend to be helpful. If MVs of larger blocks are similar to MVs of smaller blocks, it is likely that no occlusion or uncovering occurs in MB, so one reference frame may be enough. If MVs of larger blocks are more different from MVs of smaller blocks, MB often crosses object boundaries and thus requires more reference frames. If the texture of a MB is very complicated, it may require more reference frames.

Proposed algorithm

Simulation results

Rate distortion curves of various sequences.

Average searched frames for various sequences

A Novel Approach To Fast Multi-Frame Selection For H.264 Video Coding Andy Chang, Oscar C. Au, Y. M. Yeung Department of Electrical and Electronic Engineering The Hong Kong University of Science and Technology

Observation on Multi-frame motion estimation Two types of temporal redundancy that can be captured by using multi-frame but not traditional single frame ME/MC. –Short-term memory Ex. The blinking of an eye. –Sub-pixel movement of textures textures and objects with different version of sub- pixel movement (“integer-pixel location”, “half-pixel location” and “quarter-pixel location”) may occur in successive video frames.

Examples of pixel movement

Pixel movement properties The optimal reference frame tends to be the one with the same “sub-pixel location” as the current frame. When there is more than one frame with the same “sub-pixel location”, the one closer to the current frame is usually better.

Selection of Frame for motion estimation

Proposed multi-frame selection process Each macroblock in each reference frame will have a “sub-pixel location” which is calculated and updated by adding the motion vector obtained in the previous search with the previous “sub-pixel location”. When two or more macroblocks have the same “sub-pixel location”, only one frame is enabled for motion estimation. The “sub-pixel location” of two macroblocks are considered the same if both the x and y component of “sub-pixel location” are equal.

Proposed multi-frame selection process The previous reference frame has the highest probability of being selected. Motion estimation is always performed in previous reference frame and the motion vector obtained will be used to update the “sub-pixel location” of the current macroblock. If we assume frame t-3 is chose to be the reference frame for the black (current) macroblock in frame t, the “sub-pixel location” of the black macroblock in frame t is obtained by adding the motion vector information to the “sub- pixel position” of black macroblock in frame t-3.

Flow Chart of the proposed algorithm

Experimental Results QCIF(176x144), QP=16