Ai-Mei Huang And Truong Nguyen Image processing, 2006 IEEE international conference on Motion vector processing based on residual energy information for motion compensated frame interpolation

Ai-Mei Huang And Truong Nguyen IEEE, WORLD OF WIRELESS, MOBILE AND MULTIMEDIA NETWORKS. (WOWMOM), 2008 IEEE, WORLD OF WIRELESS, MOBILE AND MULTIMEDIA NETWORKS. (WOWMOM), 2008 Motion vector processing using bidirectional frame difference in motion compensated frame interpolation

INTRODUCTION Low bandwidth requirements, i.e. video telephony by skipping frames low frame rate video usually results in motion jerkiness Motion-compensated frame interpolation (MCFI) Directly uses the received MVs Suffer from annoying artifacts such as blockiness and ghost effect. Motion re-estimation is not suited for mobile devices Video in encoder decoder Reconstructed frame Bit stream Video coding/compression

Motion compensated frame interpolation In MCFI methods, a skipped frame is often bi- directionally interpolated from its two neighboring reconstructed frames. by using the received MVs of the second frame. MV is not always reliable!

The correlation between motion vector reliability and residual energy. Macroblock(MB) contain areas of different motion. Encoder favors the MV that can represent most of the region The prediction residual will be generated and encoded. Correct those unreliable MVs for frame interpolation Avoid using those unreliable MVs to correct other MVs.

The proposed method Based on residual energy associated with each motion vector iterative process, stops when the MAP is no longer changed, or reaches a pre-defined maximum iteration, max_ite.

(1)motion vector classification based on residual energy Classify MVs to 3 Groups. Calculate every 8x8 block’s residual energy E m,n By taking the sum of the absolute value of each reconstructed prediction error for each pixel. E m,n <threshold  MV will be classified as reliable and place into first group G1. E m,n >=threshold  MV will be classified as unreliable and place into first group G3. Unreliable MV’s neighboring MVs within the same MB will be classified as possibly unreliable into the second group G2 (base unit) 16*16 MB=>four 8*8 blocks

(2) motion vector correction Works on those unreliable MVs Correcting from neighbor reliable MV A residual-energy constrained median filter (RECMF) is used in this process and defined as S contains the neighboring MVs centered at v* m,n. Select a new MV only from its neighboring reliable MVs. v m,n itself will be excluded from the candidates. we prevent unreliable MVs to be used to correct other unreliable MVs.

(3)motion vector similarity check Ensure v* m,n is not identical or similar to v m,n Remain unreliable Use angle variance by taking inner product of the two vectors. two MVs is similar,fail in the check => b m,n still in G3 Pass in the check => b m,n take in G1

(4)motion vector re-sampling and smoothing In [7], each 8 x 8 block is further broken into four 4 x4 sub-blocks. The MVs of these four sub-blocks can be obtained simultaneously by minimizing a smoothness measure, which is defined in the following. For example: [7] G. Dane and T. Q. Nguyen, "Smooth motion vector resampling for standard compatible video post-processing," Proc. Asilomar Conf: Signals, Systems and Computers, 2004.

SIMULATIONS two video sequences, FOREMAN and SILENT, CIF frame resolution 30 frame per second (fps). both encoded using H.263 but even frames are skipped by the encoder. fix quantization parameter (QP) values. Avg. bitrate FOREMAN is Kbps SILENT is Kbps

Ai-mei Huang And Truong Nguyen IEEE, WORLD OF WIRELESS, MOBILE AND MULTIMEDIA NETWORKS. (WOWMOM), 2008 IEEE, WORLD OF WIRELESS, MOBILE AND MULTIMEDIA NETWORKS. (WOWMOM), 2008 Motion vector processing using bidirectional frame difference in motion compensated frame interpolation

Motion compensated frame interpolation(1/3)

Motion compensated frame interpolation(2/3) A example

Motion compensated frame interpolation(3/3) Although the MV is measured reliably in [4] Still have some potential issues In Fig 2 and 3, the moving object is composed of four blocks, {B 1,B 2,B 3,B 4 } Fig. 3 shows that even MVs are correctly estimated, the moving object still cannot be reconstructed successfully. the simplest way to reduce these visual artifacts is to find the best matches bi-directionally by minimizing the difference between bidirectional predictions. [4]A.-M. Huang and T. Nguyen. Motion vector processing based on residual energy information for motion compensated frame interpolation. in Proc. ICIP’06, pages 2721–2724, 2006.

Bidirectional motion vector processing for MCFI(1/2) Use bidirectional MV processing for the received MVF in the current decoder frame f t+1. we select the best MV from a set of MV candidates V b *, is selected based on the minimum difference between forward and backward predictions.

(2/2) Eliminate the ghost artifacts due to unmatched bidirectional predictions Computation is greatly reduced Use the integer-pixel frame interpolation affect MV precision. modify eq. (4) to allow MVs have small deviation from -1 to 1.

SIMULATIONS Compare our method with direct MCFI, and vector median filter (VMF). Test sequence QVGA size: FORMULA 1, TABLE, and STEFAN CIF size: FOREMAN, FORMULA 1, FAST FOOD, WALK, and SILENT Encode using H.264, avg. bitrate: 384kbps even frames are skipped.

Adopt structure similarity (SSIM) index for quality assessment. luminance contrast Structure

Performance

CONCLUSIONS The motion vector reliability is analyzed and classified. we examine how the MV precision affect the quality of the interpolated frame in the bidirectional MCFI scheme. we propose a bidirectional MV processing method by selecting the best MV from neighboring MVs