1. Ai-Mei Huang and Truong Nguyen Image Processing (ICIP), 2009 16th IEEE International Conference on 1

2. CONTENTS INTRODUCTION COLOR INFORMATION MV PROCESSING FOR MCFI USING THE COLOR INFORMATION SIMULATIONS CONCLUSIONS 2

3. Introduction(1/2) Color information has been shown to be effective in the object edges detection  Due to its insensitivity on specular reflection  Prevent false edge detection as compared to luminance-based methods 3 Y1Y2 Y3Y4 CbCr 8 8 8 8 16 MB (Macro Block) Chrominance(color information)Luminance(intensity)

4. Introduction(2/2) Color has sharper and more consistent variations between object boundaries  Applications often take the color information to assist the image segmentation process. In our previous work [8]  The color information was found very useful for the unreliable MV detection  Especially in the areas where the luminance component tends to distribute uniformly. In this paper, we would like to  Examine the color information  Analyze how the chrominance components can be used to assist the MV processing in MCFI. 4 [8] A.-M. Huang and T. Nguyen, “A novel multi-stage motion vector processing method for motion compensated frame interpolation,” in Proc. ICIP’07, pp. 389–392, 2007.

5. Color Information(1/5) The luminance components have stronger intensity distribution than the chrominance components  Conventional motion estimation often ignore the color information due to the complexity. 5 Y Cb Cr

6. Color Information(2/5) Color characteristics are distinct from luminance  Such as the insensitivity in highlight or shadow areas  Used in preventing the false edge detection The chrominance improves the edge identification for the static text, face features, the cap, and the shirts 6

7. Color Information(3/5) If the moving objects have sharp edges, the ambiguous motions seem more unlikely to appear. From Fig. 1(b), we can observe that the luminance has very smooth variations around the face and shirts areas.  Since the motion is mainly determined using the luminance difference, the motion can be easily wrong in these areas. 7

8. Color Information(4/5) Many artifacts around the nose and the shirts in (a) MVs around the shirts can only be detected in (c)  The luminance have uniformly distribution so that the encoder always chooses the face motion.  The color has strong gradients. 8

9. Color Information(5/5) Generally, chrominance residual distribution is similar to the luminance components. The pavement and lawn have very similar intensity. The color difference will become relatively large. 9

10. Motion Vector Analysis The residual energy with color consideration be represented as follows:  r Y (i, j), r Cb (i, j), and r Cr (i, j) are the reconstructed residual signals of Y, Cb and Cr components of the 8×8 block, b m,n  α is the weight used to emphasize the degree of color difference. Empirically set α=8 for 4:2:0 YUV  The residues are embedded in the reconstructed signals during the decoding process. 10

11. MV classification process Compare E m,n to a predefined threshold, ε 1, based on the combined residual information. The adjacent MBs will be merged as a group using the residual energy distribution. 11 16 MB

12. Motion Vector Correction using the Color Information Minimizing the absolute bidirectional prediction difference (ABPD) between forward and backward predictions. 12

13. SIMULATIONS Two video sequences, FOREMAN and FORMULA 1  CIF frame resolution all encoded using H.263  with even frames skipped and the skipped frames are interpolated 13

14. Visual Comparisons(1/2) Fig. 4(c), the artifacts around the nose and the eye are reduced. These artifacts are removed in Fig. 4(d).  Since the chrominance information sharpens the residual energy.  Unreliable MVs around the shirts and face areas can be identified and be corrected accordingly. 14

15. Visual Comparisons(2/2) The intensity between grass and pavement is very similar.  So the motion estimation easily fails on the white lines areas. 15

16. CONCLUSIONS We present using color information for the MV reliability classification. Unreliable MVs with small luminance difference can be effectively detected. 16