Rate-Distortion Optimized Motion Estimation for Error Resilient Video Coding Hua Yang and Kenneth Rose Signal Compression Lab ECE Department University of California Santa Barbara, USA Mar. 2005

ICASSP Outline  Motion estimation (ME) for coding efficiency –Conventional ME –Rate-constrained ME & rate-distortion (RD) optimized ME  Motion estimation for error resilience  Proposed end-to-end distortion based RDME –Intuition behind –End-to-end distortion analysis  Simulation results  Conclusions

Mar. 2005ICASSP Motion Estimation for Coding Efficiency  Motion compensated prediction (MCP) –To remove inherent temporal redundancy of video signal –Both the motion vector and the prediction residue are encoded. Coded frame n-1Original frame n

Mar. 2005ICASSP Motion Estimation for Coding Efficiency  Conventional motion estimation –ME Criterion: minimize prediction residue Ignoring the motion vector bit-rate cost

Mar. 2005ICASSP However, not yet the ultimate rate-distortion optimization for the best overall coding performance. Motion Estimation for Coding Efficiency  Motion estimation in low bit rate video coding –In low bit rate video coding, motion vectors may occupy a significant portion of total bit rate. –Efficient bit allocation between motion vector and prediction residue coding is necessary for better overall coding efficiency. : Lagrange multiplier –Rate-constrained motion estimation

Mar. 2005ICASSP Motion Estimation for Coding Efficiency  Motion estimation for low bit rate video coding (cont’d) –Rate-distortion optimized motion estimation (RDME) –Some references [Girod `94] Theoretical analysis of rate-constrained ME [Sullivan `98] Summary of rate-constrained ME [Chung `96] Low complexity RDME for each MB using RD modeling [Schuster `97] Joint RDME for multiple MB’s

Mar. 2005ICASSP Motion Estimation for Error Resilience  In the presence of packet loss: –Packet loss & error propagation Internet – no QoS guarantee Wireless – inherent error-prone channel Error propagation due to MCP No mv for Inter-mode! –Error resilient video coding RD optimization with end-to-end distortion Coding mode selection: {Intra/Inter, QP}  Error resilience via motion compensation –Multi-frame motion compensation (MFMC) [Budagavi `01] –Reference picture selection (RPS) [H.263+] –Error resilient rate-constrained ME [Wiegand `00] Not comprehensively attack the RD optimization problem!

Mar. 2005ICASSP Motion Estimation for Error Resilience  We propose end-to-end distortion based RDME [accounting for packet loss]  The exact RD optimal ME solution for error resilience  Critical: accurate pixel-level end-to-end distortion estimation Build on: recursive optimal per-pixel estimate (ROPE) [R. Zhang, S. Regunathan, and K. Rose `00]

Mar. 2005ICASSP  Conventional motion estimation completely ignores the error resilience information. –This error resilience information should be exactly considered for each pixel. Proposed RDME  Intuition for “error resilience via ME” For coding efficiency For error resilience I I P1P1 P3P3 P4P4 P2P2 P2P2 P1P1 P1P1 Best trade-off

Mar. 2005ICASSP –D EP is explicitly affected by mv, whose minimization favors mv’s that point to reference areas with less encoder-decoder mismatch. Proposed RDME  ROPE-based end-to-end distortion analysis Error concealment Error propagated distortion ROPE

Mar. 2005ICASSP Proposed RDME  The proposed RDME solution –Comparing with existent RDME Source coding distortion  end-to-end distortion mv affects not only the R mv vs. R res trade-off, but also more importantly, the coding efficiency vs. error resilience trade-off. Packet loss impact –Comparing with existent RD optimized coding mode selection Extended Inter mode with the mv parameter Further optimize the Inter-mode performance

Mar. 2005ICASSP Simulations  Objective: to check upper-bound performance –Joint {mv, QP} optimization –RD calculation via actual encoding  Simulation settings –UBC H.263+ –Encoding: I-P-P-…… –Transmission: independent packet loss, with a uniform p –Decoding: 50 different packet loss realizations for each p –Performance: average luminance PSNR

Mar. 2005ICASSP Simulations  Simulation settings (cont’d) –Testing methods Conventional ME (cME) The proposed RDME (RDME) –Testing scenarios Random Intra updating (rI): arbitrarily assigns MB’s to 1/p groups, and cycles through them updating one group per frame. Optimal Intra updating (oI): RD optimized Intra/Inter mode selection.

Mar. 2005ICASSP Simulation Results  Random Intra PSNR vs. Packet loss rate [QCIF, 10f/s, 48kb/s] Miss_amForeman

Mar. 2005ICASSP Simulation Results  Optimal Intra PSNR vs. Packet loss rate [QCIF, 10f/s, 48kb/s] Miss_amForeman

Mar. 2005ICASSP Simulation Results  Random Intra PSNR vs. Total bit rate [QCIF, 10f/s, p=10%] Miss_amForeman

Mar. 2005ICASSP Simulation Results  Optimal Intra PSNR vs. Total bit rate [QCIF, 10f/s, p=10%] Miss_amForeman

Mar. 2005ICASSP Simulation Results Miss_am: QCIF, 10f/s, 48kb/s, p=10%, random Intra Conventional ME [29.58dB] RDME [33.83dB]

Mar. 2005ICASSP Simulation Results Foreman: 1 st 200f, QCIF, 10f/s, 112kb/s, p=10%, random Intra Conventional ME [23.92dB] RDME [26.92dB]

Mar. 2005ICASSP  Besides Intra updating, RDME presents another good alternative for error resilience. Conclusions  Identify the new opportunity of achieving error resilience via motion estimation.  Propose an RD optimal ME solution, which further optimizes the Inter-mode performance.  Investigate the upper-bound performance. –With random Intra: substantial gain –With optimal Intra: significant gain at low bit rates.

Mar. 2005ICASSP Conclusions  Future work I: more comprehensive tests –Inaccurate p, bursty loss, or over actual networks, etc.  Future work II: complexity reduction –RD modeling, separate mv and QP optimization, sophisticated ME strategies, etc.  Originally, the power of Intra coded MB’s is only recognized as stopping past error propagation, while the proposed RDME reveals their new potential on reducing future error propagation.

Mar. 2005ICASSP References  [Girod `94] B. Girod, ``Rate-constrained motion estimation,'' Nov  [Sullivan `98] G. J. Sullivan and T. Wiegand, ``Rate-distortion optimization for video compression,’’ Nov  [Chung `96] W. C. Chung, F. Kossentini, and M. J. T. Smith, ``An efficient motion estimation technique based on a rate-distortion criterion,'' May  [Schuster `97] G. M. Schuster and A. K. Katsaggeslos, ``A theory for the optimal bit allocation between displacement vector field and displaced frame difference,'' Dec  [Budagavi `01] M. Budagavi and J. D. Gibson, ``Multiframe video coding for improved performance over wireless channels,'' Feb  [H.263+] ITU-T, Rec. H,263, ``Video codeing for low bitrate communications'', version 2 (H.263+), Jan  [Wiegand `00] T. Wiegand, N. Farber, K. Stuhlmuller and B. Girod, ``Error-resilient video transmission using long-term memory motion-compensated prediction,'' June  [Zhang `00] R. Zhang, S. L. Regunathan, and K. Rose, ``Video coding with optimal intra/inter mode switching for packet loss resilience,'' June 2000.

Mar. 2005ICASSP The End