1. Rate-Distortion Optimized Motion Estimation for Error Resilient Video Coding Hua Yang and Kenneth Rose Signal Compression Lab ECE Department University of California Santa Barbara, USA Mar. 2005

2. ICASSP 20052 Outline  Motion estimation (ME) for coding efficiency –Conventional ME –Rate-constrained ME & rate-distortion (RD) optimized ME  Motion estimation for error resilience  Proposed end-to-end distortion based RDME –Intuition behind –End-to-end distortion analysis  Simulation results  Conclusions

3. Mar. 2005ICASSP 20053 Motion Estimation for Coding Efficiency  Motion compensated prediction (MCP) –To remove inherent temporal redundancy of video signal –Both the motion vector and the prediction residue are encoded. Coded frame n-1Original frame n

4. Mar. 2005ICASSP 20054 Motion Estimation for Coding Efficiency  Conventional motion estimation –ME Criterion: minimize prediction residue Ignoring the motion vector bit-rate cost

5. Mar. 2005ICASSP 20055 However, not yet the ultimate rate-distortion optimization for the best overall coding performance. Motion Estimation for Coding Efficiency  Motion estimation in low bit rate video coding –In low bit rate video coding, motion vectors may occupy a significant portion of total bit rate. –Efficient bit allocation between motion vector and prediction residue coding is necessary for better overall coding efficiency. : Lagrange multiplier –Rate-constrained motion estimation

6. Mar. 2005ICASSP 20056 Motion Estimation for Coding Efficiency  Motion estimation for low bit rate video coding (cont’d) –Rate-distortion optimized motion estimation (RDME) –Some references [Girod `94] Theoretical analysis of rate-constrained ME [Sullivan `98] Summary of rate-constrained ME [Chung `96] Low complexity RDME for each MB using RD modeling [Schuster `97] Joint RDME for multiple MB’s

7. Mar. 2005ICASSP 20057 Motion Estimation for Error Resilience  In the presence of packet loss: –Packet loss & error propagation Internet – no QoS guarantee Wireless – inherent error-prone channel Error propagation due to MCP No mv for Inter-mode! –Error resilient video coding RD optimization with end-to-end distortion Coding mode selection: {Intra/Inter, QP}  Error resilience via motion compensation –Multi-frame motion compensation (MFMC) [Budagavi `01] –Reference picture selection (RPS) [H.263+] –Error resilient rate-constrained ME [Wiegand `00] Not comprehensively attack the RD optimization problem!

8. Mar. 2005ICASSP 20058 Motion Estimation for Error Resilience  We propose end-to-end distortion based RDME [accounting for packet loss]  The exact RD optimal ME solution for error resilience  Critical: accurate pixel-level end-to-end distortion estimation Build on: recursive optimal per-pixel estimate (ROPE) [R. Zhang, S. Regunathan, and K. Rose `00]

9. Mar. 2005ICASSP 20059  Conventional motion estimation completely ignores the error resilience information. –This error resilience information should be exactly considered for each pixel. Proposed RDME  Intuition for “error resilience via ME” For coding efficiency For error resilience I I P1P1 P3P3 P4P4 P2P2 P2P2 P1P1 P1P1 Best trade-off

10. Mar. 2005ICASSP 200510 –D EP is explicitly affected by mv, whose minimization favors mv’s that point to reference areas with less encoder-decoder mismatch. Proposed RDME  ROPE-based end-to-end distortion analysis Error concealment Error propagated distortion ROPE

11. Mar. 2005ICASSP 200511 Proposed RDME  The proposed RDME solution –Comparing with existent RDME Source coding distortion  end-to-end distortion mv affects not only the R mv vs. R res trade-off, but also more importantly, the coding efficiency vs. error resilience trade-off. Packet loss impact –Comparing with existent RD optimized coding mode selection Extended Inter mode with the mv parameter Further optimize the Inter-mode performance

12. Mar. 2005ICASSP 200512 Simulations  Objective: to check upper-bound performance –Joint {mv, QP} optimization –RD calculation via actual encoding  Simulation settings –UBC H.263+ –Encoding: I-P-P-…… –Transmission: independent packet loss, with a uniform p –Decoding: 50 different packet loss realizations for each p –Performance: average luminance PSNR

13. Mar. 2005ICASSP 200513 Simulations  Simulation settings (cont’d) –Testing methods Conventional ME (cME) The proposed RDME (RDME) –Testing scenarios Random Intra updating (rI): arbitrarily assigns MB’s to 1/p groups, and cycles through them updating one group per frame. Optimal Intra updating (oI): RD optimized Intra/Inter mode selection.

14. Mar. 2005ICASSP 200514 Simulation Results  Random Intra PSNR vs. Packet loss rate [QCIF, 10f/s, 48kb/s] Miss_amForeman

15. Mar. 2005ICASSP 200515 Simulation Results  Optimal Intra PSNR vs. Packet loss rate [QCIF, 10f/s, 48kb/s] Miss_amForeman

16. Mar. 2005ICASSP 200516 Simulation Results  Random Intra PSNR vs. Total bit rate [QCIF, 10f/s, p=10%] Miss_amForeman

17. Mar. 2005ICASSP 200517 Simulation Results  Optimal Intra PSNR vs. Total bit rate [QCIF, 10f/s, p=10%] Miss_amForeman

18. Mar. 2005ICASSP 200518 Simulation Results Miss_am: QCIF, 10f/s, 48kb/s, p=10%, random Intra Conventional ME [29.58dB] RDME [33.83dB]

19. Mar. 2005ICASSP 200519 Simulation Results Foreman: 1 st 200f, QCIF, 10f/s, 112kb/s, p=10%, random Intra Conventional ME [23.92dB] RDME [26.92dB]

20. Mar. 2005ICASSP 200520  Besides Intra updating, RDME presents another good alternative for error resilience. Conclusions  Identify the new opportunity of achieving error resilience via motion estimation.  Propose an RD optimal ME solution, which further optimizes the Inter-mode performance.  Investigate the upper-bound performance. –With random Intra: substantial gain –With optimal Intra: significant gain at low bit rates.

21. Mar. 2005ICASSP 200521 Conclusions  Future work I: more comprehensive tests –Inaccurate p, bursty loss, or over actual networks, etc.  Future work II: complexity reduction –RD modeling, separate mv and QP optimization, sophisticated ME strategies, etc.  Originally, the power of Intra coded MB’s is only recognized as stopping past error propagation, while the proposed RDME reveals their new potential on reducing future error propagation.

22. Mar. 2005ICASSP 200522 References  [Girod `94] B. Girod, ``Rate-constrained motion estimation,'' Nov. 1994.  [Sullivan `98] G. J. Sullivan and T. Wiegand, ``Rate-distortion optimization for video compression,’’ Nov. 1998.  [Chung `96] W. C. Chung, F. Kossentini, and M. J. T. Smith, ``An efficient motion estimation technique based on a rate-distortion criterion,'' May 1996.  [Schuster `97] G. M. Schuster and A. K. Katsaggeslos, ``A theory for the optimal bit allocation between displacement vector field and displaced frame difference,'' Dec. 1997.  [Budagavi `01] M. Budagavi and J. D. Gibson, ``Multiframe video coding for improved performance over wireless channels,'' Feb. 2001.  [H.263+] ITU-T, Rec. H,263, ``Video codeing for low bitrate communications'', version 2 (H.263+), Jan. 1998.  [Wiegand `00] T. Wiegand, N. Farber, K. Stuhlmuller and B. Girod, ``Error-resilient video transmission using long-term memory motion-compensated prediction,'' June 2000.  [Zhang `00] R. Zhang, S. L. Regunathan, and K. Rose, ``Video coding with optimal intra/inter mode switching for packet loss resilience,'' June 2000.

23. Mar. 2005ICASSP 200523 The End