1. Adaptive Multi-path Prediction for Error Resilient H.264 Coding Xiaosong Zhou, C.-C. Jay Kuo University of Southern California Multimedia Signal Processing 2006

2. Outline  Introduction  Adaptive Multi-path Prediction for H.264  Observation and Research Motivation  Computation of Expected Decoder Distortion  Adaptive Reference Selection (ARS) Scheme  Experimental Results  Conclusion and Future Work

3. Introduction  Video communication problem Source Encode Output Decode Erroneous Channel

4. Introduction  Error Resilient tools  Error Resilient Entropy Coding (EREC)  Unequal Error Protection by Layered Coding  …  Mismatch exists, error propagation can’t be stopped properly  Not compatible with H.264

5. Introduction  One way for reducing error propagation  Intra Refreshing : insert intra macroblocks in temporally coded (P or B) video frames  Intra macroblocks in H.264 are coded by intra prediction based on neighbors  Intra macroblock have much lower coding efficiency than inter macroblocks  Not suitable for H.264

6. Introduction  Multiple reference (long-term reference) motion compensation predictive coding (LTMCP)  Enhance coding efficiency  Proposed for error resilience  T. Wiegand et. al, “Error-resilient video transmission using long-term memory motion-compensated prediction”  Select the best reference frame by evaluating the expected reconstruction calculated based on the error feedback and an error propagation model

7. Observation and Research Motivation  Sequential prediction  In H.264, LTMCP allows encoder to choose the best prediction from a number of reference frames  The best reference of some blocks may exist in a long-term reference frame  But, sequential prediction is still common in H.264

8. Observation and Research Motivation  Utilize long term reference frames for error resilience  Error resilience performance is improved using alternative prediction patterns Different predictive patterns

9. Observation and Research Motivation  If a video stream is encoded into these fixed prediction patterns  As most video frames are forced to use a distant reference frame, coding efficiency is likely to be sacrificed  Performance vary for different macroblocks since it’s largely dependent on video content  It’s difficult to design a fixed prediction pattern at the frame level  In H.264, reference frame selection is done at macroblock level  Incorporate the idea of multi-path predictive coding at the macrokblock level

10. Computation of Expected Decoder Distortion  Error map  Created and maintained for each allowed reference frame in buffer  Store the absolute value of the expected error e of every pixel in the frame Encoder Erroneous Channel Decoder Encoder Error-free Channel Decoder

11. Computation of Expected Decoder Distortion  can not be obtained directed  For a pixel in the n th frame, update its value of e by  p e : channel error rate  e p,n : expected error from error propagation  e c,n : expected error from error concealment when pixel is lost

12. Computation of Expected Decoder Distortion  To calculate e c,n, the mismatch caused by reconstruction of error concealment scheme  Consider a simple error concealment method  Intra block : copy pixel from the boundary of correctly reconstructed block above the target block  Inter block : copy the block from the same position in the previous frame  Should also consider de-blocking operation which attenuate the error generated by error concealment  d  e n-1 : error value of the pixel where MV points in reference frame  α n-1 : attenuation factor of the propagating error from n-1 st to n th frame  Expected decoder distortion

13. Adaptive Reference Selection (ARS)  in LTMCP, multiple predictions can be created to encode block M  The predictions generated from multi-reference frames are evaluated based on both coding and error resilience performance

14. Adaptive Reference Selection (ARS)  Three used vectors  x N = ( X 1, …, X N ) : the set of all N MBs in a GOP  m N = ( M 1, …, M N ) : all modes selected by each MB  q N = ( Q 1, …, Q N ) : quantization parameters used to encode these MBs  The overall mode decision problem is :

15. Adaptive Reference Selection (ARS)  Convert to an unconstraint optimization problem using Lagrange Multiplier method

16. Adaptive Reference Selection (ARS)  Overall expected distortion through erroneous channel  : expected error that has mean zero  : decoder error and uncorrelated to  Therefore

17. Adaptive Reference Selection (ARS)  Rewrite mode decision problem equation  Assumption: rate and distortion of MB i have no impact to other MBs  : expected overall distortion of the GOP due to error propagation

18. Adaptive Reference Selection (ARS)  d  α : attenuation factor  M : expected number of frames in the future prediction path of the pixel  The method to calculate M

19. Experimental Results

21.  Averaged Frame Index

22. Experimental Results  R-D performance comparison under test condition T2

23. Experimental Results  R-D performance comparison under test condition T7

24. Conclusion and Future Work  An adaptive prediction selection scheme was proposed in this work to create multiple prediction paths in the compressed video stream  The proposed scheme is able to maintain good coding efficiency of the compressed stream while serve as an effective error resilience tool in visual communication applications  In the future, we plan to develop a new model to simplify the calculation of the expected decoder distortion to reduce the complexity of the proposed scheme