1. Error-Resilient Coding and Decoding Strategies for Video Communication Thomas Stockhammer and Waqar Zia Presented by Li Ma

2. Background & Motivation 2  Video becoming more popular  Advances in bandwidth, capacity enhancements  Requirements:  data transmission rate  Real-time delivery of multimedia data  Limitation:  QoS available is not sufficient to guarantee error-free delivery for all receivers  Motivation:  Provide means of dealing with various transmission impairments

3. Content 3  Focus on MCP-coded video  Concentrate on tools and features integrated in standard H.264/AVC  Focus on specific tools for improved error resilience  Other advanced error-resilience features not covered:  Multiple description coding  Distributed video coding  Combinations with network prioritization and FEC

4. Outline 4  Video Communication Systems  Error-Resilient Video Transmission  Resynchronization and Error Concealment  Error Mitigation  Summary

5. Video Communication Systems 5  End-to-End Video Transmission

6. Video Applications 6  wide variety of applications  Different bit rate ranges:  HDTV: 20 Mbit/s  MMS on cell: 20 Kbit/s  Different tolerable end-to-end delay  Conversational applications constraints: ≤ 200-250 ms

7. Transmission Impairments 7  Differences of errors  Wireless networks:  Fading and interference cause burst errors: multiple lost bits  IP network:  Congestion results in packets lost  Methods  Detect presence of errors  Intermediate protocol layers (UDP) could drop erroneous pkts  Video data pkts treated as lost if delayed more than threshold

8. Data Losses in MCP-Coded Video 8  Transmitted over error-prone channels  Error concealment  Error propagation / Spatiotemporal error propagation

9. Example of Error Propagation 9  Pt @ t=0 is lost.  Error propagation till t=8  Intra-coded image transmitted @ t=9

10. 10  Therefore, when data units might get lost, a video coding system should provide:  Means that allow completely avoiding transmission errors  Features that allow minimizing the visual effects of errors  Features to limit spatial and spatiotemporal error propagation

11. Outline 11  Video Communication Systems  Error-Resilient Video Transmission  Resynchronization and Error Concealment  Error Mitigation  Summary

12. Error-resilient Video Transmission 12  System Overview

13. Features 13  @Sender  MBs are grouped in data units and entropy coding used  Error Control before transmission over lossy channel  Forward Error Correction(FEC)  Backward Error Correction (BEC)  Prioritization Methods  Combinations of above  @Receiver  Erroneous and missing data detected and localized  Decoder gets correct data units or error indication  Error concealment applied at positions where no data received  Report loss of data units to encoder

14. Design Principles 14  error-resilience tools decrease compression efficiency  Main goal:  Shannon’s separation principle: compression separated with transport  In low delay situations, error-free transport is impossible  System Design Principles  1. Loss correction below codec layer  2. Error detection  3. Prioritization methods  4. Error recovery and concealment  5. Encoder-decoder mismatch avoidance

15. Video Compression Tools Related to Error Resilience 15  Slice Structured Coding  Flexible MB Ordering  Scalable Coding  Data Partitioning  Flexible Reference Frame  Intra Information Coding  Pictures Switching

16. Slice Structured Coding 16  Slices provide spatially distinct resynchronization points within the data for a single frame  Several MBs grouped together: a slice header  Variable sized data units  Encoder can select the location of sync. Points  Motion vector prediction not allowed over boundaries  Encoder decides either:  Allocate fixed number of MB to one slice  Or fixed bits to one slice (matched to pkt size in network)

17. Flexible MB Ordering 17  Flexible Macroblock Ordering (FMO)  Allows mapping of MBs to Slice groups  A slice group may contain several slices  MBs can be transmitted in flexible and efficient way  Spatially collocated images areas can be interleaved in different slices  greater probability of concealing lost MB  Protection: Can map ROI (region of interest) into a separated slice group

18. Data Partitioning 18  Loss of some syntax elements of a bit stream results in larger degradation of quality compared to others  E.g. Loss of motion vector  Data partitioning results in Graceful Degradation  Categorize syntax elements  Header information  Motion information  Texture information

19. Layout of compressed data 19 2 additional sync. Points available Without Data Partitioning: With Data Partitioning:

20. Data Partitioning (Cont.) 20  Unequal Error Protection (UEP)  Protect partitions of different importance  More important data offered more protection

21. Flexible Reference Frame 21  H.263 v.1 & MPEG-2 allow only a single reference frame for predicting P frame and mostly 2 for B frame.  Possible to have significant statistical dependencies between other pictures too  Use more frames than just the recent one as reference  Advantages:  Increased compression efficiency  Improved error resilience

22. Example of Flexible Reference Frame 22

23. Example of Flexible Reference Frame 23  Enable Subsequences  Use a subsequence of “anchor frames” at lower frame rate  E.g “ P ”  Other frames inserted in between to achieve overall frame rate  E.g “ P’ ”  Error propagation:  Only till next P received

24. Outline 24  Video Communication Systems  Error-Resilient Video Transmission  Resynchronization and Error Concealment  Error Mitigation  Summary

25. Resynchronization and Error Concealment 25  Video Packetization Modes  Without FMO(flexible macroblock ordering)  1. a constant number of MBs within one slice (arbitrary size)  2. the slice size bounded to some max bytes (arbitrary # of MBs)

26. Video Packetization Modes (Cont.) 26  With FMO (more flexible)  Slice interleaving  Dispersed MB allocation using checkerboard patterns  Subpictures within a picture  etc.

27. Error Concealment 27  Basic Idea  Decoder should generate a representation for lost area  Match as close as possible to the lost info  Within manageable complexity  Techniques  Spatial Error Concealment  Temporal Error Concealment  Hybrid Concealment  Other Techniques

28. Spatial Error Concealment 28  Based on assumption of continuity of natural scene content in space  Use pixel values of surrounding available MBs  Estimate of lost pixel:  αβγ are weighing factors  Determine relative impact of vertical, Horizontal, upper, lower…  Disadvantage  Blurred reconstruction

29. Temporal Error Concealment 29  Rely on the continuity of a video sequence in time  Use temporally neighboring areas to conceal lost regions  Previous Frame Concealment (PFC)  Use previous corresponding data to copy to current frame  Only good when little motion  Widely used due to simplicity

30. Hybrid Concealment 30  When only apply spatial concealment  Concealed regions are significantly blurred  When only use temporal error concealment  Significantly discontinuities in the concealed regions  Hybrid temporal-spatial technique applied  MB mode info of reliable and concealed neighbors decide which concealment method to use

31. Hybrid (cont.) 31  For intra-coded images  Only use spatial concealment  For inter-coded images  Use temporal concealment when more than half of the available neighbor MBs are inter-coded  Otherwise, use spatial concealment  Referred to as Adaptive temporal and spatial Error Concealment (AEC)

32. Selected Results 32  Performance of different error concealment strategies

33. Selected Performance Results for Wireless 33  Low-delay and low-complexity requirements  Max allowed buffering at encoder limited to 250ms

34. 34  Using a smaller slice size of 150 bytes is lower in PSNR when error free  Because:  increased packetization overhead  prediction limitations on slice boundaries  Performs good when in lossy channel  Because the loss affects only a small area of the image for fixed slice size

35. Outline 35  Video Communication Systems  Error-Resilient Video Transmission  Resynchronization and Error Concealment  Error Mitigation  Summary

36. Motivation 36  Error propagation is major problem over lossy channels  Encoder can change encoding behavior when he finds it’s likely to be lossy or knows decoder suffering losses

37. Operational Encoder Control 37  Encoder appropriately select parameters  Motion vectors  MB modes  Quantization parameters  Reference frames  Spatial and temporal resolution

38. 38

39. Conclusion 39  Bad decisions at the encoder can lead to  Poor results in coding efficiency  Poor in error resilience  Or both  If no feedback is available  an increased percentage of intra MBs performs best  If feedback available  Interactive Error Control is best

40. Outline 40  Video Communication Systems  Error-Resilient Video Transmission  Resynchronization and Error Concealment  Error Mitigation  Summary

41. Summary 41  Important to understand video can benefit significantly when data delivered reliably  Introduced error-resilience tools and impact  For good overall performance, should take into account:  the selection of error-resilience tools  rate-distortion-optimized mode selection  the channel characteristics

42. 42