CMPT-884 Jan 18, 2010 Error Concealment Presented by: Cameron Harvey CMPT 820 October

Internet Basics  The internet is not reliable as a communication channel  Packets can become corrupted  Packets can get lost  Packets can have high latency  Packets can arrive out of order

Encoder/decoder MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 16 FIGURE 2.4

Motion Compensated Prediction  Standard compression techniques use MCP to estimate a macroblock based on previous or future frames  Packet loss will affect not just the frame being decoded, but many other frames as well  Errors are propagated temporally until a new I- frame is encountered

Slices  Slices  A slice is composed of a series of macroblocks  Slices are non-overlapping  Slices are structure in such a way that motion vectors are all contained within the slice

Slice Coding Schemes  Constant MBs/slice  Maximum bytes/slice

Macroblock Allocation Maps  FMO – Flexible Macro-block Ordering  New to the H.264/AVC standard is the ability to define slice groups  Groups can be further divided into separate slices

Flexible Macroblock Ordering  FMO provides a useful tool for error concealment  We can use FMO to arrange for slice groups in such a way that macroblocks are not adjacent to macroblocks from the same slice group  Slices are packetized separately and transmitted

Some types of FMO patterns Y Dhondt, P Lambert, Flexible Macroblock Ordering – an error resilience tool in H.264/AVC, Gent, Belgium, Dec 2004

FMO – Type 1  Slices are interleaved in such a way that a missing slice will be surrounded by many samples which are decoded correctly MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 31 FIGURE 2.12

Example  Suppose the packet containing macroblocks of slice group #0 is lost

Example (2)  The missing macroblock can be interpolated using the data from the surrounding macroblocks  This is type of concealment is called Spatial Concealment T B RL

Spatial Error Concealment  The key assumption behind Spatial Error Concealment is that image content is continuous and has smooth texture

Spatial Error Concealment  The pixel, X ij, is calculated as the weighted average of four pixel values on the edge of the macroblock MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 33 FIGURE 2.13

Spatial Error Concealment (2)  The calculation for pixel (i,j) is   The values of determine the impact the border pixels will have and are generally related to the distance from the pixel under consideration

Spatial Error Concealment (3)  This technique will result in blurred areas as the reconstructed macroblock is basically an average of the surrounding macroblocks  Refinements of this technique can be used to recover some texturing information

Spatial Error Concealment (4) Error concealment without FMO Error concealment using FMO Picture 11 from a GOP – 5% packet loss Y Dhondt, P Lambert, Flexible Macroblock Ordering – an error resilience tool in H.264/AVC, Figure 2, Gent, Belgium, Dec 2004

Temporal Error Concealment  Temporal concealment operates with the assumption that a video sequence is continuous  In its simplest form, one can simply copy a missing macroblock from the corresponding macroblock in the previous frame. This is called Previous Frame Concealment (PFC)  Useful if the scene is static  Problematic if the scene changes rapidly as this creates many artifacts

Temporal Error Concealment (2)  We can improve on PFC with an assumption of uniform motion. With this assumption, one can examine the motion vectors of the adjacent macroblocks to infer the motion of the lost macroblock  Many possibilities exist. A boundary matching technique can be used to select from the many possibilities

Boundary Matching Technique  The sum of the squares of the differences between border pixels of Top, Right, and Bottom is determined  The motion vector which minimizes this sum is a good candidate MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 35 FIGURE 2.15

Other Temporal Concealment Refinements  Overlapped Block Motion  The average of three 16X16 pixel regions is used  Use Median Motion Vector of T, R, and B  Sum of Absolute Difference (instead of square)  Estimate motion with the Top macroblock  Use PFC if top motion vector is unavailabel  Multi-hypothesis error concealment uses a weighted average of multiple frames

Example  Click to see an example of Spatial-temporal error concealment  Project_conceal_files/ecdemo.shtml Project_conceal_files/ecdemo.shtml

Temporal Concealment over a Scene Change  Packet loss over a scene change violates the assumption of temporal continuity  Scene change detection can be used at the cost of additional complexity  Macroblock information over a scene change exhibit distinct characteristics

Abrupt Scene Changes S. Pei and Y Chou, Novel Error Concealment Method With Adaptive Prediction to the Abrupt and Gradual Scene Changes, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 3, Figure 2(a) Apr 2002

Abrupt Scene Changes (2)  Use Spatial Concealment on the P-frame (Why?)  Use Temporal Error Concealment on the B frames  Use the P or I frame from the same scene as reference  For IBBP, all four frames must be processed before concealment can occur

Gradual Scene Changes S. Pei and Y Chou, Novel Error Concealment Method With Adaptive Prediction to the Abrupt and Gradual Scene Changes, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 3, Figure 2(b) Apr 2002

Gradual Scene Changes (2)  In the first sub-GOP (IBBP or PBBP), temporal concealment can be used for the final P-frame (Why?)  B frames are calculated as a weighted average of the first P or I frame and the final P- frame  Across P frame in different sub-GOP, interpolation is used aided by the large number of intracoded macroblocks

Questions He must be very ignorant for he answers every question he is asked. Voltaire