1. Project Proposal Error concealment techniques in H.264 Under the guidance of Dr. K.R. Rao By Moiz Mustafa Zaveri (moiz.mustafazaveri@mavs.uta.edu) (1001115920)

2. Contents Project objective Problem statement Sequence information Error characteristics Spatial domain error concealment Temporal domain error concealment References

3. Acronyms AVCAdvanced Video Coding AVSAudio Video Standard DSLDigital Subscriber Line HVSHuman Visual System IECInternational Electrotechnical Commission ISOInternational Organization for Standardization ITUInternational Telecommunication Union JMJoint Model LANLocal Area Network

4. Acronyms MMSMultimedia Messaging Service MPEGMoving Picture Experts Group VCEGVideo Coding Experts Group

5. Problem Statement Video transmission errors are errors in the video sequence that the decoder cannot decode properly. In real-time applications, retransmission is not an option, therefore the missing parts of the video have to be concealed. To conceal these errors, spatial and temporal correlations of the video sequence can be utilized. As H.264 [3] employs predictive coding, this kind of corruption spreads spatio-temporally to the current and consecutive frames. This is seen in fig.1.

6. Problem Statement Figure 1: Error propagation in H.264 due to predictive coding [1]

7. Objective To implement both the spatial domain and temporal domain categories of error concealment techniques in H.264 [3] with the application of the Joint Model (JM 19.0) reference software [3] and use metrics like the peak signal to noise ratio (PSNR) and mean squared error (MSE), blockiness [1], blurriness [1] and structural similarity index metric (SSIM) [4] in order to compare and evaluate the quality of reconstruction.

8. H.264/AVC Block Diagram The H.264/AVC encoder/decoder block diagram is shown in fig.2. Figure 2: H.264 encoder/decoder block diagram [7]

9. Sequence Information Temporal information Movement -It is easier to conceal linear movements in one direction because we can predict pictures from previous frames (the scene is almost the same). -If there are movements in many directions, finding a part of a previous frame to fit, is going to be more difficult. An example of this can be screen cuts. - In fig.3, five frames of three video sequences (football match, village panorama and music video clip) with different amounts of movement are seen. Speed - The slower the movement of the camera, the easier it will be to conceal an error. - We can see an example of two different video speeds if we compare village sequence with football sequence given in fig.3.

10. Sequence Information Figure 3: Movement and speed [2]

11. Sequence Information Spatial information Smoothness of the neighborhood -The smoothness of the neighborhood of the erroneous macroblock will determine the difficulty of the spatial concealment. -In fig.4, there are three different cases in which the lost macroblock has to be concealed using the neighboring blocks. -In the first one, it is going to be easy to reconstruct the lost macroblock because the neighborhood is very uniform (smooth), with almost no difference between the neighboring macroblocks. -In the second situation, it is going to be a little bit more difficult; we have to look for the edges, and then, recover the wire line. -The third case is an example where the neighbors cannot help us to recover the macroblock because they do not give any information about the lost part (in this case, the eye).

12. Sequence Information Figure 4: Smoothness of the neighborhood [2]

13. Error Characteristics Lost information Size and form of the lost region I/P frame - If the error is situated in an I frame, it is going to affect more critically the sequence because it will affect all the frames until the next I frame and I frames do not have any other reference but themselves. - If the error is situated in a P frame it will affect the rest of the frames until the next I frame but we still have the previous I frame as a reference.

14. Spatial domain error concealment Weighted Averaging The pixel values of the damaged macroblock are obtained by interpolating the neighboring four pixels. Missing pixel values can be recovered by calculating the average pixel values from the four pixels in the four 1-pixel wide boundaries of the damaged macroblock weighted by the distance between the missing pixel and the four macroblocks boundaries (upper, down, left and right boundaries).

15. Temporal domain error concealment Frame Copy algorithm Replaces the missing image part in the current frame with the spatially corresponding part inside a previously decoded frame, which has maximum correlation with the affected frame. It is easier to conceal linear movements in one direction because pictures can be predicted from previous frames (the scene is almost the same). When the current frame ‘n’ is missing (i,j) values, they can be concealed using the (i,j) values from the ‘n-1’ frame. This is shown in fig.5. Figure 5: Copying (i,j) pixel values from previous ‘n-1’ frame into current ‘n’ frame [1] [1]

16. Temporal domain error concealment Frame copy algorithm If there are movements in many directions or scene cuts, finding a part of previous frame that is similar is going to be more difficult, or even impossible. The slower is the movement of the camera, the easier will be to conceal an error. This method only performs well for low motion sequences but its advantage lies in its low complexity.

17. Temporal domain error concealment

18. Motion Vector Interpolation A motion vector of a 4x4 block can be estimated by interpolating its value from the motion vectors in the surrounding macroblocks, as shown in fig.8. The distance between the 4x4-block to be found and the surrounding 4x4 blocks can be used as a weighting factor. This is shown in fig.7. Figure 7: (a) Weighted averaging in Motion Vector Interpolation [1] (b) Motion vector interpolation [1]

19. References [1]I. C. Todoli, “Performance of Error Concealment Methods for Wireless Video,” Ph.D. thesis, Vienna University of Technology, 2007. [2]V.S. Kolkeri "Error Concealment Techniques in H.264/AVC, for Video Transmission over Wireless Networks", M.S. Thesis, Department of Electrical Engineering, University of Texas at Arlington, Dec. 2009 Online: http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/index_tem.html. [3]H.264/AVC Reference Software Download: Online: http://iphome.hhi.de/suehring/tml/download/ [4]Z. Wang, “The SSIM index for image quality assessment,” Online: http://www.cns.nyu.edu/zwang/files/research/ssim/.

20. References [5]Video Trace research group at ASU, “YUV video sequences,” Online: http://trace.eas.asu.edu/yuv/index.html. [6] MSU video quality measurement tool: Online: http://compression.ru/video/quality_measure/video_measurement_tool_en.html. [7] S. –K. Kwon, A. Tamhankar and K.R. Rao, “An overview of H.264 / MPEG-4 Part 10,” J. Visual Communication and Image Representation, vol. 17, pp.186-216, Apr. 2006. [8]Y. Xu and Y. Zhou, “H.264 Video Communication Based Refined Error Concealment Schemes,” IEEE Transactions on Consumer Electronics, vol. 50, issue 4, pp. 1135–1141, Nov. 2004.

21. References [9]M. Wada, “Selective Recovery of Video Packet Loss using Error Concealment,” IEEE Journal on Selected Areas in Communication, vol. 7, issue 5, pp. 807-814, June 1989. [10] Y. Chen et al, “An Error Concealment Algorithm for Entire Frame Loss in Video Transmission,” IEEE Picture Coding Symposium, Dec. 2004. [11] S. K. Bandyopadhyay et al, “An error concealment scheme for entire frame losses for H.264/AVC,” IEEE Sarnoff Symposium, pp. 1-4, Mar. 2006. [12]H. Ha, C. Yim and Y. Y. Kim, “Packet Loss Resilience using Unequal Forward Error Correction Assignment for Video Transmission over Communication Networks,” ACM digital library on Computer Communications, vol. 30, pp. 3676-3689, Dec. 2007.

22. References [13] T. Wiegand et al, “Overview of the H.264/AVC video coding standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, July 2003. [14]Z. Wang et al, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004. [15]D. Levine, W. Lynch and T. Le-Ngoc, “Observations on Error Detection in H.264,” 50 th IEEE Midwest Symposium on Circuits and Systems, pp. 815-818, Aug. 2007. [16]B. Hrušovský, J. Mochná and S. Marchevský, “Temporal-spatial Error Concealment Algorithm for Intra-Frames in H.264/AVC Coded Video,” 20th International Conference Radioelektronika, pp. 1-4, Apr. 2010.

23. References [17]W. Kung, C. Kim and C. Kuo “Spatial and Temporal Error Concealment Techniques for Video Transmission Over Noisy Channels,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 16, issue 7, pp. 789-803, July 2006.