Research Topic Error Concealment Techniques in H.264/AVC for Wireless Video Transmission Vineeth Shetty Kolkeri EE Graduate,UTA

Purpose of H.264 / MPEG-4 part 10 Higher coding efficiency than previous standards, MPEG-1,2,4 part 2, H.261, H.263 2. Simple syntax specifications 3. Seamless integration of video coding into all current protocols 4. More error robustness 5. Various applications like video broadcasting, video streaming, video conferencing, D-Cinema, HDTV 6. Network friendliness 7. Balance between coding efficiency, implementation complexity and cost - based on state-of the-art in VLSI design technology

Better image quality at the same compressed bitrate, or a lower compressed bitrate for the same image quality.

Basic Coding Structure for H.264 [1]

Error Control Goal of Error Control: Overcome the effect of errors, during the transmission of the video frames in the wireless medium, e.g. packet loss on a packet network on a wireless network. 2. Method used for Error Control : Error Concealment

Error Concealment Problem: Transmission errors may result in lost information 2. Goal: Estimate the lost information in order to conceal the fact that an error has occurred 3. Error concealment is performed at the decoder 4. Observation: Video exhibits a significant amount of correlation along the spatial and temporal dimensions 5. Basic approach: Perform some form of spatial/temporal Concealment to estimate the lost information from correctly received data

Error Concealment (cont.) Consider the case where a single macroblock (16x16 block of pixels) is lost Three examples of error concealment: 1.Spatial Concealment: Estimate missing pixels by smoothly extrapolating surrounding pixels Correctly recovering missing pixels is extremely difficult, however even correctly estimating the DC (average) value is very helpful 2.Temporal Concealment: Copy the pixels at the same spatial location in the previous frame Effective when there is no motion, potential problems when there is motion 3.Motion-compensated temporal Concealment: Estimate missing block as motion-compensated block from prior frame Can use coded motion vector, neighboring motion vector, or compute new motion vector

Motion Vector Extrapolation (MVE) Compensate the missed MB by extrapolating each MV that is stored in previously decoded frame. 2. 8x8 sub-block based process. 3. Large overlapped MV is selected for the sub-block. If there is no overlap, then use Zero MV.

4. Error Concealment – MB missing Zero MV Replaces missed MV as (0,0) Copy a macro-block from previously reconstructed reference slice at the exact same position Zero MV concealment in dispersed FMO slices

Error Concealment – Frame missing 1. Temporal Replacement Copy a MB/Frame from previously reconstructed reference slice at the exact same position 2. Motion Vector Copy Exploits MVs of a few past frames Estimate the MV of each pixel in last successful frame Project last frame onto an estimate of missing frame

Temporal Replacement - Frame Copy Frames# 5, 6 and 7 of the Original Sequence Frame# 5 of the decoded frame, Successfully decoded lost Frame # 6. Frame# 6 was reconstructed by Frame copy. Frame #7 is degraded.

"Inter" temporal prediction – block based motion estimation and compensation 1. Multiple reference pictures 2. Reference P pictures 3. Arbitrary referencing order 4. Variable block sizes for motion compensation Seven block sizes: 16x16, 16x8, 8x16, 8x8, 8x4, 4x8 & 4x4 5. 1/4-sample luma interpolation (1/4 or 1/8th-sample chroma interpolation) 6. Weighted prediction 7. Frame or Field based motion estimation for interlaced scanned video

Motion Vector Copy Frames# 5, 6 and 7 of the Original Sequence Frame# 5 of the decoded frame, Successfully decoded lost Frame # 6. Frame# 6 was reconstructed by Motion Copy algorithm. Frame #7 is degraded.

Different Error Concealment Techniques Ref: I.C.Todoli “Performance of Error Concealment Methods for Wireless Video”, Diploma Thesis, Vienna University of Technology, 2007 [1] Decode without residuals Original Error Copy-paste Boundary matching Weighted Average Decode I Frame without residuals Block matching

Implementation and Video Quality Analysis of the Received Sequences 1. Tested the Frame copy and Motion Estimation in the decoder. 2. Implementing the Error Concealment algorithms in the decoder of JM 13.2. Compare results of the recovered frames by error concealment technique from MSE: It calculates the “difference” between two images. It can be applied to digital video by averaging the results for each frame. PSNR: The most commonly used objective quality metric is the Peak Signal to Noise Ratio (PSNR). For a video sequence of frames. SSIM: This approach emphasizes that the Human Visual System (HVS) is highly adapted to extract structural information from visual scenes. Therefore, a measurement of structural similarity (or difference) should provide a good approximation to perceptual image quality.

Future Work Implementing the various Error Concealment algorithm using JM 13.2 Software. 2. Evaluating the quality of recovered frames.

References T. Stockhammer, M. M. Hannuksela and T. Wiegand, “H.264/AVC in Wireless Environments”, IEEE Trans. Circuits and Systems for Video Technology, Vol. 13, pp. 657- 673, July 2003. 2. Soon-kak Kwon, A. Tamhankar and K.R. Rao, ”Overview of H.264 / MPEG-4 Part 10”, J. Visual Communication and Image Representation, vol. 17, pp.186-216, April 2006. 3. S. Wenger, “H.264/AVC over IP” IEEE Trans. Circuits and Systems for Video Technology, vol. 13, pp. 645-656, July 2003. 4. M. Wada, “Selective Recovery of Video Packet Loss using Error Concealment,” IEEE Journal on Selected Areas in Communication, vol. 7, pp. 807-814, June 1989. 5. I.C.Todoli “Performance of Error Concealment Methods for Wireless Video”, Diploma Thesis, Vienna University of Technology, 2007 . 6. Video Trace research group at ASU, “Yuv video sequences,” http://trace.eas.asu.edu/yuv/index.html. 7. A.B. Watson, "Toward a perceptual video quality metric", Human Vision, Visual Processing, and Digital Display VIII, 3299, pp 139-147, 1998. 8. F. Xiao, “Dct-based video quality evaluation,” Final Project for EE392J Stanford Univ. 2000. http://compression.ru/video/quality_measure/vqm.pdf 9. Z. Wang, “The SSIM index for image quality assessment,” http://www.cns.nyu.edu/zwang/files/research/ssim/.