Download presentation
Presentation is loading. Please wait.
Published byConstance Melton Modified over 6 years ago
1
Final Project Presentation ERROR CONCEALMENT TECHNIQUES IN H
Final Project Presentation ERROR CONCEALMENT TECHNIQUES IN H.264 Under the guidance of Dr. K.R. Rao By Moiz Mustafa Zaveri ( )
2
Contents Problem statement Project objective Sequence information
Error characteristics Spatial domain error concealment Temporal domain error concealment Quality Metrics Generation of errors Simulation Test Sequences Sequence Formats Results of Frame copy algorithm Conclusions Future Work References
3
Acronyms AVC Advanced Video Coding AVS Audio Video Standard
DSL Digital Subscriber Line HVS Human Visual System IEC International Electrotechnical Commission ISO International Organization for Standardization ITU International Telecommunication Union JM Joint Model LAN Local Area Network
4
Acronyms MMS Multimedia Messaging Service MSU Moscow State University
PSNR Peak signal to noise ratio SAD Sum of absolute differences SSIM Structural similarity index metric
5
Problem Statement Video transmission errors are errors in the video sequence that the decoder cannot decode properly. In real-time applications, retransmission is not an option, therefore the missing parts of the video have to be concealed. To conceal these errors, spatial and temporal correlations of the video sequence can be utilized. As H.264 [3] employs predictive coding, this kind of corruption spreads spatio-temporally to the current and consecutive frames. This is seen in fig.1.
6
Figure 1: Error propagation in H.264 due to predictive coding [1]
Problem Statement Figure 1: Error propagation in H.264 due to predictive coding [1]
7
Objective To implement both the spatial domain and temporal domain categories of error concealment techniques in H.264 [3] with the application of the Joint Model (JM 19.0) reference software [3] and use metrics like the peak signal to noise ratio (PSNR) and mean squared error (MSE), blockiness [1], blurriness [1] and structural similarity index metric (SSIM) [4] in order to compare and evaluate the quality of reconstruction.
8
Figure 2: H.264 encoder/decoder block diagram [7]
H.264/AVC Block Diagram The H.264/AVC encoder/decoder block diagram is shown in fig.2. Figure 2: H.264 encoder/decoder block diagram [7]
9
Sequence Information Temporal information Movement
It is easier to conceal linear movements in one direction because we can predict pictures from previous frames (the scene is almost the same). If there are movements in many directions, finding a part of a previous frame to fit, is going to be more difficult. An example of this can be screen cuts. In fig.3, five frames of three video sequences (football match, village panorama and music video clip) with different amounts of movement are seen. Speed - The slower the movement of the camera, the easier it will be to conceal an error. - We can see an example of two different video speeds if we compare village sequence with football sequence given in fig.3.
10
Figure 3: Movement and speed [2]
Sequence Information Figure 3: Movement and speed [2]
11
Sequence Information Spatial information
Smoothness of the neighborhood The smoothness of the neighborhood of the erroneous macroblock will determine the difficulty of the spatial concealment. In fig.4, there are three different cases in which the lost macroblock has to be concealed using the neighboring blocks. In the first one, it is going to be easy to reconstruct the lost macroblock because the neighborhood is very uniform (smooth), with almost no difference between the neighboring macroblocks. In the second situation, it is going to be a little bit more difficult; we have to look for the edges, and then, recover the wire line. The third case is an example where the neighbors cannot help us to recover the macroblock because they do not give any information about the lost part (in this case, the eye).
12
Figure 4: Smoothness of the neighborhood [2]
Sequence Information Figure 4: Smoothness of the neighborhood [2]
13
Error Characteristics
• Lost information • Size and form of the lost region • I/P frame - If the error is situated in an I frame, it is going to affect more critically the sequence because it will affect all the frames until the next I frame and I frames do not have any other reference but themselves. - If the error is situated in a P frame it will affect the rest of the frames until the next I frame but we still have the previous I frame as a reference.
14
Spatial domain error concealment
Weighted Averaging The pixel values of the damaged macroblock are obtained by interpolating the neighboring four pixels. Missing pixel values can be recovered by calculating the average pixel values from the four pixels in the four 1-pixel wide boundaries of the damaged macroblock weighted by the distance between the missing pixel and the four macroblocks boundaries (upper, down, left and right boundaries).
15
Temporal domain error concealment
Frame Copy algorithm Replaces the missing image part in the current frame with the spatially corresponding part inside a previously decoded frame, which has maximum correlation with the affected frame. It is easier to conceal linear movements in one direction because pictures can be predicted from previous frames (the scene is almost the same). When the current frame ‘n’ is missing (i,j) values, they can be concealed using the (i,j) values from the ‘n-1’ frame. This is shown in fig.5. Figure 5: Copying (i,j) pixel values from previous ‘n-1’ frame into current ‘n’ frame [1] [1]
16
Temporal domain error concealment
Frame copy algorithm If there are movements in many directions or scene cuts, finding a part of previous frame that is similar is going to be more difficult, or even impossible. The slower is the movement of the camera, the easier will be to conceal an error. This method only performs well for low motion sequences but its advantage lies in its low complexity.
17
Temporal domain error concealment
Boundary Matching Let B be the area corresponding to a one pixel wide boundary of a missing block in the nth frame Fn. Motion vectors of the missing block as well as those of its neighbors are unknown. The coordinates [ 𝑥 , 𝑦 ] of the best match to B within the search area A in the previous frame 𝐹 𝑛−1 have to be found. This is shown in fig.6. Figure 6: Boundary matching algorithm to find best match for area B in search area A of the previous frame [1]
18
Temporal domain error concealment
Motion Vector Interpolation A motion vector of a 4x4 block can be estimated by interpolating its value from the motion vectors in the surrounding macroblocks, as shown in fig.8. The distance between the 4x4-block to be found and the surrounding 4x4 blocks can be used as a weighting factor. This is shown in fig.7. Figure 7: (a) Weighted averaging in Motion Vector Interpolation [1] (b) Motion vector interpolation [1]
19
Quality Metrics Peak Signal to Noise Ratio (PSNR) and Mean Squared Error (MSE) Blockiness and Blurriness Structural Similarity Index Metric (SSIM)
20
𝑀𝑆𝐸= 1 𝑊𝑋 𝑖=1 𝑊 𝑗=1 𝑋 [𝐹(𝑖,𝑗) − 𝐹 0 (𝑖,𝑗) ] 2 (7)
PSNR and MSE PSNR is used to evaluate the quality of reconstruction of a frame. The PSNR for an 8 bit PCM (0-255 levels) is calculated using: [1] 𝑃𝑆𝑁𝑅=10. 𝑙𝑜𝑔 10 (255) 2 𝑀𝑆𝐸 [𝑑𝐵] (6) Where, 𝑀𝑆𝐸 is the mean square error, given by: [1] 𝑀𝑆𝐸= 1 𝑊𝑋 𝑖=1 𝑊 𝑗=1 𝑋 [𝐹(𝑖,𝑗) − 𝐹 0 (𝑖,𝑗) ] 2 (7) Where, 𝑊×𝑋 is the size of the frame, 𝐹 0 is the original frame and 𝐹 is the current frame. The MSE is computed by averaging the squared intensity differences of the distorted and reference image/frame pixels. Two distorted images with the same MSE may have very different types of errors, some of which are much more visible than others.
21
𝑃𝑆𝑁𝑅 𝑌𝑈𝑉 = 6 𝑃𝑆𝑁𝑅 𝑌 + 𝑃𝑆𝑁𝑅 𝑈 + 𝑃𝑆𝑁𝑅 𝑉 /8 (8)
PSNR and MSE The YUV-PSNR for the 4:2:0 format is calculated using: [1] 𝑃𝑆𝑁𝑅 𝑌𝑈𝑉 = 6 𝑃𝑆𝑁𝑅 𝑌 + 𝑃𝑆𝑁𝑅 𝑈 + 𝑃𝑆𝑁𝑅 𝑉 /8 (8) Where, 𝑃𝑆𝑁𝑅 𝑌 is the PSNR of the Y component, 𝑃𝑆𝑁𝑅 𝑈 is the PSNR of the U component and 𝑃𝑆𝑁𝑅 𝑉 is the PSNR of the V component
22
Distortion artifacts Here measurement of distortion artifacts like blockiness and blurring is done. Blockiness is defined as the distortion of the image characterized by the appearance of an underlying block encoding structure. [1] Blurriness is defined as a global distortion over the entire image, characterized by reduced sharpness of edges and spatial detail. [1] Blockiness measure [1]: This metric was created to measure the visual effect of blocking. If the value of the metric for first picture is greater than the value for the second picture, it means that first picture has more blockiness, than the second picture. An example is seen in fig.8. Blurriness measure [1]: This metric compares the power of blurring of two images. If the value of the metric for first picture is greater than the value for the second picture, it means that second picture is more blurred, than first. An example is seen in fig.9. The blockiness and blurriness value are computed using the Moscow State University Video Quality Measurement Tool [6] shown in fig.10.
23
Figure 8: (a) Orignal image (b) Blockiness in original image [1]
Distortion artifacts Figure 8: (a) Orignal image (b) Blockiness in original image [1]
24
Figure 9: (a) Orignal image (b) Blurriness in original image [1]
Distortion artifacts Figure 9: (a) Orignal image (b) Blurriness in original image [1]
25
MSU Video Quality Measurement Tool
Figure 10: MSU Video Quality Measurement Tool [1]
26
Structural Similarity Index (SSIM)
SSIM compares local patterns of pixel intensities (for similarities) that have been normalized for luminance and contrast. The block diagram for measuring the SSIM is given in fig.11. Figure 11: SSIM measurement [1]
27
Structural Similarity (SSIM)
Let x and y be two image patches extracted from the same spatial location of two images being compared. Let 𝜇 𝑥 and 𝜇 𝑦 be their means and 𝜎 𝑥 2 and 𝜎 𝑦 2 be their variances. Also, let 𝜎 𝑥𝑦 2 be the variance of x and y. The luminance, contrast and structure comparison are given by: [1] (4) (5) (6)
28
Structural Similarity (SSIM)
Where 𝐶 1 , 𝐶 2 and 𝐶 3 are all constants given by: [1] (7) L is the dynamic range of the pixel values (L = 255 for 8 bits/pixel gray scale images), and 𝐾 1 ≪1 and 𝐾 2 ≪1 are scalar constants. The general SSIM can be calculated as follows: [1] (8) Where 𝛼,𝛽 𝑎𝑛𝑑 𝛾 are parameters which define the relative importance of the three components.
29
Generation of Errors To generate errors we use the rtp_loss file in the software solution. This program can be used to simulate frame loss on RTP files. The C random number generator is used for selecting the frames to be lost. The number of frames to be lost depends on the ‘loss percent’ that the user provides. Example: rtp_loss.exe infile outfile losspercent <keep_leading_packets> where <keep_leading_packets> is an optional parameter that specifies the number of RTP frames that are kept at the beginning of the file. The random number generator is not initialized before usage. Therefore subsequent runs of the tool will create identical loss pattern. Figs. 12 and 13 show a single frame of Akiyo and Foreman sequences with and without frame loss.
30
Figure 12: (a) Original Akiyo frame (b) Akiyo frame with error
Generation of Errors Figure 12: (a) Original Akiyo frame (b) Akiyo frame with error
31
Figure 13: (a) Original Foreman frame (b) Foreman frame with error
Generation of Errors Figure 13: (a) Original Foreman frame (b) Foreman frame with error
32
Simulation Steps Encoder Download the JM 19.0 reference software [3].
Modify the encoder and decoder configuration files to accept these desired inputs: Encoder InputFile = "foreman_qcif.yuv" # Input sequence FramesToBeEncoded = # Number of frames to be coded FrameRate = # Frame Rate per second ( ) OutputFile = "enout_ " # Bitstream QPISlice = 28 # Quant. param for I Slices (0-51) QPPSlice = 28 # Quant. param for P Slices (0-51) OutFileMode = 1 # Output file mode, 0:Annex B, 1:RTP
33
Simulation Steps Decoder
InputFile = "lossout_ " # H.264/AVC coded bitstream OutputFile = "decout_150.yuv" # Output file, YUV/RGB FileFormat = # NAL mode (0=Annex B, 1: RTP packets) ConcealMode = # Err Concealment(0:Off,1:Frame Copy,2:Motion Copy) Build the software solution in Microsoft Visual Studio. Run the ‘lencod.exe’ file in command prompt to obtain an encoded output .264 file. Run the ‘rtp_loss.exe’ file in command prompt to introduce a certain percentage of loss in the encoded output and obtain a lossy .264 file. Run the ‘ldecod.exe’ file to obtain the decoded output yuv file in which one of the two error concealment algorithms have been employed to conceal the frame losses.
34
Figure 14: Encoder simulation output
35
Figure 15: Decoder simulation output
36
Test Sequences Sequences used: akiyo_qcif.yuv, foreman_qcif.yuv, stefan_qcif.yuv Total number of frames: 300 Number of frames encoded: 150 Height: 176; Width: 144 Quantization Parameter: 28 Frame rate: 30 frames/second
37
Sequence Formats Common Intermediate Format (CIF) and Quadrature Common Intermediate Format (QCIF) determine the resolution of the frame. The resolution of CIF is 352x288 and the resolution of QCIF is 1/4 of CIF, which is 176x144. Consider the YCbCr family of color spaces where Y represents the luminance, Cb represents the blue-difference chroma component and Cr represents the red-difference chroma component. For QCIF and CIF, the luminance Y is equal to the resolution.
38
Figure 17:Quadrature Common Intermediate Format (QCIF) [2]
Sequence Formats For sampling resolution 4:2:0 Figure 16: Common Intermediate Format (CIF) [2] Figure 17:Quadrature Common Intermediate Format (QCIF) [2]
39
Results of Frame Copy algorithm
In this project, each of the three video sequences: Akiyo.yuv, Foreman.yuv and Stefan.yuv [5] have been made erroneous with loss of frames ranging from 1% to 5% of the total number of encoded frames. The errors in the resultant lossy sequences have then been concealed using the Frame copy algorithm. The resultant video sequences have been analyzed using metrics like: PSNR, MSE, SSIM [4], blockiness and blurriness and the results have been tabulated and plotted.
40
Results of Frame Copy algorithm
Results for Akiyo.yuv The error concealment seen in the decoded output sequence of the erroneous Akiyo sequence is not perfect, as seen in the story told by fig.18. Figure 18: (a) Original Akiyo sequence frame (b) Erroneous Akiyo sequence frame (c) Error concealed Akiyo sequence frame
41
Results of Frame Copy algorithm
Results for Akiyo.yuv The frame copy algorithm has simply copied the pixels of the same location from the previous frame. With an increase in the number of frames lost, it is seen that the PSNR (fig.19), SSIM [4] (fig.20) and blurriness (fig.23) of the output sequence reduce while the MSE (fig.21) and blockiness (fig.22) increase.
42
Results of Frame Copy algorithm
Results for Akiyo.yuv Figure 19: (a) Average PSNR (dB) of the error concealed output for different error percentages (b) Average PSNR (dB) vs error percentage for the Akiyo sequence Error Percentage (%) Average PSNR of concealed result (dB) 1 30.243 2 30.240 3 30.179 4 30.09 5 29.975
43
Results of Frame Copy algorithm
Results for Akiyo.yuv Figure 20: (a) Average SSIM of the error concealed output for different error percentages (b) Average SSIM vs error percentage for the Akiyo sequence Error Percentage (%) Average SSIM of concealed result 1 2 3 4 5
44
Results of Frame Copy algorithm
Results for Akiyo.yuv Figure 21: (a) Average MSE of the error concealed output for different error percentages (b) Average MSE vs error percentage for the Akiyo sequence Error Percentage (%) Average MSE of concealed result 1 2 3 4 5
45
Results of Frame Copy algorithm
Results for Akiyo.yuv Figure 22: (a) Average blockiness of the error concealed output for different error percentages (b) Average blockiness vs error percentage for the Akiyo sequence Error Percentage (%) Original sequence Blockiness Measure Concealed sequence Blockiness Measure 1 2 3 4 5
46
Results of Frame Copy algorithm
Results for Akiyo.yuv Figure 23: (a) Average blurriness of the error concealed output for different error percentages (b) Average blurriness vs error percentage for the Akiyo sequence Error Percentage (%) Original sequence Blurriness Measure Concealed sequence Blurriness Measure 1 2 3 4 5
47
Results of Frame Copy algorithm
Results for Foreman.yuv The error concealment seen in the decoded output sequence of the erroneous Foreman sequence is not perfect, as seen in the story told by fig.24. Figure 24: (a) Original Foreman sequence frame (b) Erroneous Foreman sequence frame (c) Error concealed Foreman sequence frame
48
Results of Frame Copy algorithm
Results for Foreman.yuv The frame copy algorithm has simply copied the pixels of the same location from the previous frame. With an increase in the number of frames lost, it is seen that the PSNR (fig.25), SSIM [4] (fig.26) and blurriness (fig.29) of the output sequence reduce while the MSE (fig.27) and blockiness (fig.28) increase.
49
Results of Frame Copy algorithm
Results for Foreman.yuv Figure 25: (a) Average PSNR (dB) of the error concealed output for different error percentages (b) Average PSNR (dB) vs error percentage for the Foreman sequence Error Percentage (%) Average PSNR of concealed result (dB) 1 2 3 4 5
50
Results of Frame Copy algorithm
Results for Foreman.yuv Figure 26: (a) Average SSIM of the error concealed output for different error percentages (b) Average SSIM vs error percentage for the Forman sequence Error Percentage (%) Average SSIM of concealed result 1 2 3 4 5
51
Results of Frame Copy algorithm
Results for Foreman.yuv Figure 27: (a) Average MSE of the error concealed output for different error percentages (b) Average MSE vs error percentage for the Foreman sequence Error Percentage (%) Average MSE of concealed result 1 2 3 4 5
52
Results of Frame Copy algorithm
Results for Foreman.yuv Figure 28: (a) Average blockiness of the error concealed output for different error percentages (b) Average blockiness vs error percentage for the Foreman sequence Error Percentage (%) Original sequence Blockiness Measure Concealed sequence Blockiness Measure 1 2 3 4 5
53
Results of Frame Copy algorithm
Results for Foreman.yuv Figure 29: (a) Average blurriness of the error concealed output for different error percentages (b) Average blurriness vs error percentage for the Foreman sequence Error Percentage (%) Original sequence Blurriness Measure Concealed sequence Blurriness Measure 1 2 3 4 5
54
Results of Frame Copy algorithm
Results for Stefan.yuv For the Stefan sequence, with an increase in the number of frames lost, it is seen that the PSNR (fig.30), SSIM [4] (fig.31) and blurriness (fig.34) of the output sequence reduce while the MSE (fig.32) and blockiness (fig.33) increase.
55
Results of Frame Copy algorithm
Results for Stefan.yuv Figure 30: (a) Average PSNR (dB) of the error concealed output for different error percentages (b) Average PSNR (dB) vs error percentage for the Stefan sequence Error Percentage (%) Average PSNR of concealed result (dB) 1 2 3 4 5
56
Results of Frame Copy algorithm
Results for Stefan.yuv Figure 31: (a) Average SSIM of the error concealed output for different error percentages (b) Average SSIM vs error percentage for the Stefan sequence Error Percentage (%) Average SSIM of concealed result 1 2 3 4 5
57
Results of Frame Copy algorithm
Results for Stefan.yuv Figure 32: (a) Average MSE of the error concealed output for different error percentages (b) Average MSE vs error percentage for the Stefan sequence Error Percentage (%) Average MSE of concealed result 1 2 3 4 5
58
Results of Frame Copy algorithm
Results for Stefan.yuv Figure 33: (a) Average blockiness of the error concealed output for different error percentages (b) Average blockiness vs error percentage for the Stefan sequence Error Percentage (%) Original sequence Blockiness Measure Concealed sequence Blockiness Measure 1 2 3 4 5
59
Results of Frame Copy algorithm
Results for Stefan.yuv Figure 34: (a) Average blurriness of the error concealed output for different error percentages (b) Average blurriness vs error percentage for the Stefan sequence Error Percentage (%) Original sequence Blurriness Measure Concealed sequence Blurriness Measure 1 2 3 4 5
60
Conclusions The following are observed from the results:
The PSNR reduces with an increase in the error percentage. The SSIM reduces with an increase in the error percentage. The MSE increases with an increase in the error percentage. As compared to the input, the average blockiness of the error concealed output is higher when there is an increase in the percentage of error. This means that the error concealed output suffers from more blockiness than the input. As compared to the input, the average blurriness of the error concealed output is lower when there is an increase in the percentage of error. This means that the error concealed output suffers from more blurriness than the input.
61
Conclusions Considering Akiyo sequence to have less movement, Foreman sequence to have medium movement and Stefan sequence to have high movement, we see that: The PSNR of the concealed output decreases with an increase in movement. (From the range of 30dB in Akiyo.yuv sequence to the range of 17dB in Foreman.yuv sequence to the range of 15dB in Stefan.yuv sequence) The SSIM of the concealed output decreases with an increase in movement. (From the range of 0.92 in Akiyo.yuv sequence to the range of 0.54 in Foreman.yuv sequence to the range of 0.42 in Stefan.yuv sequence) The MSE of the concealed output increases with an increase in movement. (From the range of 60 in Akiyo.yuv sequence to the range of 1220 in Foreman.yuv sequence to the range of 1730 in Stefan.yuv sequence)
62
Future Work The current JM reference software (19.0) [3], implements only two error concealment algorithms. They are - Frame Copy algorithm and Motion Vector Copy algorithm. These are basic spatial and temporal error concealment algorithms. There is a need for implementation of better and probably hybrid concealment algorithms. Also, the 19.0 JM software [3] has poor error resiliency. This is seen in the fact that it cannot decode sequences which have more than a 5% frame loss error. It is true that transmission error losses in wireless networks seldom exceed this 5% threshold, but there can be exceptional cases.
63
References [1] I. C. Todoli, “Performance of Error Concealment Methods for Wireless Video,” Ph.D. thesis, Vienna University of Technology, [2] V.S. Kolkeri "Error Concealment Techniques in H.264/AVC, for Video Transmission over Wireless Networks", M.S. Thesis, Department of Electrical Engineering, University of Texas at Arlington, Dec Online: [3] H.264/AVC Reference Software Download: Online: [4] Z. Wang, “The SSIM index for image quality assessment,” Online:
64
References [5] Video Trace research group at ASU, “YUV video sequences,” Online: [6] MSU video quality measurement tool: Online: [7] S. –K. Kwon, A. Tamhankar and K.R. Rao, “An overview of H.264 / MPEG-4 Part 10,” J. Visual Communication and Image Representation, vol. 17, pp , Apr [8] Y. Xu and Y. Zhou, “H.264 Video Communication Based Refined Error Concealment Schemes,” IEEE Transactions on Consumer Electronics, vol. 50, issue 4, pp. 1135–1141, Nov
65
References [9] M. Wada, “Selective Recovery of Video Packet Loss using Error Concealment,” IEEE Journal on Selected Areas in Communication, vol. 7, issue 5, pp , June [10] Y. Chen et al, “An Error Concealment Algorithm for Entire Frame Loss in Video Transmission,” IEEE Picture Coding Symposium, Dec [11] S. K. Bandyopadhyay et al, “An error concealment scheme for entire frame losses for H.264/AVC,” IEEE Sarnoff Symposium, pp. 1-4, Mar [12] H. Ha, C. Yim and Y. Y. Kim, “Packet Loss Resilience using Unequal Forward Error Correction Assignment for Video Transmission over Communication Networks,” ACM digital library on Computer Communications, vol. 30, pp , Dec
66
References [13] T. Wiegand et al, “Overview of the H.264/AVC video coding standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, July [14] Z. Wang et al, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp , Apr [15] D. Levine, W. Lynch and T. Le-Ngoc, “Observations on Error Detection in H.264,” 50th IEEE Midwest Symposium on Circuits and Systems, pp , Aug [16] B. Hrušovský, J. Mochná and S. Marchevský, “Temporal-spatial Error Concealment Algorithm for Intra-Frames in H.264/AVC Coded Video,” 20th International Conference Radioelektronika, pp. 1-4, Apr
67
References [17] W. Kung, C. Kim and C. Kuo “Spatial and Temporal Error Concealment Techniques for Video Transmission Over Noisy Channels,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 16, issue 7, pp , July 2006.
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.