1. Error Concealment Multimedia Systems and Standards S2 IF ITTelkom

2. Background Every communication system has to deal with the problems that may arise during transmission, such as adulteration (bit insertion, deletion or inversion) or loss of the transmitted signal To solve this problem, a set of techniques were developed, whose purpose was to minimize the influence of the transmission errors at the decoder, taking in consideration the characteristics of the video signal. These techniques are called Error Concealment Techniques 2

3. Definition error concealment is a technique in which an error in a transmitted or encoded signal is replaced by synthetic content, often interpolated from other parts of the signal, in an attempt to produce a more pleasant experience for the viewer or listener, at the cost of concealing the presence of the error in the original content. Error concealment by post-processing at the decoder is one of the important strategies for combating transmission errors in video communication 3

4. Error Concealment Techniques the video decoder attempts to benefit from previously received error-free video information the approximate recovery of lost or erroneous data without relying on additional information from the encoder 4

5. Techniques To estimate and sometimes predict missing video information : Spatial and temporal interpolation, filtering and smoothing Motion vectors, transform coefficients and administrative bits 5

6. Techniques (cont’d) take advantage of the human eyes tolerance (high-frequency components than low-frequency components) multi-layer video coding: base & enhancement layers make use of the spatial and/or temporal correlations between damaged MBs and their neighbouring MBs 6

7. The goal To reduce the visual artefacts in segments of a video stream that lie between two error-free synch words. ?: where to insert synch words 7

8. Error Concealment Basic Idea Decoder should generate a representation for lost area Match as close as possible to the lost info Within manageable complexity Techniques Spatial Error Concealment Temporal Error Concealment Hybrid Concealment Other Techniques 8

9. Spatial Error Concealment Employ the correctly received data of the current frame to conceal the erroneous data Spatial error concealment methods can be categorized in four main types: interpolation based, stochastic-based, tensor voting-based and exemplar- based. 9

10. Spatial Error Concealment (Interpolation Based) conceal each corrupted pixel by interpolation; using the correctly received neighboring pixels for instance, directional interpolation methods are proposed that utilize edge directions for interpolation The weakness of interpolation-based methods is that although they perform very fast, texture preservation is not their intrinsic characteristic and should be imposed separately Example : Multi Directional Interpolation 10

11. Multi Directional Interpolation MDI is one method used to repair the damaged image or video by covering the damaged block using spatial interpolation The process of MDI are : a.Block Classification and Directional Decision b.Block Concealment 11

12. Multi Directional Interpolation Algorithm The missing 16×16 MB is recovered 4×4 block by 4×4 block in the proposed error concealment approach. The order of 16 blocks recovered in an MB is from the blocks with dependable available neighbours to the blocks without dependable available neighbours There are mainly two steps to recover each 4×4 block. In the first step, the block is determined if it is a flat block or a block with one of the eight directional edges according to the estimation of surrounding available pixels In the second step, the flat block is recovered using weighted pixel averaging and the block with edge is recovered using Intra_4×4 prediction 12

13. Multi Directional Interpolation Algorithm 13

14. Block Classification and Directional Decision The absence or presence of edges and the most likely direction of the edge, if present, of each 4×4 block in the missing MB are obtained by the use of gradient measures to three layers of pixels of the available adjacent MB as shown in Fig. 1. The local edge gradient components of the pixel x(i,j) in the second layer of the available adjacent MB is computed by: 14

15. Block Classification and Directional Decision Cont Which is equivalent to applying 3x3 sobel operator The angular of gradient at coordinate (i,j) : And the angle of θ is rounded to the nearest 22.5 degree. 15

16. Block Classification and Directional Decision Cont In order to determine if a 4x4 block is lat or with some direction of edge, eight counters C k, k = 1,....7 are used. The initial value of C k is set zero. For every pixel x(i,j) and all pixel (y(u,v) to represent any one of these pixels) in the block to be recovered,if y(u,v) satisfy : Then C k plus 1/D, where D is given by : 16

17. Block Classification and Directional Decision Cont If the largest counter value is below a certain threshold, there is no discernible edge and the block is classified to a flat block 17

18. Block Concealment Each 4×4 block with edge in the missing MB is recovered by Intra_4×4 prediction with edge according to directional decision in the first step. Each flat block is recovered by using the method of weighted pixel averaging. The number of each block in the missing MB is shown 18

19. Block Concealment Cont Since the order of 16 blocks recovered in a MB is from blocks with dependable available neighbours to those without dependable available neighbours, the order of 16 blocks to be recovered will be changed according to the scenario of available neighbour MBs For examples, if a missing MB has four available edge neighbour MBs, the order of 16 blocks to be recovered is 0, 5, 1, 4, 10, 15, 11, 14, 2, 7, 3, 6, 8, 13, 9, 12; if a missing MB only has up and right available edge neighbour MBs, the order of 16 blocks to be recovered is 5, 4, 1, 0, 7, 6, 3, 2, 13, 12, 9, 8, 15, 14, 11, 10. Furthermore, for each block from 0 to 15, the 16 pixels a, b, …, n, o, p should also be recovered by Intra_4×4 prediction from more reliable pixels X, A, B, C, …, G, H shown at fig 4. 19

20. Spatial Error Concealment (Stochastic Based) consider each pixel of the frame as a random variable employ the Bayes rule to approximate the best pixel values of the corrupted block The main disadvantage of these methods is their stationary assumption for input images. 20

21. Spatial Error Concealment (Tensor Voting Based) a tensor vector (containing the intensity and gradient information of each pixel) is calculated for each correctly decoded pixel Using a voting scheme, tensor vectors of the corrupted pixel are approximated and translated to the intensity values The main drawback of tensor voting approaches is their high computational complexity that is not affordable in real- time applications 21

22. Spatial Error Concealment (Exemplar Based) performed by cloning the best patches from the correctly received region of the image to the corrupted block. In this method, the similarity is measured by the sum-squared difference of the outer boundaries In this approach, the subblocks near the edge are concealed first. 22

23. Temporal Error Concealment Exploits temporal redudancy in a sequence to recover the impaired MacroBlocks (MBs) by utilizing reference frames. Example : Motion Conpensated Temporal Prediction 23

24. Temporal Error Concealment Rely on the continuity of a video sequence in time Use temporally neighboring areas to conceal lost regions Previous Frame Concealment (PFC) Use previous corresponding data to copy to current frame Only good when little motion Widely used due to simplicity 24

25. Hybrid Concealment When only apply spatial concealment Concealed regions are significantly blurred When only use temporal error concealment Significantly discontinuities in the concealed regions Hybrid temporal-spatial technique applied MB mode info of reliable and concealed neighbors decide which concealment method to use 25

26. Hybrid (cont.) For intra-coded images Only use spatial concealment For inter-coded images Use temporal concealment when more than half of the available neighbor MBs are inter- coded Otherwise, use spatial concealment Referred to as Adaptive temporal and spatial Error Concealment (AEC) 26

27. Error Detection To activate error concealment technique Errors: loss of synchronization due to error-corrupted VLC parameters the number of AC coefficients of any 8x8 block of pixels is found to have exceeded 63 The decoded MV component or quantization parameter is outside the acceptable range transmission errors 27

28. Recovery of lost MVs and MB coding modes = INTER Simplest: replace the erroneous MB by the spatially coinciding MB in the previous frame Alternative: replaced by the motion- compensated MB (what if MV also corrupted?) MV recovery 28

29. Example: H.263; BER=0.01 % (a) no concealment, (b) zero-MV technique, (c) MV of spatially corresponding MB in previous frame, (d) MV of MB in previous frame that best moves in the direction of the lost MV 29

30. Recovery of lost coefficients can be interpolated from spatially corresponding coefficients: four neighbours some coefficients damaged  use coefficients in the same block all coefficients are lost  using: four one-pixel wide boundaries the nearest two one-pixel wide boundaries more suitable for INTRA coded blocks 30

31. Four-one pixel wide boundaries 31

32. Based on assumption of continuity of natural scene content in space Use pixel values of surrounding available MBs Estimate of lost pixel: 32 αβγ are weighing factors Determine relative impact of vertical, Horizontal, upper, lower… Disadvantage Blurred reconstruction Four-one pixel wide boundaries

33. the nearest two one-pixel wide boundaries 33

34. Selected Results Performance of different error concealment strategies 34