1. Unified Loop Filter for High-performance Video Coding Yu Liu and Yan Huo ICME2010, July 19-23, Singapore

2. Outline 2 Introduction Proposed Unified Loop Filter Experimental Results Conclusion

3. Introduction Conventional Video Coding Standard –Block-based DPCM coding Transform, quantization, ME/MC, in-loop deblocking filter, entropy coding In-loop Deblocking Filter –A bank of fixed low-pass filters to alleviate blocking artifacts Assume smooth image model –thus singularities, such as edges and textures, are not handled correctly –an analysis on the gradients across the boundary is performed to check whether the filtering should be skipped to preserve image sharpness. Pre-defined filter coefficients –do not retain the frequency-selective properties or have the ability to suppress quantization noise optimally 3

4. Introduction Adaptive Wiener Filter [4-7] –Well-known optimal linear filter: adaptive post/loop filter Improve the quality of reconstructed picture degraded by compression –Pros: Guarantee the optimized objective quality restoration –Cons: Can’t efficiently improve the subjective quality if used alone; thus Have to be utilized on top of in-loop deblocking filtered picture to achieve both improved objective and subjective quality 4

5. Introduction Proposed Unified Loop Filter –Is there a way to combine the advantages of deblocking filter and Wiener filter into a unified filtering framework? –Motivation The fact: multiple sources of information loss in current video coding standards, such as H.264/AVC Quantization: the only source of information loss, prior to H.264/AVC Deblocking loop filter in H.264/AVC: another source of information loss Adaptive loop filter in KTA: –may not be able to reach the capability upper-bound of picture restoration –due to additional information loss brought by the deblocking filtering –Unified Loop Filter Reduces the number of the sources causing information loss Thus further improves the capability of picture restoration 5

6. Introduction Block Diagram of Conventional Video Codec 6

7. Introduction Block Diagram of Video Codec with Unified Loop Filter 7

8. Proposed Unified Loop Filter Order Statistics Filter –Filters utilizing order statistics information [9], improved on median filters, can effectively remove the blocking artifacts and ringing artifacts, while retaining the sharpness of edges. –Although order statistics filter is a nonlinear filter, it can be optimized to minimize the mean square error by using linear combination of ordered statistics. –Here, the order statistics filter is used to combine nonlinear enhancement filter and linear restoration filter into one unified filtering framework. Unified Loop Filter –Suppose that X=(x 1, x 2, …,x c,…, x N ) T is a support vector containing N pixels of the reconstructed picture arranged by the spatial order surrounding the central pixel x c. –The unified loop filter is constructed as follows: 8

9. Proposed Unified Loop Filter Step 1: Vector of Similarity Statistics –The support vector X is converted to form a vector of similarity statistics X’=(x’ 1, x’ 2, …, x’ N ) T by using the following equation: where f(x c,x i ) is the similarity function. Real-valued similarity functions have to satisfy the following constrains: –In this paper, the following similarity function is adopted: where σ is the spread parameter controlling the strength of similarity function. 9

10. Proposed Unified Loop Filter Step 2: Similarity-Ordered Statistics Filter –The vector of similarity statistics X’ is further ordered to form a vector of similarity-ordered statistics X n =(x’ (1),x’ (2), …, x’ (N) ) T by using the following rule: –Then the output of non-linear similarity-ordered statistics filter becomes where W n is the vector of N optimized filter coefficients. 10

11. Step 3: Unified Loop Filter –In order to improve the coding efficiency, linear Wiener filter should also be incorporated into the unified loop filter. Suppose that X l =(x 1, x 2, …,x c,…, x M ) T is the support vector of Wiener filter, the output of Wiener filter becomes where W l is a vector of M optimized filter coefficients. –Generally speaking, Wiener filter is also a kind of order statistics filter, called as spatially ordered statistics filter, because its support vector X l is constructed by arranging the M pixels in a spatial order. Therefore, nonlinear similarity-ordered statistics filter is concatenated with linear spatially ordered statistics filter, aka Wiener filter, to form the unified loop filter: where X u =(x’ (1), x’ (2), …, x’ (N), x 1, x 2, …, x M ) T and W u is a vector of M+N optimized filter coefficients. Proposed Unified Loop Filter 11 Nonlinear Part Linear Part

12. Optimization of Unified Loop Filter –The optimization of the unified loop filter falls into the classical optimization framework of least mean square error (LMSE). The solution can be obtained by solving the Wiener-Hopf equations: where x o is the original video frame. The minimization problem can be solved by the Wiener-Hopf equation, which is given by where R u,u (k,l) is the auto-correlation function of x u, which is defined as and R o,u (l) is the cross-correlation function between x o and x u, which is defined as Proposed Unified Loop Filter 12

13. Proposed Unified Loop Filter Filter Design –Two considerations in the filter design of unified loop filter: not only the subjective enhancement (for removing the blocking and ringing artifacts) but also the objective restoration (for improving the coding efficiency) –For Luma Component (Y) In unified loop filter, nonlinear part consists of one 12-tap diamond filter, and linear part consists of four kinds of different taps (1-tap, 13-tap, 25-tap, and 41-tap) diamond filters with central point symmetry 13 or Nonlinear part Linear part

14. Proposed Unified Loop Filter Filter Design –For Chroma Components (Cr/Cb) In unified loop filter, nonlinear part consists of one 4-tap diamond filter, and linear part consists of two kinds of different taps (1-tap and 13-tap) diamond filters with central point symmetry Selection of Filter Tap Type –The tap type of linear part in unified loop filter is decided by rate-distortion optimization selection within the whole frame: –The filter side information includes the filter tap type and filter coefficient quantization bits, which are encoded and transmitted to the decoder side. 14 or Nonlinear part Linear part

15. Experimental Results Test Conditions –The proposed unified loop filter has been implemented within JM11.0KTA2.4r1 reference software. The test conditions are listed as follows: Table 1. Test conditions 15 Test Sequence1280x720 progressive Coding StructureIBBPBBP… Intra Frame PeriodOnly the first frame Entropy CodingCABAC R-D OptimizationON QP I(22, 27, 32, 37) P(23, 28, 33, 38) B(24, 29, 34, 39) Transform Size4x4, 8x8 Reference Frame4 Search Range± 64 Frame Number61

16. Objective Performance Comparison Experimental Results Table 2. Coding gain comparison of different coding schemes, compared with H.264/AVC High Profile, in BD bitrate reduction (BR) and BD-PSNR gain (for Luma) 16 Sequence ALF w/o DLFALF + DLFQALF + DLFULF Δ BR (%)Δ PSNR (dB)Δ BR (%)Δ PSNR (dB)Δ BR (%)Δ PSNR (dB)Δ BR (%)Δ PSNR (dB) Bigships -4.820.13-6.930.19-8.380.23-9.950.28 City -15.490.51-15.430.52-16.320.55-17.270.58 Crew -7.390.2-10.740.28-14.430.39-12.120.33 Harbour -12.640.49-14.260.55-14.210.55-14.370.56 Jet -5.510.12-6.460.15-10.640.28-11.270.30 Night -1.460.05-5.520.21-6.320.24-6.540.25 Optis -5.750.15-5.840.14-7.870.20-12.280.32 Sailormen -7.630.20-10.240.27-11.140.30-11.770.31 Sheriff -5.890.16-7.720.22-8.900.25-10.640.30 ShuttleStart -7.730.23-7.270.21-9.360.27-9.540.28 SpinCalendar -4.110.09-7.380.19-14.310.36-11.030.29 Average -7.130.21-8.890.27-11.080.33-11.530.34

17. Experimental Results Subjective Quality Comparison 17 (a) Anchor (b) ALF w/o DLF (c) ALF + DLF (e) ULF Figure. Part (128x128) of the reconstructed SpinCalendar sequence at the 54th frame with QP=38.

18. Complexity Comparison –Compared with “ALF+DLF”, extra complexity introduced by ULF includes: Similarity computation: lookup-table (LUT) technique Sorting process: counting sort algorithm with linear complexity O(n). –With the help of the two fast algorithms ULF only increases the execution time 13.70% and 26.52% on average for the encoder and the decoder, respectively, compared with “ALF+DLF” Experimental Results Avg. Exec. Time (s) AnchorALF w/o DLFALF+DLFQALF+DLFULF EncDecEncDecEncDecEncDecEncDec Bigships1019.69512.3321116.57616.8091116.71019.0741507.43316.7151368.24723.549 City1102.10512.7891198.02916.8131197.77619.1371611.10617.7071467.94523.794 Crew1010.66411.7461109.33615.3471109.75818.1451501.56115.9021173.82623.790 Harbour1158.65614.2341251.54518.1521253.10221.3601654.65221.0551396.64926.969 Jet914.74311.8401011.42113.8641012.87515.4961300.52913.3521104.61819.751 Night1101.74312.4451199.93216.9851197.75619.2191580.08716.7151292.45026.666 Optis995.51912.4611095.31215.6831093.75417.5321450.33316.0351259.53421.738 Sailormen1102.99714.0321197.46318.0001199.27220.3091612.11718.3241352.91125.752 Sheriff1006.42712.9451103.94316.5081105.29218.7931468.72817.2581258.96923.923 ShuttleStart874.49410.856973.00012.438973.94514.8281246.39313.0081085.11517.301 SpinCalendar1117.88514.3291214.54116.9451217.46920.5351614.54317.2191426.57025.406 Total Average1036.81212.7281133.73616.1401134.33718.5841504.31616.6631289.71223.513 18 Table 3. Average execution time comparison of different coding schemes

19. Conclusion Unified Loop Filter (ULF) –Combine the advantages of linear filter and nonlinear filter to achieve both objective and subjective quality optimization –The deblocking loop filter in conventional video codec can be removed, and thus replaced by the proposed unified loop filter –The proposed unified loop filter can be used in any hybrid video coding system Classification-based Unified Loop Filter (CULF) –Global v.s. Local: different regions have different quantization error characteristics –It is better to classify pixels into different groups: one group for boundary pixels with blocking artifacts one group for non-boundary and boundary pixels without blocking artifacts other group for non-filtering pixels –Different unified loop filter with different characteristics is applied to each group –Additional 1.57% bitrate reduction is achieved, compared with ULF 19

20. Thank You! Q&A 20