1. HEVC DEBLOCKING FILTER
SUBMITTED BY:
ARPITA DHIRENKUMAR YAGNIK ( )
HARSHA NAGATHIHALLI JAGADISH ( )
ROHITH REDDY ETIKALA ( )

2. TABLE OF CONTENTS What are Blocking Artifacts?
Why do blocking artifacts occur?
How the blocking artifacts are removed?
H.264 Deblocking filter
H.264 Deblocking filter logic
Comparison of H.264 and HEVC Block structures and Deblocking filters
HEVC in-loop filters
HEVC Encoder
HEVC Decoder
HEVC Block structure
HEVC Deblocking filter
Filtering decisions
Algorithm for deblocking logic
Filtering operations for luminance component
Chroma Deblocking
Clipping
Sequence and picture level adaptivity
Parallelization of HEVC deblocking filters
Advantages of parallelization
References

3. Acronyms
AVC: Advanced Video Coding BS: Boundary Strength CODEC: COder/DECoder Croma: Chrominance CTU: Coding Tree Unit CU: Coding Unit DCT: Discrete Cosine Transform DFT: Discrete Fourier Transform HEVC: High Efficiency Video Coding ITU-T: International Telecommunication Union (Telecommunication Standardization Sector) IEC: International Electrotechnical Commission ISO: International Standards Organization JBIG: Joint Bi-level Image Experts Group JPEG: Joint photographic experts group JCT-VC: Joint collaborative team on video coding LOT: Lapped Orthogonal Transform Luma: Luminance MB: Macro Block MPEG: Moving picture experts group OBMC: Overlapped Block Motion Compensation PU: Prediction Unit QP: Quantization Parameter SAO: Sample Adaptive Offset TU: Transform Unit

4. What are blocking artifacts?
Blocking artifacts are visible discontinuities occurring at block boundaries in the reconstructed signal of the coding scheme that uses block based prediction and transform coding.
These artifacts are very annoying especially at low bit rates.

5. Contd…
With blocking artifacts
The effect of blocking artifacts [2]

6. Why do Blocking Artifacts occur?
Modern video coding standards try to remove as much redundancy from the coded representation of video as possible.
For these they use block based motion prediction and transform coding.
These blocks are coded relatively independently from the neighboring blocks and approximate the original signal with some degree of similarity.
Since coded blocks only approximate the original signal, the difference between the approximations may cause discontinuities at the prediction and transform block boundaries.
[2]
[28]

7. How the blocking artifacts are removed?
Approaches to reduce blocking artifacts are implemented as deblocking algorithms or filters.
Four types of deblocking algorithms can be classified as
In-loop filtering
Post processing
Pre-processing
Overlapped block methods

8. Contd…
In-loop filtering: The in-loop filtering method inserts deblocking filter into encoding and decoding loop of the video CODEC
Post-processing: Post processing algorithms apply some low pass filters and algorithms after the image or video is decoded to improve the image or video quality.
Pre-Processing: The pre-processing algorithms pre-process the video and image so that the quality of reconstructed image and video is maintained.
Overlapped Block :The overlapped block method includes Lapped Orthogonal Transform(LOT) whose transform bases are overlaid to each other and Overlapped Block Motion Compensation(OBMC) which consider the neighboring blocks for motion estimation and motion compensation in video coding.
H.264 and HEVC use in-loop deblocking filters.

9. Block diagram of H.264 Encoder[2]
H.264 Deblocking filter
-Chapter 2:Understanding the application,An overview of H.264 Standard
Block diagram of H.264 Encoder[2]

10. H.264 Deblocking filter logic
Deblocking filter is applied to the pixels around the block edges, to smoothen the block transitions.
The filter strength is adaptable through several syntax elements.
The deblocking filter is applied before the frame is used as the reference frame, which improves the efficacy of motion compensation.
Deblocking filter is applied on the 4x4 blocks.

11. Contd…
[2]
Additionally H.264 defines two more parameters α and β to determine the type of filter to be applied.[2]

12. Contd…
[21]
[21]
[21]

13. Comparison of H.264 and HEVC Block structures and Deblocking filters
H.264 CODEC
HEVC CODEC
1)H.264 divides a picture into fixed size macroblocks of 16x16 samples.
1)HEVC divides a picture into coding tree units(CTU) of 16x16,32x32, or 64x64 samples. The CTU can further be divided into smaller blocks using a quad tree structure, called a coding unit(CU), can further be split into prediction units(PU) and is also a root of transform quad tree. Each of the child nodes of the transform quad tree defines a transform unit(TU).
2)H.264 uses 4x4 and 8x8 transforms.
2)The size of transforms used in the prediction error coding can vary from 4x4 to 32x32.
3) Deblocking filter is applied to the all 4x4 blocks except the picture boundaries.
3) Deblocking filter is applied on the 8x8 blocks.
4) Deblocking filter constitutes the significant part of decoder complexity.
4)HEVC Deblocking filter is less complex as compared to H.264.
5)Deblock filtering order is complex.
5)The deblock filtering order is made less complex as compared to H.264

14. Contd… Filtering order complexity comparison between H.264 and HEVC
[28]

15. HEVC in-loop filters
The HEVC standard specifies two in-loop filters, a deblocking and a SAO(Sample Adaptive Offset) filter.
Encoding and decoding loops employ in-loop filters after the inverse quantization and before saving the decoded picture.
Deblocking filter is applied at the first place to avoid discontinuities at the prediction and transform block boundaries.
The second in-loop filter applied is the SAO filter. It is applied to the output of deblocking filter to further improve the quality of the decoded picture by attenuating the ringing artifacts and change in sample intensity of some areas of picture.

16. HEVC Encoder
In-loop filters
HEVC Encoder Diagram
[28]

17. HEVC Decoder
HEVC Decoder Block Diagram
[28]

18. HEVC Block structure
HEVC divides a picture into coding tree units(CTU) of 16x16,32x32, or 64x64 samples. The CTU can further be divided into smaller blocks using a quad tree structure, called a coding unit(CU), can further be split into prediction units(PU) and is also a root of transform quad tree.Each of the child nodes of the transform quad tree defines a transform unit(TU).The size of transforms used in the prediction error coding can vary from 4x4 to 32x32 samples.
[32]

19. Contd…
The optimal size of the above mentioned blocks depends typically on the picture content. The reconstructed picture is composed of blocks of various sizes, each block being coded using individual prediction mode and the prediction error transform.
This results in visual discontinuities or the blocking effects.

20. HEVC Deblocking filter
The HEVC draft standard defines two in-loop filters that can be applied sequentially to the reconstructed picture. The first one is deblocking filter and the other is SAO( Sample Adaptive Offset) filter.
Deblocking filter is applied to both vertical and horizontal boundaries of the block.

21. Contd… Characteristics of HECV deblocking filters
- The deblocking filter in HEVC has been designed to improve the subjective quality while reducing the complexity.
- HEVC deblocking filter is sustainable to parallel processing. It has been designed in a way to prevent spatial dependencies across the picture, which together with design features, enables easy parallelization.

22. Contd… Challenges to design deblocking algorithm
-Blocking artifacts can be noticed easily by human visual system when the signal on both the sides of block boundary is smooth, but are difficult to notice when signal shows high variation.
-If the original signal across the block boundary is subjected to higher variations then it is difficult to say whether the changes in the reconstructed signal across the block boundary are caused by coding or belong to the original signal.
-Hence main difficulty in designing a deblocking filter is to decide whether or not to filter a particular block boundary and to decide on filtering strength to be applied. Excessive filtering may leave blocking artifacts that would reduce the quality of picture.

23. Contd…
Original image
Blocking effect
[1]
Blurred image

24. Filtering Decisions Block Boundaries
Deciding whether to filter a block boundary depends on the characteristic of reconstructed pixel values on both sides of the block boundary and on the coding parameters indicating whether it is likely that a blocking artifact has been created by the coding process.

25. Contd…
Filtering decisions are made separately for each boundary of sample length that lies on the grid dividing the picture into blocks of 8x8 samples. Only boundaries of 8x8 grid which were either prediction unit or transform unit boundaries are subjected to deblocking.
[1]

26. Contd…
[1]
[28]

27. Contd…
Hence deblocking is performed on four-sample part of a block boundary when all the following criteria are true:
-The block boundary is a prediction unit or transform unit boundary.
-The boundary strength is greater than zero
-variation on both the sides of the block boundary is below specified threshold.
When certain additional conditions hold, a strong filter is applied on the edge of the block instead of normal deblocking filter.

28. Algorithm for deblocking logic
For each edge of 8x8 grid determine the filter strength (Bs)
According to the filter strength and the average quantization parameter (QP) determine two thresholds: tC and β
3. According to the values of edge pixels
and tC , β modify (if needed) the pixels.

29. Contd… Boundary strength and Edge level adaptivity
Boundary strength(BS) is calculated for boundaries that are either prediction unit boundaries or the transform unit boundaries.
The boundary strength can take values as 0,1 or 2.
For luma component, only block boundaries with BS values 1 or 2 are filtered. Therefore there is no filtering in static areas which avoids multiple subsequent filtering of same areas where pixels are copied from one picture to another with a residual equal to zero which can cause over smoothening.
In case of chroma components, only boundaries with BS equal to 2 are filtered. Hence the block boundaries that are filtered have at least one of the two adjacent blocks intra predicted.

30. Contd…
[1]
[1]
Four pixel long vertical block boundary formed by pixels P and Q.
Deblocking decisions are based on first and last line marked
[1]

31. Coding logic for boundary strength selection
[22]

32. Contd… Local Adaptivity and Filtering Decisions
If BS>0, additional conditions are checked for luma block edges to determine whether to apply the filtering.
Where, threshold β depends on Quantizer Parameter (QP) that is used to adjust quantization step for quantizing the prediction error coefficients.
-The equation evaluates how much the signal on both sides of block boundary deviates from a straight line(a constant level signal or ramp).
-Only the first and fourth lines in the block boundary are considered for reducing complexity.

33. Coding logic for filter on/off decision
[1]
[1]
[22]

34. Contd…
First the filter is applied to the vertical block boundaries and then the horizontal filtering is taken up.
Equation (1) is true for block boundaries which have BS>0.
HEVC has two deblocking filter modes
-Normal filtering mode
-Strong filtering mode

35. Contd… Decisions between normal and strong deblocking
Where, tC depends on QP
-Whether to apply normal or strong deblocking, is also determined based on the first and the fourth lines across the block boundary of four samples.
-If Equations (2),(3),(4) hold for both the lines 0 and 3,strong filter is applied.
-(2) checks for low spatial activity on the side of block boundary
-(3) checks that the signal on the sides of block boundary is flat
-(4) checks the difference in intensities of samples on two sides of the block boundary does not exceed the threshold.

36. Contd…
Deblocking decision flow chart
[1]

37. Coding logic to decide strong/normal filtering
[1]
[22]

38. Contd…
[1]
[22]

39. Filtering Operations for luma component
Normal filtering operations
Normal filtering also has two modes of operation differing in the number of pixels being modified on each side of the block boundary.
The mode is selected based for each boundary based on following conditions:
If equation (5) is true, the two nearest pixels to the block boundary can be modified in block P. otherwise only the nearest pixel in block P can be modified.
Similarly if equation (6) holds true the pixels of block Q are modified in the same manner.

40. Contd…
In normal filtering mode for a segment of 4 lines, filtering operations are applied for each line.
The filtered pixel values are calculated for each line across the block boundary as follows:

41. Contd…
The offset value δ0 corresponds to the deviation of signal at the block boundary from perfect ramp.
If the signal forms a ramp across the block boundary, the offset is zero.
Deblock filtering is applied to the row and column boundary if and only if following equation holds true.

42. Contd…
Decision logic for normal filtering modes
[1]
[28]

43. Contd…
If equation (10) does not hold true it is likely that the change of the signal on both sides of block boundary is caused by a natural edge and not by blocking artifact.
If equation (5) and equation (6) are true the pixels P and Q are modified as follows respectively

44. Coding logic for normal deblocking
[28]
[22]

45. Contd… Strong filtering Operations
Strong filtering affects more number of pixels on each side of the block boundary
The offset values are added to pixels after clipping δs.
The offset values for modification of pixels q0, q1, q2 are calculated exchanging p and q in above equations.
[28]

46. Coding logic for strong deblocking
[22]
[28]

47. Chroma Deblocking
Chroma deblocking is only performed if the BS value is equal to 2.
Only pixel P0 and q0 are modified as per equation (7) and equation (8) and the deblocking is performed with ΔC value which is clipped using offset

48. clipping
Deblocking filtering is done on a signal after QP dependent clipping to avoid excessive blurriness.
Clipping is applied to δ values before modification of pixel values.
The Δ values used in filtering are obtained by clipping δ values to the range (-c,c).
where,
C = tC (n) for p0 and q0 and tC (n) /2 for p1 and q1 in normal filtering
2 tC (n) for strong filtering mode
n = QP when both adjacent blocks are inter predicted
(QP+2) if one of the blocks is intra predicted
The blocking artifacts strength is generally greater for intra predicted blocks. Therefore larger modification of pixels is allowed for intra predicted blocks.
The filtered pixel values p0’, q0’, p1’ and q1’ for normal and p0’, q0’ for chroma are also clipped to stay in range defined by bit depth N.

49. Full flow chart for luma as well as chroma deblocking
[19]

50. Deblocking filter in HM code
Library: TlibCommon
Source file: TComLoopFilter.cpp
Class: TComLoopFilter (calls deblocking function for every CU)
Function: xDeblockCU()- performs deblocking

51. Sequence and picture level adaptibility
The main source of blocking artifacts are block transforms and quantization. Therefore blocking artifact severity depends on the quantization parameter to a large extent and so QP value is taken into consideration in deblocking filter decisions.
Thresholds β and depend tC on average QP value of two neighboring blocks with common edge.
Dependence of β on QP[1]
Dependence of tC on QP[1]

52. Contd…
As seen clearly the β increases with QP. Therefore deblocking is enabled more at high QP as compared to low QP values. Deblocking is effectively disabled for low QP values by setting both thresholds to zero.
The parameter β controls what edges are filtered and selection between normal and strong filtering along with controlling the number of pixels to modify in normal filtering mode.
While the parameter tC controls the selection between normal and strong filtering modes and determines the max absolute value of modifications are allowed to the pixel values for a certain QP.
Hence it provides adaptivity and limits the amount of blurriness introduced by deblocking filters.

53. Parallelization of HEVC Deblocking filters
Deblocking in HEVC has been designed to prevent spatial dependencies of the deblocking process across the picture.
There is no overlap between the filtering operations for one block edge, which can modify 3 pixels ,and the filtering decisions for the neighboring parallel block edge, which involves at most 4 pixels from block edge.
Hence any vertical block edge in the picture can be deblocked in a parallel way to any other vertical edge. The same holds for horizontal edges.
The picture is divided into non overlapping 8x8 blocks of samples.
Each of those 8x8 blocks contains data for deblocking. consequently deblocking can be performed independently for each of the 8x8 blocks.
Moreover the order of vertical and horizontal filtering for each block is exactly the same irrespective of the block position.

54. Advantages of parallel processing
Reduced hardware complexity as the order of filtering the block boundaries does not change with different orders of CTU decoding.
Useful for parallel processing on multi core processors.
Improves throughput and greatly reduces the bandwidth requirement for multicore based HEVC implementation.
Highly parallelized HEVC deblocking filter provides enough cycle margins to enable a combination of deblocking filter and SAO in the same building block.

55. References
Andrey Norkin et al.,”HEVC Deblocking Filter”, IEEE Transactions on CSVT, vol. 22, no. 12, pp , Dec. 2012
Wei-Yi Wei, “Deblocking Algorithms in Video and Image Compression Coding”, National Taiwan University, Taipei, Taiwan, ROC
B. Bross, et al., High Efficiency Video Coding (HEVC) Text Specification Draft 8, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVTC-J1003, Joint Collaborative Team on Video Coding (JCTVC), Stockholm, Sweden, Jul
ITU-T and ISO/IEC JCT 1, Advanced Video Coding for Generic Audiovisual Services, ITU-T Rec. H.264 and ISO/IEC (AVC), May 2003 (and subsequent editions).
T. Wedi and H. G. Musmann, “Motion and aliasing compensated prediction for hybrid video coding,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 577–586, Jul
P. List, et al., “Adaptive deblocking filter,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 614–619, Jul
K. Ugur, K. R. Andersson, and A. Fuldseth, Video Coding Technology Proposal by Tandberg, Nokia, and Ericsson, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-A119, Joint Collaborative Team on Video Coding (JCTVC), Dresden, Germany, Apr
A. Norkin, et al., CE12: Ericsson’s and MediaTek’s Deblocking Filter, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-F118, Joint Collaborative Team on Video Coding (JCTVC), Turin, Italy, Jul
M. Ikeda and T. Suzuki, Non-CE10: Introduction of Strong Filter Clipping in Deblocking Filter, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-H0275, Joint Collaborative Team on Video Coding (JCTVC), San Jose, CA, Feb
M. Ikeda, J. Tanaka, and T. Suzuki, CE12 Subset2: Parallel Deblocking Filter, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-E181, Joint Collaborative Team on Video Coding (JCTVC),Geneva, Switzerland, Mar

56. Contd…
M. Narroschke, S. Esenlik, and T. Wedi, CE12 Subtest 1: Results for Modified Decisions for Deblocking, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-G590, Joint Collaborative Team on Video Coding (JCTVC), Geneva, Switzerland, Nov
A. Norkin, CE10.3: Deblocking Filter Simplifications: Bs Computation and Strong Filtering Decision, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-H0473, Joint Collaborative Team on Video Coding (JCTVC), San Jose, CA, Feb
A. Fuldseth, et al., Tiles, ITU-TSG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-F335,Joint Collaborative Team on Video Coding (JCTVC), Turin, Italy, Jul
T. Yamakage, et al.,CE12: Deblocking Filter Parameter Adjustment in Slice Level, ITUT SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVCG174,Joint Collaborative Team on Video Coding (JCTVC), Geneva, Switzerland, Nov. 2011
G. Van der Auwera,et al. (Panasonic), Support of Varying QP in Deblocking, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVCG1031,Joint Collaborative Team on Video Coding (JCTVC), Geneva, Switzerland, Nov
M. Zhou, O. Sezer, and V. Sze, CE12 Subset 2: Test Results and Architectural Study on De-Blocking Filter Without Parallel on/off Filter Decision, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-G088, Joint Collaborative Team on Video Coding (JCTVC), Geneva, Switzerland, Nov
G. Bjontegaard, Calculation of Average PSNR Differences Between RDCurves,ITU-T-T SG16 document VCEG-M33, Joint Collaborative Team on Video Coding (JCTVC), 2001.
F. Bossen, Common Test Conditions, JCTVC-H1100, Joint Collaborative Team on Video Coding (JCTVC), San Jose, CA, 2012.
Po-Kai Hsu and Chung-An Shen, The VLSI Architecture of a Highly Efficient Deblocking Filter for HEVC Systems, DOI /TCSVT , IEEE Transactions on Circuits and Systems for Video Technology
HEVC presentation:
Overview of H.264/AVC:
Detailed overview of HEVC/H.265:

57. Contd…
I.E.G. Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, 2002.
I.E.G. Richardson, “The H.264 advanced video compression standard”, 2nd Edition, Hoboken, NJ, Wiley, 2010.
K. Sayood, “Introduction to Data compression”, Third Edition, Morgan Kaufmann Series in Multimedia Information and Systems, San Francisco, CA, 2005.
V. Sze and M. Budagavi, “Design and Implementation of Next Generation Video Coding Systems (H.265/HEVC Tutorial)”, IEEE International Symposium on Circuits and Systems (ISCAS), Melbourne, Australia, June 2014.
V. Sze, M. Budagavi and G.J. Sullivan (Editors), “High Efficiency Video Coding (HEVC): Algorithms and Architectures”, Springer, 2014.
G. J. Sullivan et al, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Trans. on Circuits and Systems for Video Technology, Vol. 22, No. 12, pp , Dec
G. J. Sullivan et al ,“Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, vol. 7, pp , Dec
K.R. Rao, D.N. Kim and J.J. Hwang, “Video Coding Standards: AVS China, H.264/MPEG-4 Part 10, HEVC, VP6, DIRAC and VC-1”, Springer, 2014.
D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard (Version 2) including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial) Sunday 27 Sept 2015, 9:00 am to 12:30 pm), IEEE ICIP, Quebec City, Canada, 27 – 30 Sept
Generic quadtree based approach for block partitioning
The tutorial below is for personal use only [Password: a2FazmgNK ]
Please find the links to YouTube videos on the tutorial - HEVC/H.265 Video Coding Standard including the Range Extensions Scalable Extensions and Multiview Extensions below:
HEVC tutorial by I.E.G. Richardson:
“Special issue on HEVC extensions and efficient HEVC implementations”, IEEE Trans. on Circuits and Systems for Video Technology, Vol. 26, pp , Jan
K.R. Rao and J.J. Hwang, “Techniques and standards for image/video/audio coding”, Prentice Hall, 1996.

58. Contd… Video lectures from IITs and IISC: http://nptel.iitm.ac.in/
Image and video processing courses at UT Arlington (EE 5351, EE 5355, EE 5356 and EE 5359) :
HEVC chapter 1:
Online course on fundamentals of digital image and video processing from Coursera:
Access to HM 16.0 Software Manual:
Test Sequences: ftp://ftp.kw.bbc.co.uk/hevc/hm-11.0-anchors/bitstreams/
HEVC white paper-Ittiam Systems:
HEVC white paper-Elemental Technologies:
Access to HM 16.0 Reference Software:
Han W-J, et al. (2010), “Improved video compression efficiency through flexible unit representation and corresponding extension of coding tools”, IEEE Trans. on Circuits and Systems for Video Technology, Vol. 20, no.12, pp , Dec
Norkin A (2012) Non-CE1: non-normative improvement to deblocking filtering, Joint Collaborative Team on Video Coding (JCT-VC), Document JCTVC-K0289, Shanghai, Oct. 2012
Norkin A, Andersson K, Fuldseth A, Bjøntegaard G (2012) HEVC deblocking filtering and decisions. In: Proc. SPIE. 8499, Applications of Digital Image Processing XXXV, no , Oct. 2012
Norkin A, Andersson K, Kulyk V (2013) “Two HEVC encoder methods for block artifact reduction”. In: Proceedings of the IEEE international conference on visual communications and image processing (VCIP) 2013, Kuching, Sarawak, pp. 1–6, Nov. 2013
Norkin A, Andersson K, Sjöberg R (2013) AHG6: on deblocking filter and parameters signaling, Joint Collaborative Team on Video Coding (JCT-VC), Document JCTVC-L0232, Geneva, Jan. 2013

59. Thank you