Image deblocking using local segmentation By Mirsad Makalic Supervisor: Dr. Peter Tischer

Presentation Outline An introduction to JPEG  Lossy image compression  Discrete Cosine Transform (DCT) and quantization Local segmentation and prior research Measuring image quality Deblocking Filter Super-Resolution Filter Conclusion

Lossy image compression JPEG is the most common lossy image compression format  Easy to implement  Good quality with high compression ratios Block based transform approach  Divide image into 8x8 blocks  Perform discrete cosine transform (DCT) on each 8x8 block  Quantize the DCT coefficients

Discrete Cosine Transform Transform 8x8 block of pixels into a set of weighted basis functions (DCT coefficients) DC coefficient AC coefficients

DCT - Subbands An image consisting of just one coefficient from each 8x8 block is called the sub-band image An image which is divided into 8x8 blocks has 64 possible sub-band images An 8x8 image split into 2x2 blocks has 4 subbands

Quantization Exploit visual redundancy Quantization is a many-to-one mapping  Divide each DCT coefficient by a value and round to the nearest integer The decoder makes a guess from a range of values (pick midpoint by default) = 1 Quantizer = 10

Quantization Coarse quantization introduces artifacts into reconstructed image  DCT coefficients are reconstructed inaccurately Most visually distracting artifact is blockiness JPEG compressed image at quality 10 PSNR: dB

Deblocking Techniques Three approaches in the literature:  Filter the reconstructed pixel values  Attempt to reconstruct DCT coefficients more accurately  A hybrid approach

Local Segmentation Most deblocking filters introduce excessive blurring  Destroys the structure of the image  Edges lose their sharpness Local segmentation takes into account the structure of the image

Local Segmentation Divide a mask of pixels into N segments and filter each segment independently Average of whole mask = 43.22, average of yellow segment = 20.8 Two questions:  How many segments do we use?  How do we segment a mask of pixels?

Prior Research Lukasz Kizewski, BSE (hons) 2004 DC subband approach  Filter using a mask of DC subbands How do we segment a mask of pixels?  Segment the pixel mask using thresholding How many segments do we use?  Do-No-Harm heuristic

Prior Research Do-No-Harm heuristic  Try a 1-segment model (average of the whole mask) If filtered value is implausible reject and try a 2- segment model A plausible value is one which falls inside the quantization range: midpoint +/- ½ Quantum  Try a 2-segment model If still implausible then don’t filter

Prior Research Results (a) Unfiltered image(a) Filtered image

Room for improvement No objective measure used to test the effectiveness of the filter  Difficult to make comparisons  Difficult to rate changes in filter Works only on DC subband  AC subbands contain edge and texture information A very simple local segmentation method

Measuring image quality Peak-signal-to-noise-ratio  Most commonly used metric  Does not necessarily reflect the subjective visual quality Generalized Block-Edge Impairment Metric (GBIM) – H.R. Wu, M. Yuen

Measuring image quality - GBIM Measures the quality of DCT encoded images  Assume that what happens inside a block is the same as what happens across blocks Take absolute mean difference of pixels inside a block (vertical/horizontal) Take absolute mean difference of pixels across blocks (vertical/horizontal) Compare them, if the two differ greatly than it is a sign of blockiness x2 block example with vertical blockiness

A new deblocking filter Filter all coefficients  Treat each 8x8 block as a 64 element vector where each value in the vector is one of the DCT coefficients Local segmentation no longer as simple (need to segment masks of vectors)  K-Means or K-Nearest Neighbours DNH needs to work on vectors

A new deblocking filter Basic structure of filter is same:  Create an NxN mask where each item is a 64 element DCT coefficient vector  Segment mask using K-Means or K-Nearest Neighbours segmentation  Check if segmentation produces valid result using DNH, if not, try different segment  If no segmentation produces valid result, leave alone

K-Means Start with one segment (average of mask) If segmentation is invalid, increase number of segments by one until a maximum number of segments is reached Try largest change first

K-Nearest Neighbours Set the number of nearest neighbours to find as the number of items in the entire mask Keep decreasing by segment size by one until a valid segment is found

Vector DNH Center DNH  Compare the filtered vector against only the center vector in the mask Strict Segment DNH  Compare the filtered vector against all the vectors in the segment the center vector is in Lesser Strict Segment DNH  Same as strict segment DNH with some error tolerance

Results Best found parameters for the filter:  3x3 vector mask (covers 24x24 pixels)  K-Nearest Neighbour segmentation  Lesser strict segment DNH with 1% error tolerance Tested:  5x5 mask, K-Means, Center DNH, Strict DNH etc.  Many variations of the filter parameters

Results (a) JPEG compressed image(b) Filtered image An improvement of 0.07 dB in PSNR and a reduction of 0.62 in GBIM

Results (a) JPEG compressed image(b) Filtered image An improvement of 0.07 dB in PSNR and a reduction of 0.55 in GBIM

Other uses for local segmentation Super-resolution  Combine multiple slightly different images to form one higher quality image  Can we extract more information out of a single image? Neighbouring pixels are similar and share information Use local segmentation

Super-resolution filter Very similar to the deblocking filter  Instead of using a 64 element vector for each DCT coefficient, use one element vector containing each pixel value in the mask How to test if it works?  Convert 8 bit image to 4 bits and attempt to reconstruct back an 8 bit image

Results Best found parameters for filter:  3x3 mask  K-Means segmentation with up to three segments  Center DNH  More than one iteration of the filter can further improve PSNR

Results An improvement of 1.38 dB in PSNR (a) An image rounded to 4 bits per pixel (a) Filtered image (8 bits per pixel)

Results (a) An image rounded to 4 bits per pixel (a) Filtered image (8 bits per pixel) An improvement of 1.32 dB in PSNR

Conclusion Filtering AC subbands is difficult because most have been quantized to zero or have very large quantization ranges Most improvements in image quality are from the stricter DNH and better local segmentation techniques The increase in computational complexity may not be worth the increase in image quality

Conclusion Super-resolution filter shows a lot of promise  Large increase in PSNR and image quality A different DNH heuristic may work better with the super-resolution filter

Future Research Allow more variation in pixels for the strict DNH  Assume local mask is linear and not constant Try different segmentation techniques  Region growing Further investigate iterative filtering

The End Questions?