1. Transform-based Non-local Methods for Image Restoration IT530, Lecture Notes

2. Non-local Techniques Natural images have a great deal of redundancy: patches from different regions can be very similar NL-Means: a non-local pixel- based method (Buades et al, 2005) Awate and Whitaker (PAMI 2007) Popat and Picard (TIP 1998) De-Bonet (MIT Tech report 1998) Wang et al (IEEE SPL 2003) 2 Difference between patches

3. Transform-based Techniques DCT coefficients of a typical natural image Observed property of natural images: Sparse coefficients in Fourier/DCT/ Wavelet domain 3

4. Transform-based Techniques Project Patch onto Transform Basis (DCT/Wavelet etc.) Transform Coefficients Manipulate coefficients of noisy patch coefficients of filtered patch Get filtered patch by inversion of the transform from filtered coefficients Repeat for all patches in sliding window fashion: final filtered image obtained by averaging overlapping filtered patches. Noisy Patch Clean Patch 4 DCT Coefficients (absolute value) Transform basis Patch matrix

5. Non-local transform based techniques Machine learning approach. Make use of non-local self-similarity at the patch level to learn good transform bases for denoising.

6. Non-local PCA PCA is the most basic transform to learn! Assume Gaussian noise (mean zero, known variance). Given a patch (called reference patch) in a noisy image, search for similar patches elsewhere in the image. Given such a collection of patches, compute their PCA bases. Compute the eigen-coefficients of the reference patch.

7. Non-local PCA Manipulate these eigen-coefficients (by hard- thresholding or Wiener filter). Reconstruct the reference patch. Repeat the procedure for every patch in the noisy image in sliding window fashion. Average the multiple hypotheses that appear at a pixel. Ref: Muresan and Parks, Adaptive principal components for image denoising, ICIP 2003.

8. Criterion: Patch Similarity Two patches: identical modulo noise Cumulative of chi-squared is incomplete gamma function. A patch P 1 is considered similar to patch P ref if their squared difference is less than or equal to 8

9. Non-local PCA Noisy patch The l-th coefficient of y i – computed by projection onto the l-th local PCA bases (obtained from similar patches) We are modelling as zero-mean Gaussian random variable with variance

10. Two-stage non-local PCA Ref: Zhang et al, Two stage image denoising by principal components analysis with local pixel grouping

13. BM3DPCA KSVD

14. BM3DPCA KSVD

15. BM3DPCA KSVD

16. Time complexity The patches of size p x p are treated as vectors of size p 2 x 1. Hence the covariance matrices are of size p 2 x p 2. Eigen-analysis will have a time complexity of O(p 6 ) per patch (not counting the time to search for similar patches). This is quite expensive.

17. Non-local collaborative filtering: Block Matching in 3D (BM3D) The state of the art method today. Based on the idea of non-local similarity at the patch-level. Given a reference patch in the noisy image, this method again collects similar patches. But this time, the similar patches and the reference patch are arranged in the form of a 3D stack (of say some K patches in all).

18. Non-local collaborative filtering: Block Matching in 3D (BM3D) The stack is projected onto 3D transform bases (typically 3D DCT, or tensor product of 2D DCT and 1D Haar wavelets). The 3D transform coefficients are manipulated – usually by hard thresholding using the universal threshold. All the patches in the entire stack are reconstructed using an inverse 3D transform. This is repeated for every patch in the image. The multiple answers appearing at any pixel are averaged.

19. BM3D – second stage The preceding steps form the first stage of BM3D. The output image of the first step is used to compute patch similarities (this will be more robust than computing the similarities in the noisy image). Patches from the first-stage image are then appropriately assembled into a stack. Corresponding patches from the noisy image are assembled into a second stack.

20. BM3D – second stage 3D transform coefficients of both the stacks are computed. The second stack is denoised using Wiener filtering as follows: This is again repeated in sliding-window fashion with averaging.

21. Note: BM3D does allow individual patches from the patch to be independently filtered. It filters the similar patches from the stack collectively, assuming a dependence between them! This is the reason for its superior performance.

22. Results State of the art denoising method. Impressive results even at high noise levels. Residual images produced are very noisy. Preserves textures and fine details/edges very well.

24. NL-SVD: PSNR 30.88 SSIM 0.882 Noisy PSNR: 22.13 BM3D1: PSNR 31.02 SSIM 0.884 HOSVD: PSNR 31.53 SSIM 0.897 BM3D2: PSNR 31.66 SSIM 0.903 NLMeans: PSNR 29.42 SSIM 0.821 KSVD: PSNR 30.762 SSIM 0.877 24

25. NL-SVDBM3D1BM3D2HOSVDKSVD 25

26. NL-SVD: PSNR 30.187 SSIM 0.801 Noisy PSNR: 22.13 BM3D1: PSNR 30.395 SSIM 0.809 HOSVD: PSNR 30.491 SSIM 0.814 BM3D2: PSNR 30.8 SSIM 0.824 NLMeans: PSNR 28.91 SSIM 0.753 KSVD: PSNR 30.36 SSIM 0.803 26

27. NL-SVDBM3D1BM3D2HOSVDKSVD 27

28. 28 Gaussian Noise sigma = 15