1. Lecture 7
Patch based methods: nonlocal means, BM3D, K- SVD, data-driven (tight) frame

2. Outline Nonlocal methods Dictionary learning methods Low rank models
Nonlocal means
BM3D
Dictionary learning methods
K-SVD
Data-driven tight frame
Low rank models

3. Nonlocal Methods
Local means and BM3D

4. Denoising: Gaussian Smoothing Revisited* = *h h
𝑥 𝑖 = 1 𝐶 𝑖 𝑗 𝑦(𝑗) 𝑒 − 𝑖−𝑗 𝜎 2

5. Perona-Malik: Anisotropic Filtering
Edges ⇒ smooth only along edges
“Smooth” regions ⇒ smooth isotropically
gradient

6. Ideal: Spatially Adaptive Smoothing
Non uniform smoothing Depending on image content:
Smooth where possible
Preserve fine details
How?
*h3
*h2
*h1

7. Bilateral Filtering (Smith and Brady, 1997)
Formula

8. * * * Gaussian Smoothing Same Gaussian kernel everywhere
Slides taken from Sylvain Paris, Siggraph 2007
Gaussian Smoothing *
input
output * *
Same Gaussian kernel everywhere
Averages across edges ⇒ blur

9. * * * Bilateral Filtering Kernel shape depends on image content
Slides taken from Sylvain Paris, Siggraph 2007
Bilateral Filtering *
input
output * *
Kernel shape depends on image content
Avoids averaging across edges

10. Nonlocal Means (NLM): Motivation
Assume a static scene
Consider multiple images y(𝑡) at different times
The signal 𝑥(𝑡) remains constant
𝑛(𝑡) varies over time with zero mean

11. Nonlocal Means (NLM): Motivation
Average multiple images over time

12. Nonlocal Means (NLM): Motivation
Average multiple images over time

13. Nonlocal Means (NLM): Motivation
Average multiple images over time

14. Redundancy in natural images
Glasner et al. (2009)

15. Single image “time-like” denoising
Unfortunately, patches are not exactly the same
⇒ simple averaging just won’t work

16. NLM Buades, Cole and Morel (2005)
𝑤(𝑝,𝑞)
𝑤(𝑝,𝑟)
𝑤(𝑝,𝑠)
Use a weighted average based on similarity
𝑥 𝑖 = 1 𝐶 𝑖 𝑗 𝑦(𝑗) 𝑒 − 𝐺𝑆𝑆𝐷 y 𝑁 𝑖 −y 𝑁 𝑗 2 𝜎 2
𝑤 𝑖,𝑗

17. From Bilateral Filters to NLM
𝑥 𝑖 𝑁𝐿𝑀 1𝑥1 = 1 𝐶 𝑖 𝑗 𝑦(𝑗) 𝑒 − 𝑦 𝑖 −𝑦(𝑗) 𝜎 2
𝑥 𝑖 𝑁𝐿𝑀 = 1 𝐶 𝑖 𝑗 𝑦(𝑗) 𝑒 − 𝐺𝑆𝑆𝐷 y 𝑁 𝑖 −y 𝑁 𝑗 2 𝜎 2
Patch similarity

18. Performance Evaluation
Gaussian
Smoothing
Anisotropic
Filtering
Bilateral Filtering
NLM
Windowed
Weiner
Hard WT
Soft WT
Buades et al. (2005)

19. Variational Formulation of NLM
Gilboa and Osher (2008)
Nonlocal (partial) derivative
Nonlocal gradient： 𝛻 𝑤 𝑢:𝛺→𝛺×𝛺
Inner product: 𝑣 1 , 𝑣 2 ≔ Ω×Ω 𝑣 1 𝑥,𝑦 𝑣 2 𝑥,𝑦 𝑑𝑥𝑑𝑦
With such inner product, we can define divergence 𝑑𝑖 𝑣 𝑤 𝑣 𝑥 : Ω×Ω→Ω
And Laplacian

20. Variational Formulation of NLM
Gilboa and Osher (2008)
Nonlocal TV
Nonlocal ROF
Nonlocal TV- 𝐿 1

21. Variational Formulation of NLM
Gilboa and Osher (2008)

22. Application in Surface Denoising
Dong et al. (2008)

23. Application in Surface Denoising
Dong et al. (2008)

24. Application in 4D CT Reconstruction
Tian et al. (2011)

25. What’s Next? The idea of grouping sounds good ⇒ reduces mixing
Denoise = “extract the common (the signal)”
NLM: common = weighted average
Can a sparser representation do better?

26. BM3D: Dabov et al. (2007) Block Matching 3D collaborative filtering
Group patches with similar local structure (BM)
Jointly denoise each group (3D)
Smart Fusion of multiple estimates

27. BM3D: Dabov et al. (2007) R R Single BM3D Estimate Filter / R
Block matching
Inverse 3D transform
Filter /
thresholding R R R
3D grouping
Denoised 3D group
3D transform

28. BM3D: Dabov et al. (2007) For every noisy reference block:
Calculate SSD between noisy blocks
If SSD<thr ⇒ add to group

29. BM3D: Dabov et al. (2007) 3D transform 3D transform Image Patch Domain
Sparse Domain

30. BM3D: Dabov et al. (2007) Collaborative filtering
Use hard thresholding or Wiener filter
Each patch in the group gets a denoised estimate
Unlike NLM – where only central pixel in reference patch got an estimate
Filter / Thresholding R R
Noisy patches
Denoised Patches

31. BM3D: Dabov et al. (2007) Multiple BM3D Estimate R R R R R R R R
Collaborative filtering R R
thr R R
Multiple BM3D Estimate R
Collaborative filtering R
thr R R

32. BM3D: Dabov et al. (2007) R t ?
Fusion

33. BM3D: Dabov et al. (2007)
Each pixel gets multiple estimates from different groups
Naive approach Average all estimates of each pixel …. not all estimates are as good
Suggestion Give higher weight to more reliable estimates

34. 𝑤∝ 1 #𝑁𝑜𝑛 𝑍𝑒𝑟𝑜 𝐶𝑜𝑒𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡𝑠
BM3D: Dabov et al. (2007)
Give each estimate a weight according to denoising quality of its group
Quality = Sparsity induced by the denoising
Hard thresholding
𝑤∝ 1 #𝑁𝑜𝑛 𝑍𝑒𝑟𝑜 𝐶𝑜𝑒𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡𝑠
Weiner filtering
𝑤∝ 1 𝐹𝑖𝑙𝑡𝑒𝑟 2

35. BM3D: Dabov et al. (2007)
Noise may result in poor matching ⇒ Degrades de-noising performance
Improvements:
Match using a smoothed version of the image
Perform BM3D in 2 phases:
Basic BM3D estimate ⇒ improved 3D groups
Final BM3D
Variational formulation: BM3D frame
Danielyan, Katkovnik and Egiazarian. BM3D frames and variational image deblurring. IEEE TIP. 21(4):

36. BM3D: Dabov et al. (2007)
Results

37. Dictionary Learning
K-SVD, Data-Driven Tight Frame

38. Sparse Coding and Dictionary Learning

39. Dictionary Selection Which D to use? A fixed set of basis:
Steerable wavelet
Contourlet
DCT Basis ……
Data adaptive dictionary – learn from data
K-SVD ( ℓ 0 -norm)
Data-Driven Tight Frame (DDTF)