Lecture 7 Patch based methods: nonlocal means, BM3D, K- SVD, data-driven (tight) frame

Outline Nonlocal methods Dictionary learning methods Low rank models Nonlocal means BM3D Dictionary learning methods K-SVD Data-driven tight frame Low rank models

Nonlocal Methods Local means and BM3D

Denoising: Gaussian Smoothing Revisited * = *h h 𝑥 𝑖 = 1 𝐶 𝑖 𝑗 𝑦(𝑗) 𝑒 − 𝑖−𝑗 2 2 𝜎 2

Perona-Malik: Anisotropic Filtering Edges ⇒ smooth only along edges “Smooth” regions ⇒ smooth isotropically gradient

Ideal: Spatially Adaptive Smoothing Non uniform smoothing Depending on image content: Smooth where possible Preserve fine details How? *h3 *h2 *h1

Bilateral Filtering (Smith and Brady, 1997) Formula

* * * Gaussian Smoothing Same Gaussian kernel everywhere Slides taken from Sylvain Paris, Siggraph 2007 Gaussian Smoothing * input output * * Same Gaussian kernel everywhere Averages across edges ⇒ blur

* * * Bilateral Filtering Kernel shape depends on image content Slides taken from Sylvain Paris, Siggraph 2007 Bilateral Filtering * input output * * Kernel shape depends on image content Avoids averaging across edges

Nonlocal Means (NLM): Motivation Assume a static scene Consider multiple images y(𝑡) at different times The signal 𝑥(𝑡) remains constant 𝑛(𝑡) varies over time with zero mean

Nonlocal Means (NLM): Motivation Average multiple images over time

Nonlocal Means (NLM): Motivation Average multiple images over time

Nonlocal Means (NLM): Motivation Average multiple images over time

Redundancy in natural images Glasner et al. (2009)

Single image “time-like” denoising Unfortunately, patches are not exactly the same ⇒ simple averaging just won’t work

NLM Buades, Cole and Morel (2005) 𝑤(𝑝,𝑞) 𝑤(𝑝,𝑟) 𝑤(𝑝,𝑠) Use a weighted average based on similarity 𝑥 𝑖 = 1 𝐶 𝑖 𝑗 𝑦(𝑗) 𝑒 − 𝐺𝑆𝑆𝐷 y 𝑁 𝑖 −y 𝑁 𝑗 2 𝜎 2 𝑤 𝑖,𝑗

From Bilateral Filters to NLM 𝑥 𝑖 𝑁𝐿𝑀 1𝑥1 = 1 𝐶 𝑖 𝑗 𝑦(𝑗) 𝑒 − 𝑦 𝑖 −𝑦(𝑗) 2 2 𝜎 2 𝑥 𝑖 𝑁𝐿𝑀 = 1 𝐶 𝑖 𝑗 𝑦(𝑗) 𝑒 − 𝐺𝑆𝑆𝐷 y 𝑁 𝑖 −y 𝑁 𝑗 2 𝜎 2 Patch similarity

Performance Evaluation Gaussian Smoothing Anisotropic Filtering Bilateral Filtering NLM Windowed Weiner Hard WT Soft WT Buades et al. (2005)

Variational Formulation of NLM Gilboa and Osher (2008) Nonlocal (partial) derivative Nonlocal gradient： 𝛻 𝑤 𝑢:𝛺→𝛺×𝛺 Inner product: 𝑣 1 , 𝑣 2 ≔ Ω×Ω 𝑣 1 𝑥,𝑦 𝑣 2 𝑥,𝑦 𝑑𝑥𝑑𝑦 With such inner product, we can define divergence 𝑑𝑖 𝑣 𝑤 𝑣 𝑥 : Ω×Ω→Ω And Laplacian

Variational Formulation of NLM Gilboa and Osher (2008) Nonlocal TV Nonlocal ROF Nonlocal TV- 𝐿 1

Variational Formulation of NLM Gilboa and Osher (2008)

Application in Surface Denoising Dong et al. (2008)

Application in Surface Denoising Dong et al. (2008)

Application in 4D CT Reconstruction Tian et al. (2011)

What’s Next? The idea of grouping sounds good ⇒ reduces mixing Denoise = “extract the common (the signal)” NLM: common = weighted average Can a sparser representation do better?

BM3D: Dabov et al. (2007) Block Matching 3D collaborative filtering Group patches with similar local structure (BM) Jointly denoise each group (3D) Smart Fusion of multiple estimates

BM3D: Dabov et al. (2007) R R Single BM3D Estimate Filter / R Block matching Inverse 3D transform Filter / thresholding R R R 3D grouping Denoised 3D group 3D transform

BM3D: Dabov et al. (2007) For every noisy reference block: Calculate SSD between noisy blocks If SSD<thr ⇒ add to group

BM3D: Dabov et al. (2007) 3D transform 3D transform Image Patch Domain Sparse Domain

BM3D: Dabov et al. (2007) Collaborative filtering Use hard thresholding or Wiener filter Each patch in the group gets a denoised estimate Unlike NLM – where only central pixel in reference patch got an estimate Filter / Thresholding R R Noisy patches Denoised Patches

BM3D: Dabov et al. (2007) Multiple BM3D Estimate R R R R R R R R Collaborative filtering R R thr R R Multiple BM3D Estimate R Collaborative filtering R thr R R

BM3D: Dabov et al. (2007) R t ? Fusion

BM3D: Dabov et al. (2007) Each pixel gets multiple estimates from different groups Naive approach Average all estimates of each pixel …. not all estimates are as good Suggestion Give higher weight to more reliable estimates

𝑤∝ 1 #𝑁𝑜𝑛 𝑍𝑒𝑟𝑜 𝐶𝑜𝑒𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡𝑠 BM3D: Dabov et al. (2007) Give each estimate a weight according to denoising quality of its group Quality = Sparsity induced by the denoising Hard thresholding 𝑤∝ 1 #𝑁𝑜𝑛 𝑍𝑒𝑟𝑜 𝐶𝑜𝑒𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡𝑠 Weiner filtering 𝑤∝ 1 𝐹𝑖𝑙𝑡𝑒𝑟 2

BM3D: Dabov et al. (2007) Noise may result in poor matching ⇒ Degrades de-noising performance Improvements: Match using a smoothed version of the image Perform BM3D in 2 phases: Basic BM3D estimate ⇒ improved 3D groups Final BM3D Variational formulation: BM3D frame Danielyan, Katkovnik and Egiazarian. BM3D frames and variational image deblurring. IEEE TIP. 21(4):1715-28. 2012.

BM3D: Dabov et al. (2007) Results

Dictionary Learning K-SVD, Data-Driven Tight Frame

Sparse Coding and Dictionary Learning

Dictionary Selection Which D to use? A fixed set of basis: Steerable wavelet Contourlet DCT Basis …… Data adaptive dictionary – learn from data K-SVD ( ℓ 0 -norm) http://bicmr.pku.edu.cn/~dongbin/Teaching_files/%E5%9B%BE%E5%83%8F%E5%A4%84%E7%90%86%E4%B8%AD%E7%9A%84%E6%95%B0%E5%AD%A6%E6%96%B9%E6%B3%95-18-19/ClassMaterials/K-SVD.pptx Data-Driven Tight Frame (DDTF) http://bicmr.pku.edu.cn/~dongbin/Teaching_files/%E5%9B%BE%E5%83%8F%E5%A4%84%E7%90%86%E4%B8%AD%E7%9A%84%E6%95%B0%E5%AD%A6%E6%96%B9%E6%B3%95-18-19/ClassMaterials/DDTF.pdf