1. RED: Regularization by Denoising
The Little Engine that Could
Joint work with:
Michael Elad Peyman Milanfar
Yaniv Romano
Technion - Israel Institute of Technology
SIAM Annual Meeting 2017
The research leading to these results has been received funding from the European union's Seventh Framework Program (FP/ ) ERC grant Agreement ERC-SPARSE
Google Research

2. Background and Main Objective

3. This is the “simplest” inverse problem
Image Denoising: Definition
This is the “simplest” inverse problem
Measured
Clean
Noise (AWGN)
Filter
An output as close as possible to

4. Image Denoising – Can we do More?
Instead of improving image denoising algorithms
lets seek ways to leverage these “engines” in order to solve
OTHER (INVERSE) PROBLEMS

5. Prior-Art 1: Laplacian Regularization
[Elmoataz, Lezoray, & Bougleux 2008] [Szlam, Maggioni, & Coifman 2008] [Peyre, Bougleux & Cohen 2011] [Milanfar 2013] [Kheradmand & Milanfar 2014] [Liu, Zhai, Zhao, Zhai, & Gao 2014] [Haque, Pai, & Govindu 2014] [Romano & Elad 2015]…

6. Pseudo-Linear Denoising
Often times we can describe our denoiser as a pseudo- linear Filter
True for K-SVD, EPLL, NLM, BM3D and other algorithms, where the overall processing is divided into a non-linear stage of decisions, followed by a linear filtering
We may propose an image-adaptive Laplacian:
The “residual”

7. Laplacians as Regularization
Laplacian Regularization ℓ
The problems with this line of work are that:
The regularization term is hard to work with since L/W is a function of x. This is circumvented by cheating and assuming a fixed W per each iteration
If so, what is really the underlying energy that is being minimized?
When the denoiser cannot admit a pseudo-linear interpretation of W(x)x, this term is not possible to use

8. [Venkatakrishnan, Wohlberg & Bouman, 2013]
Prior-Art 2:
The Plug-and-Play-Prior (P3) Scheme
[Venkatakrishnan, Wohlberg & Bouman, 2013]

9. The P3 Scheme ℓ Use a denoiser to solve general inverse problems
Main idea: Use ADMM to minimize the MAP energy
The ADMM translates this problem (difficult to solve) into 2 simple sub-problems:
Solve a linear system of equations, followed by
A denoising step ℓ

10. P3 Shortcomings
The P3 scheme is an excellent idea, as one can use ANY denoiser, even if (⋅) is not known, but…
Parameter tuning is TOUGH when using a general denoiser
This method is tightly tied to ADMM without an option for changing this scheme
CONVERGENCE ? Unclear (steady-state at best)
For an arbitrarily denoiser, no underlying & consistent COST FUNCTION
In this work we propose an alternative which is closely related to the above ideas (both Laplacian regularization and P3) which overcomes the mentioned problems: RED

11. RED: First Steps

12. Regularization by Denoising [RED]
We suggest:
… for an arbitrary denoiser f(x)
[Romano, Elad & Milanfar, 2016]

13. We shall require f(x) to satisfy several properties as follows …
Which f(x) to Use ?
Almost any algorithm you want may be used here, from the simplest Median (see later), all the way to the state-of-the-art CNN-like methods
We shall require f(x) to satisfy several properties as follows …

14. Denoising Filter Property I
Differentiability:
Some filters obey this requirement (NLM, Bilateral, Kernel Regression, TNRD)
Others can be -modified to satisfy this (Median, K-SVD, BM3D, EPLL, CNN, …)

15. Denoising Filter Property II
Local Homogeneity: for , we have that
Filter =
Filter

16. Denoising Filter Property II
Local Homogeneity: for , we have that
Holds for state-of-the-art algorithms such as K-SVD, NLM, BM3D, EPLL & TNRD...
Filter =
Filter

17. Implication (1)
Directional Derivative:
Homogeneity

18. Looks Familiar ? We got the property
n×n Matrix
This is much more general than and applies to any denoiser satisfying the above conditions

19. Implication (2) For small
Implication: Filter stability. Small additive perturbations of the input don’t change the filter matrix
Directional Derivative

20. Denoising Filter Property III
Passivity via the spectral radius:
Passivity
Filter

21. Denoising Filter Property III
Passivity via the spectral radius:
Holds for state-of-the-art algorithms such as K-SVD, NLM, BM3D, EPLL & TNRD...
Filter

22. Directional Derivative
Summary of Properties
The 3 properties that f(x) should follow:
Why are these needed?
Differentiability
Homogeneity
Passivity
Directional Derivative
Filter Stability ?

23. RED: Advancing

24. Regularization by Denoising (RED)*
* Why not ?
Surprisingly, this expression is differentiable:
the residual
and Homogeneity

25. Regularization by Denoising (RED)
Relying on the differentiability ≽0
Passivity guarantees positive definiteness of the Hessian and hence convexity

26. RED for Linear Inverse Problems
L2-based Data Fidelity
Regularization
This energy-function is convex
Any reasonable optimization algorithm will get to the global minimum if applied correctly

27. Numerical Approach We proposed three ways to minimize this objective
Steepest Descent – simple but slow
ADMM – reveals the differences between the P3 and RED
Fixed Point – the most efficient method
Will are about to concentrate on the last one

28. Numerical Approach: Fixed Point
Guaranteed to converge due to the passivity of f(x)

29. Numerical Approach III: Fixed Pointb M b M

30. A connection to CNN M M M
While CNN use a trivial and weak non-linearity f(●), we propose a very aggressive and image-aware denoiser
Our scheme is guaranteed to minimize a clear and relevant objective function

31. So… Again, Which f(x) to Use ?
Almost any algorithm you want may be used here, from the simplest Median (see later), all the way to the state-of-the-art CNN-like methods
Comment: Our approach has one hidden parameter – the level of the noise (σ) the denoiser targets. We simply fix this parameter for now. But more work is required to investigate its effect

32. RED in Practice

33. Examples: Deblurring Uniform 9×9 kernel and WAGN with 2=2
Ground Truth
Input 20.83dB
RED+Median 25.87dB
NCSR 28.39dB
P3 +TNRD 28.43dB
RED+TNRD 28.82dB

34. Examples: 3x Super-Resolution
Degradation:
A Gaussian 7×7 blur with width 1.6
A 3:1 down-sampling and
WAGN with =5
Bicubic 20.68dB
NCSR 26.79dB
P3 +TNRD 26.61dB
RED+TNRD 27.39dB

35. Sensitivity to Parameters
RED versus P3
P3: Sensitivity to parameter changes in the ADMM

36. Sensitivity to Parameters
RED: Robustness to parameter changes in the ADMM
RED: The effect of the input noise-level to f(⋅)

37. Conclusions

38. What Have We Seen Today ?
RED – a method to take a denoiser and use it sequentially for solving inverse problems
Main benefits: Clear objective being minimized, Convexity, flexibility to use almost any denoiser and any optimization scheme
One could refer to RED as a way to substantiate earlier methods (Laplacian-Regularization and the P3) and fix them
Challenges: Trainable version? Compression?

39. Relevant Reading
“Regularization by Denoising”, Y. Romano, M. Elad, and P. Milanfar, To appear, SIAM J. on Imaging Science.
“A Tour of Modern Image Filtering”, P. Milanfar, IEEE Signal Processing Magazine, no. 30, pp. 106–128, Jan. 2013
“A General Framework for Regularized, Similarity-based Image Restoration”, A. Kheradmand, and P. Milanfar, IEEE Trans on Image Processing, vol. 23, no. 12, Dec. 2014
“Boosting of Image Denoising Algorithms”, Y. Romano, M. Elad, SIAM J. on Image Science, vol. 8, no. 2, 2015
“How to SAIF-ly Improve Denoising Performance”, H. Talebi , X. Zhu, P. Milanfar, IEEE Trans. On Image Proc., vol. 22, no. 4, 2013
“Plug-and-Play Priors for Model-Based Reconstruction”, S.V. Venkatakakrishnan, C.A. Bouman, B. Wohlberg, GlobalSIP, 2013
“BM3D-AMP: A New Image Recovery Algorithm Based on BM3D Denoising”, C.A. Metzler, A. Maleki, and R.G. Baraniuk, ICIP 2015
“Is Denoising Dead?”, P. Chatterjee, P. Milanfar, IEEE Trans. On Image Proc., vol. 19, no. 4, 2010
“Symmetrizing Smoothing Filters”, P. Milanfar, SIAM Journal on Imaging Sciences, Vol. 6, No. 1, pp. 263–284
“What Regularized Auto-Encoders Learn from The Data-Generating Distribution”, G. Alain, and Y. Bengio, JMLR, vol.15, 2014,
“Representation Learning: A Review and New Perspectives”, Y. Bengio, A. Courville, and P. Vincent, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, Aug. 2013

40. Thank You