1. Adaptive Regularization of the NL-Means : Application to Image and Video Denoising IEEE TRANSACTION ON IMAGE PROCESSING ， VOL ， 23 ， NO,8 ， AUGUST 2014 Sutour, C.Sutour, C. ; Deledalle, C.-A. ; Aujol, J.-F. “Adaptive regularization of the NL-means: Application to image and video denoising.“ IEEE Trans. Image Process., vol. 23, no. 8, pp. 3506 – 3521, Aug. 2014Deledalle, C.-A.Aujol, J.-F Camille Sutour, Charles-Alban Deledalle, and Jean-François Aujol

2. A. ROF Model [1] The usual model is the case of additive white Gaussian noise: The general problem in denoising is to recover the image based on the noised observation. ROF model [1] relies on the total variation (TV), hence forcing smoothness while preserving edges. The restored image is obtained by minimizing the following energy: [1] L. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D, vol. 60, no. 1, pp. 259–268, 1992. (cited by 7418) (1) (2) (3)

3. Original image Horizontal Vertical TV

4. Results Sigma = 20OriginalROF Shortcoming: 1、 the textures tend to be overly smoothed； 2、 the flat areas are approximated by a piecewise constant surface resulting in a staircasing effect； 3、 the image suffers from losses of contrast.

5. B. Non-local Means [2-3] [2]Buades, Antoni, Bartomeu Coll, and J-M. Morel. “A non-local algorithm for image denoising.” Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 2. IEEE, 2005.(cited by 1792 ) [3] Buades, Antoni, Bartomeu Coll, and Jean-Michel Morel. "A review of image denoising algorithms, with a new one." Multiscale Modeling & Simulation 4.2 (2005): 490-530. (cited by 1916) Fig. Scheme of NL-means strategy. Similar pixel neighborhoods give a large weight, w(p,q1) and w(p,q2), while much different neighborhoods give a small weight w(p,q3). These weights are define as, where is the normalizing constant (1) (2) (3)

6. Shortcoming Fig. Illustration of the defaults of the NL-means: the rare patch effect (red circle) can be observed around the head and the camera, while the patch jittering effect (blue circles) can be observed on the background. 1、On singular structures the algorithm might fail to find enough similar patches and thus performs insufficient denoising. This is referred to as the rare patch effect. 2、False detections. It can result in averaging several pixel values that do not truly belong to the same underlying structure, creating an over- smoothing sometimes referred to as the patch jittering blur effect.

7. C. Non-local TV [4] [4]Gilboa G, Osher S. Nonlocal operators with applications to image processing[J]. Multiscale Modeling & Simulation, 2008, 7(3): 1005-1028.(cited by 437) Define a nonlocal gradient as follows: where is the weight that measures the similarity between pixels i and j. This leads to the definition of a nonlocal framework, including the nonlocal ROF model: with (1) (2) (3) Characteristics : free of the staircasing effect but it is still subject to the rare patch effect.

8. Contributions 1、Adaptive Regularization of the NL-Means(combine TV with NL-means)  reduce the patch jittering blur effect  correct the rare patch effects  without introducing over-smoothing, staircasing or contrast losses inherent to the non-adaptive TV minimization 2、Propose a model that adapts to different noise models  Gaussian Case  Poisson Case  Gamma Case 3、Application to Video Denoising with 3D patches

9. A. Dejittering of the NL-means(NLDJ) With reference to the literature [5], [6], we locally perform a convex combination between the nonlocal estimation and the noisy data g, according to the following formula: Where is a confidence index defined by: [5]Lee, Jong-Sen. "Refined filtering of image noise using local statistics." Computer graphics and image processing 15.4 (1981): 380-389. (cited by 585) [6] Kuan D T, Sawchuk A A, Strand T C, et al. Adaptive noise smoothing filter for images with signal-dependent noise[J]. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 1985 (2): 165-177. (cited by 1055) (1) (2) (3) with

10. The solution can be rewritten as the following weighted sum where if0 otherwise. The residual variance of the dejittered solution can be approached at pixel i by: (1) (2) (3)

11. B. Regularization of the NL-Means(R-NL) The proposed model combines both the NL-means and the TV minimization: (1) (2)

13. How to set the regularization parameter

14. Regularization parameter the lower, the better!!! What's the relationship between and

17. R-NL for the exponential family Both TV and NL-means are robust to different kind of noises, the proposed model can then be extended to other types of (uncorrelated) noise with a weighted data fidelity of the form [8] A probability law belongs to the exponential family [7] if it can be written under the following form: where c, T, η and A are known functions. The extended model is then the following: [7]Collins M, Dasgupta S, Schapire R E. A generalization of principal components analysis to the exponential family[C]//Advances in neural information processing systems. 2001: 617-624.(cited by 249) [8] Polzehl, Jörg, and Vladimir Spokoiny. "Propagation-separation approach for local likelihood estimation." Probability Theory and Related Fields 135.3 (2006): 335-362.(cited by 125) (1) (2) (3)

18. A. Gaussian Case As in the Gaussian case, it can be reformulated with a weighted NL- means based fidelity term: where is the weighted log-likelihood (1) (2) In this case, solving (1) is equivalent to solving The expected nonlocal variance involved in the dejittering step is chosen as:

19. [9]Chambolle, Antonin, and Thomas Pock. "A first-order primal-dual algorithm for convex problems with applications to imaging." Journal of Mathematical Imaging and Vision 40.1 (2011): 120-145.(cited by 775)

20. B. Poisson Case The negative log-likelihood offor an observed intensity g is given by The expected nonlocal variance involved in the dejittering step is chosen as: where Q is the non-negative integer. The solution of the NL-means and the adaptive regularization parameters can then be computed accordingly and the variational problem becomes: (1) (2)

21. C. Gamma Case The negative log-likelihood offor an observed intensity g is given by where L is the “number of looks” that sets the level of the noise. The expected nonlocal variance involved in the dejittering step is chosen as: The solution of the NL-means and the adaptive regularization parameters can then be computed accordingly and the variational problem becomes: (1) (2)

22. Results A. Gaussian Case

23. B. Poisson Case

24. C. Gamma Case

26. References [1] L. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D, vol. 60, no. 1, pp. 259–268, 1992. (cited by 7418) [2]Buades, Antoni, Bartomeu Coll, and J-M. Morel. “A non-local algorithm for image denoising.” Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 2. IEEE, 2005.(cited by 1792 ) [3] Buades, Antoni, Bartomeu Coll, and Jean-Michel Morel. "A review of image denoising algorithms, with a new one." Multiscale Modeling & Simulation 4.2 (2005): 490-530. (cited by 1916) [4]Gilboa G, Osher S. Nonlocal operators with applications to image processing[J]. Multiscale Modeling & Simulation, 2008, 7(3): 1005-1028.(cited by 437) [5]Lee, Jong-Sen. "Refined filtering of image noise using local statistics." Computer graphics and image processing 15.4 (1981): 380-389. (cited by 585) [6] Kuan D T, Sawchuk A A, Strand T C, et al. Adaptive noise smoothing filter for images with signal-dependent noise[J]. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 1985 (2): 165-177. (cited by 1055) [7]Collins M, Dasgupta S, Schapire R E. A generalization of principal components analysis to the exponential family[C]//Advances in neural information processing systems. 2001: 617-624.(cited by 249) [8] Polzehl, Jörg, and Vladimir Spokoiny. "Propagation-separation approach for local likelihood estimation." Probability Theory and Related Fields 135.3 (2006): 335-362.(cited by 125) [9]Chambolle, Antonin, and Thomas Pock. "A first-order primal-dual algorithm for convex problems with applications to imaging." Journal of Mathematical Imaging and Vision 40.1 (2011): 120-145.(cited by 775) [10] Combettes, Patrick L., and Jean-Christophe Pesquet. "Proximal splitting methods in signal processing." Fixed-point algorithms for inverse problems in science and engineering. Springer New York, 2011. 185-212. (cited by 539)