Discriminative Approach for Transform Based Image Restoration SIAM – Imaging Science, July 2008 Discriminative Approach for Transform Based Image Restoration Yacov Hel-Or Doron Shaked Gil Ben-Artzi The Interdisciplinary Center Israel Bar-Ilan Univ. Israel HP Las

Motivation – Image denoising - Can we clean Lena?

Broader Scope inverse problem Inpainting De-blurring De-noising De-mosaicing All the above deal with degraded images. Their reconstruction requires solving an inverse problem

Key point: Stat. Prior of Natural Images likelihood prior Bayesian estimation:

Problem: P(x) is complicated to model Defined over a huge dimensional space. Sparsely sampled. Known to be non Gaussian. A prior p.d.f. of a 2x2 image patch form Mumford & Huang, 2000

The Wavelet Transform Marginalizes Image Prior Observation1: The Wavelet transform tends to de-correlate pixel dependencies of natural images. W.T.

Observation2: The statistics of natural images are homogeneous. Share the same statistics

Donoho & Johnston 94 Wavelet Shrinkage Denoising: Unitary Case Degradation Model: MAP estimation in the transform domain

Modify coefficients via scalar mapping functions The Wavelet domain diagonalizes the system. The estimation of a coefficient depends solely on its own measured value This leads to a very useful property: Modify coefficients via scalar mapping functions where Bk stands for the k’th band

Shrinkage Pipe-line  x y + x= BTkk(Bky) k(Bky) Bky BTkk(Bky) BT1  B2 BT3 BT3 B2 y x= BTkk(Bky)   k(Bky) Bky BTkk(Bky) xiB yiB Image domain Transform domain Image domain Result

Wavelet Shrinkage as a Locally Adaptive Patch Based Method KxK DCT xiB yiB DCT-1 xiB yiB xiB yiB

Can be viewed as shrinkage de-noising in a Unitary Transform (Windowed DCT). KxK bands WDCT Unitary Transform xiB WDCT-1 yiB

Alternative Approach: Sliding Window KxK DCT xiB yiB DCT-1 xiB yiB xiB yiB

Can be viewed as shrinkage de-noising in a redundant transform (U. D Can be viewed as shrinkage de-noising in a redundant transform (U.D. Windowed DCT). UWDCT Redundant Transform xiB UWDCT-1 yiB

How to Design the Mapping Functions? Descriptive approach: The shape of the mapping function j depends solely on Pj and the noise variance . noise variance () Modeling marginal p.d.f. of band j MAP objective yw

Commonly Pj(yB) are approximated by GGD: for p<1 from: Simoncelli 99

Hard Thresholding Soft Thresholding Linear Wiener Filtering MAP estimators for GGD model with three different exponents. The noise is additive Gaussian, with variance one third that of the signal. from: Simoncelli 99

Due to its simplicity Wavelet Shrinkage became extremely popular: Thousands of applications. Thousands of related papers What about efficiency? Denoising performance of the original Wavelet Shrinkage technique is far from the state-of-the-art results. Why? Wavelet coefficients are not really independent.

Redundant Representation Joint (Local) Coefficient Recent Developments Since the original approach suggested by D&J significant improvements were achieved: Original Shrinkage Redundant Representation Joint (Local) Coefficient Modeling Overcomplete transform Scalar MFs Simple Not considered state-of-the-art Multivariate MFs Complicated Superior results

What’s wrong with existing redundant Shrinkage? Mapping functions: Naively borrowed from the unitary case. Independence assumption: In the overcomplete case, the wavelet coefficients are inherently dependent. Minimization domain: For the unitary case MFs are optimized in the transform domain. This is incorrect in the overcomplete case (Parseval is not valid anymore). Unsubstantiated Improvements are shown empirically.

Questions we are going to address How to design optimal MFs for redundant bases. What is the role of redundancy. What is the role of the domain in which the MFs are optimized. We show that the shrinkage approach is comparable to state-of-the-art approaches where MFs are correctly designed.

Optimal Mapping Function: Traditional approach: Descriptive Modeling marginal p.d.f. of band k MAP objective x

Optimal Mapping Function: Suggested approach: Discriminative Off line: Design MFs with respect to a given set of examples: {xei} and {yei} On line: Apply the obtained MFs to new noisy signals. Denoising Algorithm

Option 1: Transform domain – independent bands + B1 Bk B1 BTk + B1 Bk B1 BTk

Option 2: Spatial domain – independent bands + B1 Bk B1 BTk + B1 Bk B1 BTk

Option 3: Spatial domain – joint bands + B1 Bk B1 BTk + B1 Bk B1 BTk

The Role of Optimization Domain Theorem 1: For unitary transforms and for any set of {k}: Theorem 2: For over-complete (tight-frame) and for any set of {k}: =

Unitary v.s. Overcomplete Spatial v.s. Transform Domain Spatial domain > = > = Transform domain

Is it Justified to optimized in the transform domain? In the transform domains we minimize an upper envelope. It is preferable to minimize in the spatial domain.

Optimal Design of Non-Linear MF’s Problem: How to optimize non-linear MFs ? Solution: Span the non-linear {k} using a linear sum of basis functions. Finding {k} boils down to finding the span coefficients (closed form). Mapping functions k(y) y For more details: see Hel-Or & Shaked: IEEE-IP, Feb 2008

z=[0,,0,1-r(z),r(z),0,]q = Sq(z)q z=r(z) qj+(1-r(z)) qj-1 Let zR be a real value in a bounded interval [a,b). We divide [a,b) into M segments q=[q0,q1,...,qM] w.l.o.g. assume z[qj-1,qj) Define residue r(z)=(z-qj-1)/(qj-qj-1) q0 q1 qM qj-1 qj a r(z) b z z=[0,,0,1-r(z),r(z),0,]q = Sq(z)q z=r(z) qj+(1-r(z)) qj-1

The Slice Transform We define a vectorial extension: We call this the Slice Transform (SLT) of z.       ith row

The SLT Properties z’ z =Sq(z) p q Substitution property: Substituting the boundary vector q with a different vector p forms a piecewise linear mapping. q0 q1 q2 q3 q4 z p0 p1 p2 p3 p4 z’ z =Sq(z) p q z’ z q1 q2 q3 z q4

Back to the MFs Design We approximate the non-linear {k} with piece-wise linear functions: Finding {pk} is a standard LS problem with a closed form solution!

Results

Training Images

Tested Images

Simulation setup Transform used: Undecimated DCT Noise: Additive i.i.d. Gaussian Number of bins in SLT: 15 Number of bands: 3x3 .. 10x10

Option 1 Option 2 Option 3 MFs for UDCT 8x8 (i,i) bands, i=1..4, =20 MFs are non monotonic anymore ! Option 3

Why considering joint band dependencies produces non-monotonic MFs ? noisy image Unitary MF image space Redundant MF

Comparing psnr results for 8x8 undecimated DCT, sigma=20.

8x8 UDCT =10

8x8 UDCT =20

8x8 UDCT =10

Comparison with BLS-GSM

Comparison with BLS-GSM

Other Degradation Models

JPEG Artifact Removal

JPEG Artifact Removal

Image Sharpening

Image Sharpening

Conclusions New and simple scheme for over-complete transform based denoising. MFs are optimized in a discriminative manner. Linear formulation of non-linear minimization. Eliminating the need for modeling complex statistical prior in high-dim. space. Seamlessly applied to other degradation problems as long as scalar MFs are used for reconstruction.

Recent Results What is the best transform to be used (for a given image or for a given set)?

The End Thank You