Nonlinear Approximation Based Image Recovery Using Adaptive Sparse Reconstructions Onur G. Guleryuz Epson Palo Alto Laboratory Palo Alto, CA (Full screen mode recommended. Please see movies.zip file for some movies, or me. Audio of the presentation will be uploaded soon.)

Overview Algorithm. Five examples and movies to discuss transform properties. Conclusion. Problem definition. Notation and main idea. Difficulties with nonstationary statistics. Properties. Many more (~20) simulation examples, movies, etc. Please stay after questions. Working paper: (google: onur guleryuz)

Problem Statement Image Lost Block Use surrounding spatial information to recover lost block via adaptive sparse reconstructions. Generalizations: Irregularly shaped blocks, partial information,... Applications: Error concealment, damaged images,... Any image prediction scenario. Any signal prediction scenario.

Notation: Transforms signal transform basis transform coefficient (scalar) Assume orthonormal transforms:

Notation: Approximation Linear approximation: Nonlinear approximation: : the index set of significant coefficients, : the index set of insignificant coefficients, (K largest) (N-K smallest) Keep K<N coefficients. apriori ordering signal dependent ordering “Nonlinear Approximation Based Image …”

Notation: Sparse Linear app: Nonlinear approximation: sparse classes for linear approximation sparse classes for nonlinear approximation

Main Idea Given T>0 and the observed signal Fix transform basis Original Lost Block Image Predicted available pixels lost pixels (assume zero mean)

Sparse Classes Pixel coordinates for a “two pixel” image x x Transform coordinates Linear app: or Nonlinear app: convex set non-convex, star-shaped set Onur G. Guleryuz, E. Lutwak, D. Yang, and G. Zhang, ``Information-Theoretic Inequalities for Contoured Probability Distributions,'‘ IEEE Transactions on Information Theory, vol. 48, no. 8, pp , August T class(K,T) Rolf Schneider, ``Convex Bodies : The Brunn-Minkowski Theory,’’ Cambridge University Press, March 2003.

Examples dB dB dB dB 3. Image prediction has a tough audience! 2. MSE improving. 1. Interested in edges, textures, …, and combinations (not handled well in literature)

Small amount of data, make a mistake, learn mixed statistics. Difficulties with Nonstationary Data Estimation is a well studied topic, need to infer statistics, then build estimators. With nonstationary data inferring statistics is very difficult. Regions with different statistics. Perhaps do edge detection? Need accurate segmentation (very difficult) just to learn! Higher order method, better edge detection? Statistics are not even uniform and order must be very high. Statistics change rapidly without apriori structure.

Important Properties Applicable for general nonstationary signals. No non-robust edge detection, segmentation, training, learning, etc., required. Very robust technique. Use it on speech, audio, seismic data, … This technique does not know anything about images. Just pick a transform that provides sparse decompositions using nonlinear approximation, the rest is automated. (DCTs, wavelets, complex wavelets, etc.)

Main Algorithm : orthonormal linear transformation. : linear transform of y ( ). Start with an initial value. Get c Threshold coefficients to determine V(x,T) sparsity constraint Recover by minimizing Reduce threshold (found solution becomes initial value). (equations or iterations)

Progression of Solutions Search over non-convex, star-shaped set Nonlinear app: class(K,T) Pixel coordinates for a “two pixel” image x available pixel missing pixel available pixel constraint T decreases Search over … Class size increases

Estimation Theory Proposition 1: Solution of subject to sparsity constraint results in the linear estimate Proposition 2: Conversely suppose that we start with a linear estimate for via } Sparsity Constraint = Linear Estimation restricted to dimensional subspace sparsity constraints

Required Statistics? None. The statistics required in the estimation are implicitly determined by the utilized transform and V(x). (V(x) is the index set of insignificant coefficients) I will fix G and adaptively determine V(x). (By hard-thresholding transform coefficients)

Apriori v.s. Adaptive Method 1: Can at best be ensemble optimal for second order statistics. Method 2: Can at best be THE optimal! Do not capture nonstationary signals with edges. J.P. D'Ales and A. Cohen, “Non-linear Approximation of Random Functions”, Siam J. of A. Math 57-2, , 1997 Albert Cohen, Ingrid Daubechies, Onur G. Guleryuz, and Michael T. Orchard, “On the importance of combining wavelet-based nonlinear approximation with coding strategies,” IEEE Transactions on Information Theory, July optimality?

Conclusion Simple, robust technique. Very good and promising performance. Estimation of statistics not required (have to pick G though). Applicable to other domains. Q: Classes of signals over which optimal? A: Nonlinear approximation classes of the transform. Signal dependent basis to expand classes over which optimal. Help design better signal representations. (intuitive)

“Periodic” Example DCT 9x dB Lower thresholds, larger classes. PSNR

Properties of Desired Transforms Periodic, approximately periodic regions: Transform should “see” the period Example: Minimum period 8 at least 8x8 DCT (~ 3 level wavelet packets). Localized … … M-M s(n) … … |S(w)| zeroes Want lots of small coefficients wherever they may be …

Periodic Example (period=8) DCT 8x8 Perf. Rec. (Easy base signal, ~fast decaying envelope).

“Periodic” Example DCT 24x dB (Harder base signal.)

Edge Example DCT 8x dB (~ Separable, small DCT coefficients except for first row.)

Edge Example DCT 24x dB (similar to vertical edge, but tilted)

Properties of Desired Transforms Periodic, approximately periodic regions: Frequency selectivity Edge regions: Transform should have the frequency selectivity to “see” the slope of the edge. Localized

Overcomplete Transforms smooth edge DCT block over an edge (not very sparse) DCT block over a smooth region (sparse) DCT1 ++ DCT2=DCT1 shiftedDCT3 Onur G. Guleryuz, ``Weighted Overcomplete Denoising,‘’ Proc. Asilomar Conference on Signals and Systems, Pacific Grove, CA, Nov Only the insignificant coefficients contribute. Can be generalized to denoising:

Properties of Desired Transforms Frequency selectivity for “periodic” + “edge” regions. Localized ! Nonlinear Approximation does not work for non-localized Fourier transforms. J.P. D'Ales and A. Cohen, “Non-linear Approximation of Random Functions”, Siam J. of A. Math 57-2, , 1997 (Overcomplete DCTs have more mileage since for a given freq. selectivity, have the ~smallest spatial support.)

“Periodic” Example DCT 16x dB

“Periodic” Example DCT 16x dB

“Periodic” Example DCT 24x dB

Edge Example DCT 16x dB

“Edge” Example DCT 24x dB

Combination Example DCT 24x dB

Combination Example DCT 16x dB

Combination Example DCT 24x dB (not enough to “see” the period)

Unsuccessful Recovery Example DCT 16x dB

Partially Successful Recovery Example DCT 16x dB

Combination Example DCT 24x dB

“Periodic” Example DCT 32x dB

Edge Example DCT 16x dB

Edge Example DCT 24x dB

Robustness remains the same but changes.

Determination Start by layering the lost block. Estimate layer at a time. Recover layer P by using information from layers 0,…,P-1 (the lost block is potentially large)

th k DCT block u w DCT (LxL) tiling 1 Outer border of layer 1 Image Lost block o (k) w u Hard threshold block k coefficients if o (k) < L/2 w u OR Fix T. Look at DCTs that have limited spatial overlap with missing data. Establish sparsity constraints by thresholding these DCT coefficients with T. (If | c(i)|<T add to sparsity constraints.) Determination II