Wavelet-based Texture Synthesis Basic idea: two visually similar textures will also have similar statistics Pyramid-based (using steerable pyramids) Facilitate the statistical modeling Histogram matching Enforce the first-order statistical constraint Texture matching Alternate histogram matching in spatial and wavelet domain Boundary handling: use periodic extension Color consistency: use color transformation EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Histogram Matching Generalization of histogram equalization (the target is the histogram of a given image instead of uniform distribution) EE565 Advanced Image Processing Copyright Xin Li 2008

Histogram Equalization Uniform Quantization Note: y cumulative probability function L 1 x L EE565 Advanced Image Processing Copyright Xin Li 2008

MATLAB Implementation function y=hist_eq(x) [M,N]=size(x); for i=1:256 h(i)=sum(sum(x= =i-1)); End y=x;s=sum(h); I=find(x= =i-1); y(I)=sum(h(1:i))/s*255; end Calculate the histogram of the input image Perform histogram equalization EE565 Advanced Image Processing Copyright Xin Li 2008

Histogram Equalization Example                                 Histogram Equalization Example EE565 Advanced Image Processing Copyright Xin Li 2008

Histogram Specification ? EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Texture Matching Objective: the histogram of both subbands and synthesized image matches the given template Basic hypothesis: if two texture images visually look similar, then they have similar histograms in both spatial and wavelet domain EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Image Examples EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 I.I.D. Assumption Challenged If wavelet coefficients of each subband are indeed i.i.d., then random permutation of pixels should produce another image of the same class (natural images) The fundamental question here: does WT completely decorrelate image signals? EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Image Example High-band coefficients permutation You can run the MATLAB demo to check this experiment EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Another Experiment Y X Joint pdf of two uncorrelated random variables X and Y EE565 Advanced Image Processing Copyright Xin Li 2008

Joint PDF of Wavelet Coefficients Y= X= Joint pdf of two correlated random variables X and Y Neighborhood I(Q): {Left,Up,cousin and aunt} EE565 Advanced Image Processing Copyright Xin Li 2008

Texture Synthesis via Parametric Models in the Wavelet Space Basic idea: two visually similar textures will also have similar statistics Instead of matching histogram (nonparametric models), we can build parametric models for wavelet coefficients and enforce the synthesized image to inherit the parameters of given image Model parameters: 710 parameters were used in Portilla and Simoncelli’s experiment (4 orientations, 4 scales, 77 neighborhood) EE565 Advanced Image Processing Copyright Xin Li 2008

Statistical Constraints Four types of constraints Marginal Statistics Raw coefficient correlation Coefficient magnitude statistics Cross-scale phase statistics Alternating Projections onto the four constraint sets Projection-onto-convex-set (POCS) EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Convex Set A set Ω is said to be convex if for any two point We have Convex set examples Non-convex set examples EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Projection Operator f Projection onto convex set C g C In simple words, the projection of f onto a convex set C is the element in C that is closest to f in terms of Euclidean distance EE565 Advanced Image Processing Copyright Xin Li 2008

Alternating Projection X1 X∞ X0 X2 C2 Projection-Onto-Convex-Set (POCS) Theorem: If C1,…,Ck are convex sets, then alternating projection P1,…,Pk will converge to the intersection of C1,…,Ck if it is not empty Alternating projection does not always converge in the case of non-convex set. Can you think of any counter-example? EE565 Advanced Image Processing Copyright Xin Li 2008

Convex Constraint Sets ● Non-negative set ● Bounded-value set or ● Bounded-variance set A given signal EE565 Advanced Image Processing Copyright Xin Li 2008

Non-convex Constraint Set Histogram matching used in Heeger&Bergen’1995 Bounded Skewness and Kurtosis skewness kurtosis The derivation of projection operators onto constraint sets are tedious are referred to the paper and MATLAB codes by Portilla&Simoncelli. EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Image Examples original synthesized EE565 Advanced Image Processing Copyright Xin Li 2008

Image Examples (Con’d) original synthesized EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 When Does It Fail? original synthesized EE565 Advanced Image Processing Copyright Xin Li 2008

Summary on Wavelet-based Texture Synthesis Textures represent an important class of structures in natural images – unlike edges characterizing object boundaries, textures often associate with the homogeneous property of object surfaces Wavelet-domain parametric models provide a parsimonious representation of high-order statistical dependency within textural images EE565 Advanced Image Processing Copyright Xin Li 2008

EE591b Advanced Image Processing Copyright Xin Li 2006 Image Denoising Theory of linear estimation Spatial domain denoising techniques Conventional Wiener filtering Spatially adaptive Wiener filtering Wavelet domain denoising Wavelet thresholding: hard vs. soft Wavelet-domain adaptive Wiener filtering Latest advances (we will cover it after patch-based modeling) Patch-based image denoising Learning-based image denoising EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Denoising Problem Noisy measurements signal N(0,σw2) MMSE estimator Difficulty: we need to know conditional pdf Wiener’s idea To simplify estimation by computing the best estimator that is a linear scaling of Y EE591b Advanced Image Processing Copyright Xin Li 2006

Orthogonality Principle ^ A linear estimator X of a random variable X ^ Minimizes E{(X-X)2} if and only if Geometric Interpretation X ^ X-X Y ^ X EE591b Advanced Image Processing Copyright Xin Li 2006

Linear MMSE Estimation For Gaussian signal The optimal LMMSE estimation is given by And it achieves Note: it can be shown such linear estimator is indeed E[X|Y] for Gaussian signal EE591b Advanced Image Processing Copyright Xin Li 2006

What if Signal Variance is Unknown? Maximum-likelihood estimation of is given by Since variance is nonnegative, we modify it When multiple observations yi’s are available, we have EE591b Advanced Image Processing Copyright Xin Li 2006

From Scalar to Vector Case Suppose X is a Gaussian process whose covariance matrix is a diagonalized matrix RX=diag{ηm}(m=0,…,N-1), the linear MMSE estimator is given by (A) and the minimal MSE is given by EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Decorrelating Q: What if X={X[0],…,X[N-1]} is correlated (i.e., Rx is not diagonalized)? A: We need to transform X into a set of uncorrelated basis and then apply the above result. The celebrated Karhunen-Loeve Transform does this job! Diagonal matrix Karhunen-Loeve Transform EE591b Advanced Image Processing Copyright Xin Li 2006

Transform-Domain Denoising Forward Transform Denoising operation Inverse Transform denoised signal Noisy signal e.g., KLT DCT WT e.g., Linear Wiener filtering Nonlinear Thresholding The performance of such transform-domain denoising is determined by how well the assumed probability model in the transform domain matches the true statistics of source signal (optimality can only be established for the Gaussian source so far). EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Image Denoising Theory of linear estimation Spatial domain denoising techniques Conventional Wiener filtering Spatially adaptive Wiener filtering Wavelet domain denoising Wavelet thresholding: hard vs. soft Wavelet-domain adaptive Wiener filtering Latest advances Patch-based image denoising Learning-based image denoising EE591b Advanced Image Processing Copyright Xin Li 2006

Conventional Wiener Filtering Basic assumption: image source is modeled by a stationary Gaussian process Signal variance is estimated from the noisy observation data Can be interpreted as a linear frequency weighting EE591b Advanced Image Processing Copyright Xin Li 2006

Linear Frequency Weighting FT Power spectrum |X|2 EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Image Example Noisy, =50 (MSE=2500) denoised (MSE=1130) EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Image Example (Con’d) Noisy, =10 (MSE=100) denoised (MSE=437) EE591b Advanced Image Processing Copyright Xin Li 2006

Conclusions from the Experiments Why did it Fail? Nonstationary NonGaussian Poor modeling How to improve? Achieve spatial adaptation Use linear transform Putting them together EE591b Advanced Image Processing Copyright Xin Li 2006

Spatially Adaptive Wiener Filtering Basic assumption: image source is modeled by a nonstationary Gaussian process Signal variance is locally estimated from the windowed noisy observation data Recall T N=T2 T EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Image Example Noisy, =10 (MSE=100) denoised (T=3,MSE=56) EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Image Example (Con’d) Noisy, =50 (MSE=2500) denoised (MSE=354) EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Image Denoising Theory of linear estimation Spatial domain denoising techniques Conventional Wiener filtering Spatially adaptive Wiener filtering Wavelet domain denoising Wavelet thresholding: hard vs. soft Wavelet-domain adaptive Wiener filtering Latest advances Patch-based image denoising Learning-based image denoising EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 A Toy Example N-1 1 f(n) An integer uniformly distributed on [0,N-1] σ2=4/N Wiener Filter Estimator: Nonlinear thresholding Estimator: Conclusion: for sparse signals, linear filtering is not as effective as Nonlinear thresholding EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Why Wavelet Denoising? We need to distinguish spatially-localized events (edges) from noise components More about noise components Wavelet is such a basis because exceptional event generates identifiable exceptional coefficients due to its good localization property in both spatial and frequency domain As long as it does not generate exceptions Additive White Gaussian Noise is just one of them EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Wavelet Thresholding ~ Y X Noisy signal denoised signal DWT Thresholding IWT Hard thresholding Soft thresholding EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Choice of Threshold Donoho and Johnstone’1994 Gives denoising performance close to the “ideal weighting” Reference: S. Mallat, “A Wavelet Tour of Signal Processing”, Section 10.2 (pp. 435-453) EE591b Advanced Image Processing Copyright Xin Li 2006

Soft vs. Hard thresholding ● Shrinking the amplitude by T guarantees with a high probability that. ● Soft-thresholding has the same upper bound as hard-thresholding asymptotically and larger error than hard-thresholding at the same threshold value, but perceptually it works better. ● It can be also viewed as a computationally efficient approximation of ideal weighting ideal soft EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Denoising Example noisy image (σ2=100) Wiener-filtering (ISNR=2.48dB) Wavelet-thresholding (ISNR=2.98dB) ~ X: original, Y: noisy, X: denoised Improved SNR EE591b Advanced Image Processing Copyright Xin Li 2006

Duality with Image Coding* DWT Thresholding IWT Image denoising system DWT Q Channel Q-1 IWT Image coding system EE591b Advanced Image Processing Copyright Xin Li 2006

Basis Selection Problem s(n) 2 H0 2 G0 x(n) d(n) 2 H1 2 G1 Tce Tce-1 complete expansion (with decimation) H0 y0(n) G0 x(n) H1 y1(n) G1 Toe Toe-1 overcomplete expansion (without decimation) EE591b Advanced Image Processing Copyright Xin Li 2006

What do We Buy from Redundancy? x(n) 1 … … N-1 H1 T -T EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Translation Invariance (TI) Denoising x(n) Toe Thresholding Toe-1 Implementation based on overcomplete expansion x(n) Tce Thresholding Tce-1 + z z-1 Tce Thresholding Tce-1 Implementation based on complete expansion EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 2D Extension WD = Tce Thresholding Tce-1 shift(m1,n1) WD shift(-m1,-n1) Avg   denoised image Noisy image shift(mK,nK) WD shift(-mK,-nK) (mk,nk): a pair of integers, k=1-K (K: redundancy ratio) EE591b Advanced Image Processing Copyright Xin Li 2006

Example Wavelet-thresholding (ISNR=2.98dB) Translation-Invariant thresholding (ISNR=3.51dB) EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Go Beyond Thresholding Challenges with wavelet thresholding Determination of a global optimal threshold Spatially adjusting threshold based on local statistics How to go beyond thresholding? We need an accurate modeling of wavelet coefficients – nonlinear thresholding is a computationally efficient yet suboptimal solution EE591b Advanced Image Processing Copyright Xin Li 2006

Spatially Adaptive Wiener Filtering in Wavelet Domain Wavelet high-band coefficients are modeled by a Gaussian random variable with zero mean and spatially varying variance Apply Wiener filtering to wavelet coefficients, i.e., estimated in the same way as spatial-domain (Slide 15) EE591b Advanced Image Processing Copyright Xin Li 2006

Practical Implementation Conceptually very similar to its counterpart in the spatial domain Recall (ML estimation of signal variance) T N=T2 T In demo3.zip, you can find a C-coded example (de_noise.c) EE591b Advanced Image Processing Copyright Xin Li 2006

Example Translation-Invariant thresholding (ISNR=3.51dB) Spatially-adaptive wiener filtering (ISNR=4.53dB) EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Further Improvements Gaussian scalar mixture (GSM) based denoising (Portilla et al.’ 2003) Instead of estimating the variance, it explicitly addresses the issue of uncertainty with variance estimation Hidden Markov Model (HMM) based denoising (Romberg et al.’ 2001) Build a HMM for wavelet high-band coefficients (refer to the posted paper) EE591b Advanced Image Processing Copyright Xin Li 2006

Gaussian Scalar Mixture (GSM) Model definition: u~N(0,1) Gaussian pdf scale (variance) parameter Noisy observation model EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Basic Idea In spatially adaptive Wiener filtering, we estimate the variance from the data of a local window. The uncertainty with such variance estimation is ignored. In GSM model, such uncertainty is addressed through the scalar z (it determines the variance of GSM). Instead of using a single z (estimated variance), we build a probability model over z, i.e., E{x|y}=Ez{E{x|y,z}} EE591b Advanced Image Processing Copyright Xin Li 2006

Posterior Distribution Bayesian formula where Due to (proof left as exercise) is so-called Jeffery’s prior Question: What is E{xc|y,z}? EE591b Advanced Image Processing Copyright Xin Li 2006

GSM Denoising Algorithm MATLAB codes available at: http://decsai.ugr.es/~javier/denoise/index.html EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2006 Image Examples Noisy, =50 (MSE=2500) denoised (MSE=201) EE591b Advanced Image Processing Copyright Xin Li 2006

Image Examples (Con’d) Noisy, =10 (MSE=100) denoised (MSE=31.7) EE591b Advanced Image Processing Copyright Xin Li 2006

EE591b Advanced Image Processing Copyright Xin Li 2003 Early Attempts Each band is modeled by a Guassian random variable with zero mean and unknown variance (e.g., WSQ) Only modest gain over JPEG (DCT-based) is achieved Question: is this an accurate model? and how can we test it? EE591b Advanced Image Processing Copyright Xin Li 2003

Proof by Contradiction (I) Assumption: our modeling target Ω is the collection of natural images Suppose each coefficient X in a high band does observe Gaussian distribution, i.e., X~N(0,σ2), then flip the sign of X (i.e., replace X with –X) should not matter and generates another element in Ω (i.e., a different but meaningful image) Let’s test it! EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 Proof by Contradiction (II) DWT sign flip IWT EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 What is wrong with that? Think of two coefficients: one in smooth region and the other around edge, do they observe the same probabilistic distribution? Think of all coefficients around the same edge, do they observe the same probabilistic distribution? Ignorance of topology and geometry EE591b Advanced Image Processing Copyright Xin Li 2003

The Importance of Modeling Singularity Location Uncertainty Singularities carry critical visual information: edges, lines, corners … The location of singularities is important Recall locality of wavelets in spatial-frequency domain Singularities in spatial domain → significant coefficients in wavelet domain EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 Where-and-What Coding Communication context Where The location of significant coefficients What The sign and magnitude of significant coefficients ? communication channel picture Alice Bob EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 1993-2003 Embedded Zerotree Wavelet (EZW)’1993 Set Partition In Hierarchical Tree (SPIHT)’1995 Space-Frequency Quantization (SFQ)’ 1996 Estimation Quantization (EQ)’1997 Embedded Block Coding with Optimal Truncation (EBCOT)’2000 Least-Square Estimation Quantization (LSEQ)’2003 EE591b Advanced Image Processing Copyright Xin Li 2003

Zerotree Data Structure EE591b Advanced Image Processing Copyright Xin Li 2003

Zerotree Data Structure Parent-and-Children Ancestor-and-Descendent EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 Zerotree Terminology Zerotree root (ZRT): it and its all descendants are insignificant Isolated zero (IZ): it is insignificant but its descendant is not Positive significant (POS): it is significant and have a positive sign Negative significant (NEG): it is significant and have a negative sign EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 Coding Process EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 Toy Example EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 Toy Example Note: T=32 LH1 contains POS LH1 contains POS EE591b Advanced Image Processing Copyright Xin Li 2003

Where-and-What Interpretation Zerotree data structure effectively resolves the location uncertainty (where) of insignificant coefficients The dominant and subordinate passes defined in EZW can be viewed as “where” and “what” coding respectively Dyadic choice of T values (i.e., T=128,64, 32,16,…) renders embedded coding EE591b Advanced Image Processing Copyright Xin Li 2003

A Simpler Two-Stage Coding Position coding stage (where) Generate a binary map indicating the location of significant coefficients (|X|>T) Use context-based adaptive binary arithmetic coding (e.g., JBIG) to code the binary map Intensity coding stage (what) Code the sign and magnitude of significant coefficients EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 A Different Interpretation Two-class modeling of high-band coefficients Significant class: |X|>T Insignificant class: |X|<T Why does classification help? Nonstationarity of image source A probabilistic modeling perspective EE591b Advanced Image Processing Copyright Xin Li 2003

Classification-based Modeling Insignificant class Significant class Mixture EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 Classification Gain Without classification With classification Classification gain EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 Example EE591b Advanced Image Processing Copyright Xin Li 2003

EE591b Advanced Image Processing Copyright Xin Li 2003 Summary of Wavelet Coding SPIHT: a simpler yet more efficient implementation of EZW coder SFQ: Rate-Distortion optimized zerotree coder EQ: Rate-Distortion optimized backward classification strategy EBCOT (adopted by JPEG2000): a versatile embedded coder EE591b Advanced Image Processing Copyright Xin Li 2003