Transform-based Models Principal component analysis (PCA) or Karhunen-Loeve transform (KLT) Application into Face Recognition and MATLAB demo DFT, DCT and Wavelet transforms Statistical modeling of transform coefficients (sparse representations) Application into Texture Synthesis and MATLAB demos EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 PCA/KLT What is principal components? direction of maximum variance in the input space (physical interpretation) principal eigenvector of the covariance matrix (mathematical definition) Theoretic derivations (This is not a Theory Course like EE513) There exist several different approaches in the literature of statistics, economics and communication theory EE565 Advanced Image Processing Copyright Xin Li 2008

Standard Derivation (Covariance method) (unitary condition, refer to EE465) Basic idea: Diagonalization = EE565 Advanced Image Processing Copyright Xin Li 2008

Geometric Interpretation (direction of maximum variation/information) EE565 Advanced Image Processing Copyright Xin Li 2008

Why Does PCA/KLT Make Sense? In Pattern Recognition (e.g., R. Duda’s textbook “Pattern Classification”) or in Signal Processing (e.g., S. Mallat’s textbook “A Wavelet Tour of Signal Processing”) Analytical results are available for stationary Gaussian processes except the unknown parameters (low-order statistics) Classical ML/Bayes parameter estimation works most effectively under the independence assumption (recall the curse of dimensionality) Transform facilitates the satisfaction of this assumption In Economics, Google “Hotteling transform” EE565 Advanced Image Processing Copyright Xin Li 2008

Example: Transform Facilitates Modeling y2 y1 x1 x1 and x2 are highly correlated y1 and y2 are less correlated p(x1x2)  p(x1)p(x2) p(y1y2)  p(y1)p(y2) EE565 Advanced Image Processing Copyright Xin Li 2008

Comparison Between LR and LT Linear regression (AR model) Hyperplane fitting (a local strategy) Dimensionality reduction: data space mapped to parameter space Distortion not preserved (refer to EE467 closed-loop opt. in speech coding) Linear transform (PCA/KLT) Rotation of coordinate (a global strategy) Dimensionality reduction: only preserve the largest eigenvalues in the data space Preserves distortion (unitary property of P) EE565 Advanced Image Processing Copyright Xin Li 2008

Transform-based Models Principal component analysis (PCA) or Karhunen-Loeve transform (KLT) Application into Face Recognition and MATLAB demo DFT, DCT and Wavelet transforms Statistical modeling of transform coefficients (sparse representations) Application into Texture Synthesis and MATLAB demos EE565 Advanced Image Processing Copyright Xin Li 2008

Appearance-based Recognition (adapted from CMU Class 15385-s06) Directly represent appearance (image brightness), not geometry. Why? Avoids modeling geometry, complex interactions between geometry, lighting and reflectance. Why not? Too many possible appearances! m “visual degrees of freedom” (eg., pose, lighting, etc) R discrete samples for each DOF “nature is economical of structures but of principles” –Abdus Salam

The Space of Faces + = An image with N pixels is a point in N-dimensional space A collection of M images is a cloud of M point in RN We can define vectors in this space as we did in the 2D case [Apologies to former President Bush]

Key Idea: Linear Subspace Images in the possible set are highly correlated. So, compress them to a low-dimensional linear subspace that captures key appearance characteristics of the visual DOFs. Linearity assumption is a double-bladed sword: it facilitates analytical derivation and computational solution but Nature seldom works in a linear fashion EIGENFACES: [Turk and Pentland’1991] USE PCA!

Example of Eigenfaces Training set of face images. 15 principal components (eigenfaces or eigenvectors corresponding to 15 largest eigenvalues). Eigenfaces look somewhat like ghost faces.

Linear Subspaces explained by 2D Toy Example (Easier for Visualization) convert x into v1, v2 coordinates What does the v2 coordinate measure? What does the v1 coordinate measure? distance to line use it for classification—near 0 for orange pts position along line use it to specify which orange point it is Classification can be expensive Must either search (e.g., nearest neighbors) or store large probability density functions. Suppose the data points are arranged as above Idea—fit a line, classifier measures distance to line

Dimensionality Reduction We can represent the orange points with only their v1 coordinates since v2 coordinates are all essentially 0 This makes it much cheaper to store and compare points A much bigger deal for higher dimensional problems

Linear Subspaces (PCA in 2D) Consider the variation along direction v among all of the orange points: What unit vector v minimizes var? What unit vector v maximizes var? Solution: v1 is eigenvector of A with largest eigenvalue v2 is eigenvector of A with smallest eigenvalue

PCA in Higher Dimensions Suppose each data point is N-dimensional Same procedure applies: The eigenvectors of A define a new coordinate system eigenvector with largest eigenvalue captures the most variation among training vectors x eigenvector with smallest eigenvalue has least variation We can compress the data by only using the top few eigenvectors corresponds to choosing a “linear subspace” represent points on a line, plane, or “hyper-plane” these eigenvectors are known as the principal components

Problem: Size of Covariance Matrix A Suppose each data point is N-dimensional (N pixels) The size of covariance matrix A is N x N The number of eigenfaces is N Example: For N = 256 x 256 pixels, Size of A will be 65536 x 65536 ! Number of eigenvectors will be 65536 ! Typically, only 20-30 eigenvectors suffice. So, this method is very inefficient! 2 2

Efficient Computation of Eigenvectors Efficient Computation of Eigenvectors* (You can skip it if you don’t like matrix) If B is MxN and M<<N then A=BTB is NxN >> MxM M  number of images, N  number of pixels use BBT instead, eigenvector of BBT is easily converted to that of BTB (BBT) y = y => BT(BBT) y =  (BTy) => (BTB)(BTy) =  (BTy) => BTy is the eigenvector of BTB

Eigenfaces – summary in words Eigenfaces are the eigenvectors of the covariance matrix of the probability distribution of the vector space of human faces Eigenfaces are the ‘epitomized face ingredients’ derived from the statistical analysis of many pictures of human faces A human face may be considered to be a combination of these epitomized faces

Generating Eigenfaces – in words Large set of images of human faces is taken. The images are normalized to line up the eyes, mouths and other features. The eigenvectors of the covariance matrix of the face image vectors are then extracted. These eigenvectors are called eigenfaces. Well – please also keep in mind that if BTB is too large, you can use BBT instead (an algebraic trick)

Eigenfaces for Face Recognition When properly weighted, eigenfaces can be summed together to create an approximate gray-scale rendering of a human face. Remarkably few eigenvector terms are needed to give a fair likeness of most people's faces. Hence eigenfaces provide a means of applying “data compression” to faces for identification purposes (note NOT for transmission purpose).

Detection with Eigenfaces The set of faces is a “subspace” of the set of images Suppose it is K dimensional We can find the best subspace using PCA This is like fitting a “hyper-plane” to the set of faces spanned by vectors v1, v2, ..., vK Any face:

Eigenfaces PCA extracts the eigenvectors of A Gives a set of vectors v1, v2, v3, ... Each one of these vectors is a direction in face space what do these look like?

Projecting onto the Eigenfaces (it is easier to understand projection by using the 2D toy example though conceptually high-D works the same way) The eigenfaces v1, ..., vK span the space of faces A face is converted to eigenface coordinates by

Key Property of Eigenspace Representation Given 2 images that are used to construct the Eigenspace is the eigenspace reconstruction of image Then, That is, distance in Eigenspace is approximately equal to the correlation between two images.

Advantage of Dimensionality Reduction xRHxW →a RK Training set: x1 ,x2 ,…,xM → a1 ,a2 ,…,aM New image: x → a For detection: thresholding d=mean(||a-ak||) For recognition: select the index minimizing ||a-ak|| EE565 Advanced Image Processing Copyright Xin Li 2008

Detection: Is this a face or not?

Recognition: Whose Face is This? Algorithm Process the image database (set of images with labels) Run PCA—compute eigenfaces Calculate the K coefficients for each image Given a new image (to be recognized) x, calculate K coefficients Detect if x is a face If it is a face, who is it? Find closest labeled face in database nearest-neighbor in K-dimensional space

An Excellent Toy Example to Help Your Understanding (M’ is the number of eigenfaces used)

Choosing the Dimension K NM i = eigenvalues How many eigenfaces to use? Look at the decay of the eigenvalues the eigenvalue tells you the amount of variance “in the direction” of that eigenface ignore eigenfaces with low variance

New Ideas We can Play with A localized version of eigenfaces-based recognition For each subject k=1,…,K, obtain its associated N eigen-faces vk1 ,…, vkN For a new image x, project it onto all K sets of “localized” eigen-face spaces (so we can obtain K reconstructed copies x1,…, xK) Select the one minimizing ||x-xk|| Connection with sparse representation (or l1 regularization) - refer to Prof. Yi Ma’s homepage and his new PAMI paper ^ ^ ^

Transform-based Models Principal component analysis (PCA) or Karhunen-Loeve transform (KLT) Application into Face Recognition and MATLAB demo DFT, DCT and Wavelet transforms Statistical modeling of transform coefficients (sparse representations) Application into Texture Synthesis and MATLAB demos EE565 Advanced Image Processing Copyright Xin Li 2008

Alternative Tools (more suitable for Telecommunication Applications) Discrete Fourier Transform Can be shown to be KLT for circular stationary process (eigen-vectors take the form of discrete Fourier basis) Discrete Cosine Transform (DCT) Good approximation of KLT for AR(1) with high-correlation (e.g., a=0.9) – you are asked to show this in HW#1 Wavelet Transform Effective tool for characterizing transient signals EE565 Advanced Image Processing Copyright Xin Li 2008

One-Minute Tour of Wavelet x(n) s(n) d(n) EE565 Advanced Image Processing Copyright Xin Li 2008

Wavelet Transform on Images s(m,n) d(m,n) LL HL HH LH After row transform: each row is decomposed into low-band (approximation) and high-band (detail) Note that the order of row/column transform does not matter EE565 Advanced Image Processing Copyright Xin Li 2008

From one-level to multi-level EE565 Advanced Image Processing Copyright Xin Li 2008

Relationship to Linear Regression (Sparsity Perspective) In AR model, the effectiveness is measured by the energy (sparsity) of prediction errors In transform-based models, the effectiveness is measured by the energy (sparsity) of transform coefficients Improved sparsity (or lower energy) implies a better match between the assumed model and observation data EE565 Advanced Image Processing Copyright Xin Li 2008

Empirical Observation Extract HL band After WT A single peak at zero EE565 Advanced Image Processing Copyright Xin Li 2008

Univariate Probability Model Laplacian: Gaussian: EE565 Advanced Image Processing Copyright Xin Li 2008

Gaussian Distribution EE565 Advanced Image Processing Copyright Xin Li 2008

Laplacian Distribution EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Statistical Testing How do we know which parametric model better fits the empirical distribution of wavelet coefficients? In addition to visual inspection (which is often subjective and less accurate), we can use various statistical testing tools to objectively evaluate the closeness of an empirical cumulative distribution function (ECDF) to the hypothesized one One of the most widely used techniques is Kolmogorov-Smirnov Test (MATLAB function: >help kstest). EE565 Advanced Image Processing Copyright Xin Li 2008

Kolmogorov-Smirnov Test* The K-S test is based on the maximum distance between empirical CDF (ECDF) and hypothesized CDF (e.g., the normal distribution N(0,1)). EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Example Usage: [H,P,KS,CV] = KSTEST(X,CDF) If CDF is omitted, it assumes pdf of N(0,1) Accept the hypothesis Reject the hypothesis d: high-band wavelet coefficients of lena image (note the normalization by signal variance) x: computer-generated samples (0<P<1, the larger P, the more likely) EE565 Advanced Image Processing Copyright Xin Li 2008

Generalized Gaussian/Laplacian Distribution P: shape parameter : variance parameter where Laplacian Gaussian EE565 Advanced Image Processing Copyright Xin Li 2008

Model Parameter Estimation* Maximum Likelihood Estimation Method of moments Linear regression method Ref. [1] Sharifi, K. and Leon-Garcia, A. “Estimation of shape parameter for generalized Gaussian distributions in subband decompositions of video,” IEEE T-CSVT, No. 1, February 1995, pp. 52-56. [2] www.cimat.mx/reportes/enlinea/I-01-18_eng.pdf EE565 Advanced Image Processing Copyright Xin Li 2008

Transform-based Models Principal component analysis (PCA) or Karhunen-Loeve transform (KLT) Application into Face Recognition and MATLAB demo DFT, DCT and Wavelet transforms Statistical modeling of transform coefficients (sparse representations) Application into Texture Synthesis and MATLAB demos EE565 Advanced Image Processing Copyright Xin Li 2008

Wavelet-based Texture Synthesis Basic idea: two visually similar textures will also have similar statistics Pyramid-based (using steerable pyramids) Facilitate the statistical modeling Histogram matching Enforce the first-order statistical constraint Texture matching Alternate histogram matching in spatial and wavelet domain Boundary handling: use periodic extension Color consistency: use color transformation EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Histogram Matching Generalization of histogram equalization (the target is the histogram of a given image instead of uniform distribution) EE565 Advanced Image Processing Copyright Xin Li 2008

Histogram Equalization Uniform Quantization Note: y cumulative probability function L 1 x L EE565 Advanced Image Processing Copyright Xin Li 2008

MATLAB Implementation function y=hist_eq(x) [M,N]=size(x); for i=1:256 h(i)=sum(sum(x= =i-1)); End y=x;s=sum(h); I=find(x= =i-1); y(I)=sum(h(1:i))/s*255; end Calculate the histogram of the input image Perform histogram equalization EE565 Advanced Image Processing Copyright Xin Li 2008

Histogram Equalization Example                                 Histogram Equalization Example EE565 Advanced Image Processing Copyright Xin Li 2008

Histogram Specification ? EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Texture Matching Objective: the histogram of both subbands and synthesized image matches the given template Basic hypothesis: if two texture images visually look similar, then they have similar histograms in both spatial and wavelet domain EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Image Examples EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 I.I.D. Assumption Challenged If wavelet coefficients of each subband are indeed i.i.d., then random permutation of pixels should produce another image of the same class (natural images) The fundamental question here: does WT completely decorrelate image signals? EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Image Example High-band coefficients permutation You can run the MATLAB demo to check this experiment EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Another Experiment Y X Joint pdf of two uncorrelated random variables X and Y EE565 Advanced Image Processing Copyright Xin Li 2008

Joint PDF of Wavelet Coefficients Y= X= Joint pdf of two correlated random variables X and Y Neighborhood I(Q): {Left,Up,cousin and aunt} EE565 Advanced Image Processing Copyright Xin Li 2008

Texture Synthesis via Parametric Models in the Wavelet Space Basic idea: two visually similar textures will also have similar statistics Instead of matching histogram (nonparametric models), we can build parametric models for wavelet coefficients and enforce the synthesized image to inherit the parameters of given image Model parameters: 710 parameters were used in Portilla and Simoncelli’s experiment (4 orientations, 4 scales, 77 neighborhood) EE565 Advanced Image Processing Copyright Xin Li 2008

Statistical Constraints Four types of constraints Marginal Statistics Raw coefficient correlation Coefficient magnitude statistics Cross-scale phase statistics Alternating Projections onto the four constraint sets Projection-onto-convex-set (POCS) EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Convex Set A set Ω is said to be convex if for any two point We have Convex set examples Non-convex set examples EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Projection Operator f Projection onto convex set C g C In simple words, the projection of f onto a convex set C is the element in C that is closest to f in terms of Euclidean distance EE565 Advanced Image Processing Copyright Xin Li 2008

Alternating Projection X1 X∞ X0 X2 C2 Projection-Onto-Convex-Set (POCS) Theorem: If C1,…,Ck are convex sets, then alternating projection P1,…,Pk will converge to the intersection of C1,…,Ck if it is not empty Alternating projection does not always converge in the case of non-convex set. Can you think of any counter-example? EE565 Advanced Image Processing Copyright Xin Li 2008

Convex Constraint Sets ● Non-negative set ● Bounded-value set or ● Bounded-variance set A given signal EE565 Advanced Image Processing Copyright Xin Li 2008

Non-convex Constraint Set Histogram matching used in Heeger&Bergen’1995 Bounded Skewness and Kurtosis skewness kurtosis The derivation of projection operators onto constraint sets are tedious are referred to the paper and MATLAB codes by Portilla&Simoncelli. EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Image Examples original synthesized EE565 Advanced Image Processing Copyright Xin Li 2008

Image Examples (Con’d) original synthesized EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 When Does It Fail? original synthesized EE565 Advanced Image Processing Copyright Xin Li 2008

EE565 Advanced Image Processing Copyright Xin Li 2008 Summary Textures represent an important class of structures in natural images – unlike edges characterizing object boundaries, textures often associate with the homogeneous property of object surfaces Wavelet-domain parametric models provide a parsimonious representation of high-order statistical dependency within textural images EE565 Advanced Image Processing Copyright Xin Li 2008