ICA Alphan Altinok

Outline  PCA  ICA  Foundation  Ambiguities  Algorithms  Examples  Papers

PCA & ICA  PCA  Projects d-dimensional data onto a lower dimensional subspace in a way that is optimal in Σ|x 0 – x| 2.  ICA  Seek directions in feature space such that resulting signals show independence.

PCA  Compute d-dimensional μ (mean).  Compute d x d covariance matrix.  Compute eigenvectors and eigenvalues.  Choose k largest eigenvalues.  k is the inherent dimensionality of the subspace governing the signal and (d – k) dimensions generally contain noise.  Form a d x k matrix A with k columns of eigenvalues.  The representation of data by principal components consists of projecting data into k-dimensional subspace by x = A t (x – μ).

PCA  A simple 3-layer neural network can form such a representation when trained.

ICA  While PCA seeks directions that represents data best in a Σ|x 0 – x| 2 sense, ICA seeks such directions that are most independent from each other.  Used primarily for separating unknown source signals from their observed linear mixtures.  Typically used in Blind Source Separation problems. ICA is also used in feature extraction.

ICA – Foundation  q source signals s 1 (k), s 2 (k), …, s q (k)  with 0 means  k is the discrete time index or pixels in images  scalar valued  mutually independent for each value of k  h measured mixture signals x 1 (k), x 2 (k), …, x h (k)  Statistical independence for source signals  p[s 1 (k), s 2 (k), …, s q (k)] = П p[s i (k)]

ICA – Foundation  The measured signals will be given by  x j (k) = Σs i (k)a ij + n j (k)  For j = 1, 2, …, h, the elements a ij are unknown.  Define vectors x(k) and s(k), and matrix A  Observed:x(k) = [x 1 (k), x 2 (k), …, x h (k)]  Source:s(k) = [s 1 (k), s 2 (k), …, s q (k)]  Mixing matrix:A = [a 1, a 2, …, a q ]  The equation above can be stated in vector-matrix form  x(k) = As(k) + n(k) = Σs i (k)a i + n(k)

Ambiguities with ICA  The ICA expansion  x(k) = As(k) + n(k) = Σs i (k)a i + n(k)  Amplitudes of separated signals cannot be determined.  There is a sign ambiguity associated with separated signals.  The order of separated signals cannot be determined.

ICA – Using NNs  Prewhitening – transform input vectors x(k) by  v(k) = V x(k)  Whitening matrix V can be obtained by NN or PCA  Separation (NN or contrast approximation)  Estimation of ICA basis vectors (NN or batch approach)

ICA – Fast Fixed Point Algorithm  FFPA converges rapidly to the most accurate solution allowed by the data structure.

ICA – Example

 BSS of recorded speech and music signals. speech / musicspeech / speech speech / speech in difficult environment mic1 mic2 separated1 separated2

ICA – Example  Source images  separation demo separation demo

ICA – Papers  Hinton – A New View of ICA  Interprets ICA as a probability density model.  Overcomplete, undercomplete, and multi-layer non-linear ICA becomes simpler.  Cardoso – Blind Signal Separation, Statistical Principles  Modelling identifiability.  Contrast functions.  Estimating functions.  Adaptive algorithms.  Performance issues.

ICA – Papers  Hyvarinen – ICA Applied to Feature Extraction from Color and Stereo Images  Seeks to extend ICA by contrasting it to the processing done in neural receptive fields.  Hyvarinen – Survey on ICA  Lawrence – Face Recognition, A Convolutional Neural Network Approach  Combines local image sampling, a SOM, and a convolutional NN that provides partial invariance to translation, rotation, scaling, and deformations.

ICA – Papers  Sejnowski – Independent Component Representations for Face Recognition  Sejnowski – A Comparison of Local vs Global Image Decompositions for Visual Speechreading  Bartlett – Viewpoint Invariant Face Recognition Using ICA and Attractor Networks  Bartlett – Image Representations for Facial Expression Coding

ICA – Links    