An Introduction of Independent Component Analysis (ICA) Xiaoling Wang Jan. 28, 2003

2 What Is ICA? Application: blind source separation (BSS) and deconvolution Motivation: “cocktail party problem” Assumption: two people speaking simultaneously, two microphones in different locations

3 Principles of ICA Algorithm Assumption: sources are statistically independent Goal: it seeks a transformation to coordinates in which the data are maximally statistically independent Definition: Mixing process Demixing process – mixing matrix, – separation matrix

4 Hierarchy of ICA Models Nonlinear mixing Linear mixing Classical ICA Flexible Source model Infomax Non-stationary sources Non-stationary mixing No noise Factor Analysis R diagonal Gaussian sources Independent Factor analysis Non-Gaussian sources Cumulant based methods Approximations to mutual information Switching source model Probabilistic PCA FastICA Kurtosis minimization Fixed source model PCA orthogonal mixing No noise

5 Independence of Sources Independence: the pdf of sources can be factorized Nongaussian is independent Seek the separation matrix W which maximize the nongaussianity of the estimated sources

6 Measures of Nongaussianity Kurtosis (4th order cumulant): Subgaussian: negative kurtosis Supergaussian: positive kurtosis Negentropy: differential entropy negentropy

7 Measures of Nongaussianity (Cont.) Mutual information: For,

8 FastICA Algorithm Basic form: Choose an initial (e.g. Random) weight vector Let If not converged, go back to step 2 For several units: decorrelation Let

9 Nonlinear ICA Model: Existence and uniqueness of solutions There always exists an infinity of solutions if the space of the nonlinear mixing functions is not limited Post-nonlinear problem mixing demixing

10 Algorithms for Nonlinear ICA Burel’s approach: neural solution, known nonlinearities on unknown parameters Krob & Benidir: high order moments, polynomial mixtures Pajunen et al.: SOMs, locally factorable pdf Pajunen et al.: GTM(generative topographic mapping), output distribution matches the known source distributions Post nonlinear mixtures: Taleb & Jutten: adaptive componentwise separation Yang et al.: two-layer neural network Puntonet et al.: nonlinearities are a power function, geometrical considerations