2. Introduction to Independent Component Analysis Math 285 project Fall 2015 Jingmei Lu Xixi Lu 12/10/2015

3. Agenda The “Cocktail Party” Problem ICA model Principle of ICA Fast ICA algorithm Separate mixed audio signal Reference

4. Sources Observations s1s1 s2s2 x1x1 x2x2 Purpose: estimate the two original speech signals s 1 (t) and s 2 (t), using only the recorded signals x 1 (t) and x 2 (t) The “Cocktail Party” Problem

5. Motivation Independent SourcesMixture signal

6. Motivation Independent Sources Recovered signals

7. What is ICA? “ Independent component analysis (ICA) is a method for finding underlying factors or components from multivariate (multi-dimensional) statistical data. What distinguishes ICA from other methods is that it looks for components that are both statistically independent, and nonGaussian.” A.Hyvarinen, A.Karhunen, E.Oja ‘Independent Component Analysis ’

8. ICA Model Observe n linear mixtures x 1,…x n of n independent components x j = a j1 s 1 + a j2 s 2 +.. + a jn s n, for all j x j: observed random variable s j : independent source variable ICA model: x = As a ij is the entry of A Task: estimate A and s using only the observeable random vector x

9. ICA Model Two assumptions: 1. The components s i are statistically independent 2. The independent components must have nongaussian distributions.

10. Why non-Gaussian Assume : 1) s 1 and s 2 are gaussian 2) mixing matrix A is orthogonal Then x 1 and x 2 are gaussian, uncorrelated, and of unit variance. Their joint density is

11. Why non-Gaussian Since the density is completely symmetric, it does not contain any information on the direction of the columns of the mixing matrix A.

12. Why non-Gaussian Assume s1 and s2 follow uniform distribution with zero mean and unit variance Mixing matrix A is x=As The edges of the parallelogram are in the direction of the columns of A

13. Principle of ICA y is a linear combination of s i, with weights given by z i Central Limit Theorem: the distribution of a sum of independent random variables tends toward a guassian distribution, under certain condition. z T s is more gaussian than either of s i. And becomes least gaussian when its equal to one of s i. So we could take w as a vector which maximizes the non-gaussianity of w T x.

14. Measure of Nongaussianity Entropy (H): degree of information that an observation gives A Gaussian variable has the largest entropy among all random variables of equal variance Negentropy J Computationally difficult

15. Negentropy approximations In fastICA algorithm, use G is some nonquadratic function. v is a Gaussian variable of zero mean and unit variance. Maximize J(y) to maximize nongaussianity.

16. Fast ICA Data Preprocessing Centering Whitening Fast ICA algorithm Maximize non gaussianity

17. Data Preprocessing

18. Fast ICA Algorithm 1. Choose an initial weight vector w. 2. Let w + = E{xg(w T x)} – E{g ′ (w T x)}w g() is the derivatives of functions G 3. w = w + /||w + ||. (Normalization step) 4. If not converged go back to 2 converged if norm(w new – w old ) < ξ ξ typically around 0.0001

19. Separate mixed audio signal

20. Mixed signals

21. Separated signals

22. Separated signals by PCA

23. Other applications Separation of Artifacts in MEG Data Finding Hidden Factors in Financial Data Reducing Noise in Natural Images Telecommunications

24. Reference Hyvärinen, A., Karhunen, J., Oja, E.: 2001, Independent Component Analysis: Algorithms and Applications, Wiley, New York. Särelä. "COCKTAIL PARTY PROBLEM." COCKTAIL PARTY PROBLEM. N.p., 20 Apr. 2005. Web. Dec.-Jan. 2015. http://research.ics.aalto.fi/ica/cocktail/cocktail_en.cgi