Machine Learning Independent Component Analysis Supervised Learning Classification and Regression K-Nearest Neighbor Classification Fisher’s Criteria & Linear Discriminant Analysis Perceptron: Linearly Separable Multilayer Perceptron & EBP & Deep Learning, RBF Network Support Vector Machine Ensemble Learning: Voting, Boosting(Adaboost) Unsupervised Learning Dimensionality Reduction: Principle Component Analysis Independent Component Analysis Clustering: K-means Semi-supervised Learning & Reinforcement Learning

Motivation Cocktail Party Problem 제4세부과제

= X 제4세부과제

Problem Statement u W x A s Unknown mutually independent source signals with zero means The model for sensor signals Problem To recover the original signals except for a permutation of indices and scales Estimation of an unmixing matrix u W x A s

Blind Source Separation by ICA ? mixing environment initial system Unknown sources and the mixing environment

Assumptions and limitations on ICA Basic assumptions: Individual source signals are statistically independent of the other source signals. # of sources ≤ # of observations x = As s=Wx Limitations: Scale and sign indeterminacy We cannot determine the variance of sources x = As = (A/α) (αs)=A’s’ Permutation indeterminacy We cannot determine the order of sources Sources should be non-Gaussian. At most one source can be Gaussian.

InfoMax Algorithm What is Entropy? Definition: The average amount of information of the source. Ex) Two Symbol Sources p 1 1/2 제4세부과제

Entropy Differential Entropy Joint Entropy Mutual Information Bounded Case: Uniform Distribution Unbounded Case: Normal Distribution (Gaussian) Joint Entropy Mutual Information 제4세부과제

Objective Function of ICA Minimizing Mutual Information Maximizing Entropy H(y) u W x A s 제4세부과제

Probability Density Function

Maximizing Output Entropy: InfoMax Joint entropy at the outputs Pdf of the outputs where

One Input One Output Stochastic gradient ascent learning rule

N→N Network Stochastic gradient ascent learning rule Jacobian of the transform

Learning rule where (Score function)

Natural Gradient Natural Gradient (or Relative Gradient) (Amari, Neural Comuptation, 1998. [3]) How can we decide the score ft.:Ext. ICA (T.-W. Lee et al., 1999.[4])

Flex. ICA: Generalized Gaussian (S. Choi. 2000.[6])

III. Comparison with PCA and Unifying Framework Calculate covariance matrix Using SVD, find M eigenvalues and eigenvectors. Choose L eigen vectors corresponding to L largest eigen values. L Largest eigen values M-L small eigen values

Comparison of PCA and ICA Independent = uncorrelated ? Only for Gaussian data dependence = correlation PCA basis ICA basis Generally, dependence ≠ correlation

Whitened signal and independent signal Observations Whitened Independent

Unifying Information-Theoretic Framework(1) Max. H(y): InfoMax Minimizing Mutual Information Negentropy Maximization

Unifying Information-Theoretic Framework(2) Maximum Likelihood Estimation

Unifying Information-Theoretic Framework(3) High-Order Moments and Cumulants Nonlinear PCA Bussgang Algorithms (Te-Won Lee et al., Computers and Mathematics with Applications, 2000. [1])

Biomedical data analysis Speech and audio processing Telecommunication Applications of ICA Image processing e.g. denoising, feature extraction, etc Biomedical data analysis e.g. anaysis of EEG, MEG, ECG, etc… Speech and audio processing e.g. speech separation, feature extraction Telecommunication e.g. MIMO System, etc. Bioinformatics e.g. micro-array data analysis, etc Financial data analysis

Image Processing Independent Components of Natural Scenes (Bell and Sejnowski, Vision Research 1996.[2])

Independent Components are Edge Filters

Biomedical Signal Processing Blind Source Separation of EEG Signals

Convolved Mixtures: ANC vs. BSS Signal Source x(n) u1(n) Mixing System (?) ANC BSS W(z) Noise Source r1(n) x1(n) u1(n) Source 1 Mixing System (?) W21(z) W12(z) x2(n) u2(n) Source 2

Adaptive Noise Canceling System output Minimize the output power The LMS algorithm

ICA-Based Approach to ANC [8] Maximizing entropy Set dummy output Learning rules of adaptive filter coefficients in ANC (Park et al. 2002) y y 1 2 where u v w(k) x r 1 where s n

Input Signal with Noise Known Noise Source 차례대로 자동으로 play 됩니다. (단 스피커 모양은 편집모드에서는 더블클릭/슬라이드쇼 환경에서는 원클릭에 재생됩니다) 실험을 위한 오디오환경 Adaptive 후 Speech Signal 입력 출력 Input Signal with Noise Output Signal after Process 제4세부과제

ANC + Blind Source Separation Hyung-Min Park, Sang-Hoon Oh and Soo-Young Lee, 2001

Mic.1 Mic.2 Output 1 Output 2

Time Domain Approach to BSS Convolved mixtures Feedback architecture Learning rules

Frequency Domain Approach to BSS (1) In the frequency domain Complex score function Learning rule

Frequency Domain Approach to BSS (2) Performance limitation Contradiction between long reverberation covering and insufficient learning data Long reverberation long frame size Small number of frames insufficient learning data Mixtures combined from different time ranges of ICs Delayed mixtures

Filter Bank Approach to BSS[9] 2x2 network for the oversampled filter bank approach to BSS

Nonuniform Filter Bank Approach to BSS 2x2 network for the nonuniform oversampled filter bank approach to BSS

V. Independent Vector Analysis Independent Component Analysis Independent Vector Analysis[11] = X = X

Model Definition of IVA = X dependent Independent

VI. Summary ICA is motivated to solve cocktail party problem. It is also a method for data driven linear decomposition. InfoMax Algorithm Unifying Information Theoretic Framework Successful Applications in Many Areas IVA is motivated to solve BSS of convolutive mixtures in Frequency Domain.

References Te-Won Lee, M. Girolami, A. J. Bell, and T. J. Sejnowski, “A unifying information-theoretic framework for independent component analysis,” Computers and Mathematics with Applications, Vol. 39, pp. 1-21, 2000. A. J. Bell and T. J. Sejnowski, “The “independent components” of natural scenes are edge filters,” Vision Research, Vol. 37, pp. 3327-3328, 1997. S. Amari, “Natural gradient works efficiently in learning,” Neural Computation, Vol. 10, pp. 251-276, Feb. 1998. T.-W. Lee, M. Girolami, and T. J. Sejnowski, “Independent component analysis using an extended infomax algorithm for mixed sub-Gaussian and super-Gaussian sources,” Neural Computation, Vol. 11, pp. 417-441, 1999. M. Girolami, A. Cichocki, and S.-I. Amari, "A common neural-network model for unsupervised exploratory data analysis and independent component analysis," IEEE Trans. Neural Networks, vol. 9, no. 6, Nov. 1998. S. Choi, A. Cichocki, and S. Amari, “Flexible independent component analysis,” J. VLSI Signal Processing-Systems forSignal, Image, and Video technology, Vol. 26, pp. 25-38, Aug. 2000.

References S.-H. Oh, A. Cichocki, S. Choi, S.-I. Amari, and S.-Y. Lee., "Comparison of ICA/BSS algorithms in noisy environment," Proc. ICONIP, vol. 2, pp. 1192-1197, Nov. 2000. Hyung-Min Park, Sang-Hoon Oh, and Soo-Young Lee, “Adaptive noise cancelling based on independent component analysis,” IEE Electronics Letters, vol. 38, no. 15, pp. 832-833, July 2002. Hyung-Min Park, Chandra Shekhar Dhir, Sang-Hoon Oh, and Soo-Young Lee, “A filter bank approach to independent component analysis for convolved mixtures,” Neurocomputing, Vol. 69, pp. 2065-2077, Oct. 2006. Jong-Hwan Lee, Sang-Hoon Oh, and Soo-Young Lee, “Binaural semi-blind dereverberation of noisy convoluted speech signals,” Accepted for publication in Neurocomputing. T. Kim, T. Eltoft, and T.-W. Lee, “Independent vector analysis: an extension of ICA to multivariate components,” Proc. ICA2006, pp. 165-172, 2006. J.-H. Lee, et al., “Independent vector analysis(IVA): multivariate approach for fMRI group study,” NeuroImage, Vol. 40, pp. 86-109, 2008.