1. Independent Component Analysis
Reference:
Independent Component Analysis: A Tutorial
by Aapo Hyvarinen,

2. Motivation of ICA The Cocktail-Party Problem
在Party上三個人在不同的位置,同時在說話 (S)
三人的聲音混雜在一起,無法分辨出誰說了什麼
利用三隻麥克風，在不同的地點收聽會場中的聲音 (X)
是否可以將麥克風所錄到的聲音(X)還原回三個人原來的講話聲音 (S)
提出問題:雞尾酒酒會問題
Demo

3. Formulation of ICA
two speech signal s1(t) and s2(t), received by two microphones, the mixed signals are: x1(t) and x2(t):
It will be very useful if we could estimate the original signals s1(t) and s2(t), from only the recorded signals x1(t) and x2(t)
對問題做定義

4. Formulation of ICA
Suppose aii’s are known, then solving the linear Equations 1 and 2 can retrieve the s1(t), s2(t)
the problem is we don’t know the aii.
One approach is to use some information on the statistical properties of signals si(t) to estimate aii
Assume s1(t) and s2(t) are statistical independent, then Independent Component Analysis techniques can retrieve s1(t) and s2(t), from the mixture x1(t) and x2(t).

5. Original signals s1(t), s2(t) Mixture signals x1(t), x2(t) Recovered
signals for
s1(t), s2(t)
範例圖

6. Definition of ICA
For n linear mixtures x1, …,xn from n independent components
The independent component si are latent variables, meaning that they can not be directly observed, and the mixing matrix A is assumed to unknown.
We would like to estimate both A and s using the observable random vector x and some statistical assumption

7. Definition of ICA X=AS ; Y=BX ; y is a copy of s
If C is non-mixing then y=Cs is a copy of s
A square matrix is said to be non-mixing if it has one and only one non-zero entry in each row and each column

8. Illustration of ICA
We use two independent components with the following uniform distributions to illustrate the ICA model:
The distribution has zero mean and the variance equal to one
Let us mixing these two independent components with the following mixing matrix
This gives us two mixed variable x1 and x2.
The mixed data has a uniform distribution on a parallelogram.
But x1 and x2 are not independent any more. Since when x1 attains to its maximum, or minimum, then this also determine the value of x2

9. Illustration of ICA S2
Fig 6. Joint density distribution of the observed mixtures x1 and x2
Fig 5. Joint density distribution of the original signal s1 and s2

10. Illustration of ICA
The problem of estimating the data model of ICA is now to estimate the mixing matrix A0 using only information contained in the mixtures x1 and x2 .
We can see from Fig 6 an intuitive way of estimating A:
the edges of the parallelogram are in the directions of the columns of A. That is estimate the ICA model by
first estimating the joint density of x1 and x2 , and then
locating the edges.
However, this only works for random variables with uniform distributions
We need a method that works for any types of distribution

11. Ambiguities of ICA Because y=Bx is just a copy of S:
we can not determine the variance (energies) of the independent components.
we can not determine the order of the independent components.
applying a permutation matrix P to x=As, i.e., x=AP-1Ps, then Ps is still like the original signals, and
AP -1 is just a new unknown mixing matrix, to be solved by the ICA algorithms,
the order of s will be changed.

12. Properties of ICA Independence
the variables y1 and y2 are said to be independent if information on the value of y1 does not give any information the value of y2, and vice versa.
Let p(y1, y2 ) be the joint probability density function (pdf) of y1 and y2, and let p(y1 ) be the marginal pdf of y1 then
y1 and y2 are independent if and only if the joint pdf is factorizable.
Thus, given two functions h1 and h2 , we always have

13. Properties of ICA Uncorrelated variables are only partly independent
Two variables y1 and y2 are said to be uncorrelated if their covariance is zero
If the variables are independent, they are uncorrelated, but the reverse is not true!
For example: sin(x) and cos(x) is dependent on x, but cov(sin(x),cos(x))=0

14. Gaussian variables are forbidden
The fundamental restriction in ICA is that the independent components must be nongaussian for ICA to be possible
assume the mixing matrix is orthogonal and si are gaussian, then x1 and x2 are gaussian, uncorrelated, and of unit variance.
The joint pdf is
the distribution is completely symmetric (shown in figure next page), it does not contain any info on the direction of the columns of the mixing matrix A.
Thus A can not be estimated

15. Fig 7. Multivariate distribution of two independent gaussian variables

16. ICA Basic
source separation by ICA must go beyond second order statistics
ignoring any time structure because the information contained in the data is exhaustively represented by the sample distribution of the observed vector
source separation can be obtained by optimizing a ‘contrast function’
i.e.,: a function that measure independence.

17. Measures of independence
Nongaussian is independent
The key to estimate the ICA model is the nongaussianity
The central limit theorem (CLT) tells us that the distribution of a sum of independent random variables tends toward a gaussian distribution. In other words,
a mixing of two independent signals usually has a distribution that is closer to gaussian than any of the two original signals
Suppose, we want to estimate y, one of the independent components of s from x,
let us denotes this by y=WTx=Siwixi, w is a vector to be determined
How can we use CLT to determine w so that it would equal to one of the rows of the inverse of A ?

18. Nongaussian is independent
let us make a change of variables,z = ATw
then we have y = wTx = wTAs = zTs = Sizisi
thus y=zTs is more gaussian than the original variables si
y becomes least gaussian, when it equals to one of si, a trivial way is to
let only one of the elements zi of z be nonzero
Maximizing the nongaussianity of wTx, gives us one of the independent components.

19. Measures of nongaussianity
To use nongaussianity in ICA, we must have a quantitative measure of nongaussianity of a random variable yi
Kurtosis
the classical measure of nongaussianity is kurtosis or the fourth-order cumulant
Assume y is of unit variance, then kurt(y)= E{y4}-3.
A kurtosis is simply a normalized fourth moment E{y4}
For a gaussian y, the fourth moment equals to 3(E{y2})2
thus, kurtosis is zero for a gaussian random variable.

20. Kurtosis Kurtosis can be both positive and negative
RV have a negative kurtosis are called subgaussian
subgaussian RV have typically a flat pdf, which is rather constant near zero, and very small for larger values
uniform distribution is a typical example for subgaussian
supergaussian RV have a spiky pdf, with a heavy tail
Laplace distribution is a typical example for supergaussian

21. Kurtosis (c)
Typically nongaussianity is measured by the absolute value of kurtosis.
Kurtosis can be measured by using the fourth moments of the sample data
if x1 and x2 are two independent RV, it holds
To illustrate a simple example what optimization landscape for kurtosis looks like, let us look at a 2-d model x=As.
We seek for one of the independent components as y = wTx
let z = ATw, then y = wTx = wATs = zTs = z1 s1 + z2 s2 ,

22. Kurtosis (c) Using additive property of kurtosis, then we have
kurt(y) = kurt (z1 s1)+ kurt(z2 s2)= z14 kurt (s1)+ z24 kurt (s2)
let’s apply a constraint on y that the variance of y is equal to 1, that is the same assumption concerning s1 and s2.
Thus, z: E{y2}=z12 + z22 =1, this means that the vector z is constrained to the unit circle on a 2-d plane.
The optimization becomes “what are the maxima of the function | kurt(y)| =| z14 kurt (s1)+ z24 kurt (s2)| on the unit circle”?
The maxima are the points where vector z is (0,1) or (0,-1).
These points correspond to where y equals one of si and -si.

23. Kurtosis (c)
In practice we could start from a weight vector w, and compute the direction in which the kurtosis of y=wTx is growing or decreasing most strongly based on the available sample x(1),…, x(T) of mixture vector x, and use a gradient method for finding a new vector w.
However, kurtosis has some drawbacks,
the main problem is that kurtosis can be very sensitive to outliers, in other words kurtosis is not a robust measure of nongaussianity.
In the following sections, we would like to introduce negentropy, whose properties are rather opposite to those of kurtosis.

24. Negentropy Negentropy is based on the information-theoretic entropy.
The entropy of a RV is a measure of the degree of randomness of the observed variables.
The more unpredictable and unstructured the variable is, the larger is its entropy.
Entropy is defined for a RV Y as:
A fundamental property of information theory for gaussian variable is it has the largest entropy among all random variables of equal variance.
Thus, entropy can be used to measure nongaussianity.

25. Negentropy
To obtain a measure the nongaussianity that is zero for a gaussian variable and always nonnegative, one often uses Negentropy J, which is defined as:
J(y)= H(yGauss)-H(y) (22)
where ygauss is a gaussian RV of the same covariance matrix as y.
the advantage of using Negentropy is it is in some sense the optimal estimator of nongaussianity, as far as statistical properties are concerned.
The problem in using negentroy is that it is still computationally very difficult.
Thus simpler approximations of negentropy seems necessary and useful.

26. Approximations of negentropy
The classical method of approximating negentropy is using higher-order-moments, for example
The RV y is assumed to be of zero mean and unit variance.
This approach still suffer from the nonrobustness as kurtosis
Another approximation were developed based on the maximum-entropy principle:
Where  is a Gaussian variable of zero mean and unit variance, and G is a nonquadratic function

27. Approximations of negentropy
Taking G (y) = y4, then (25) becomes (23)
suppose G is chosen to be slow growing as the following contrast functions:
This approximation is conceptually simple, fast to compute, and especially robustness.
A practical algorithm based on these contrast function will be presented in Section 6

28. Preprocessing - centering
Some preprocessing techniques make the problem of ICA estimation simpler and better conditioned.
Centering
Center variable x, i.e., subtract its mean vector m=E{x}, so as to make x a zero-mean variable.
This preprocessing is solely to simplify the ICA algorithms
After estimating the mixing matrix A with centered data, we can complete the estimation by adding the mean vector of s back to the centered estimates of s.
the mean vector of s is given by A-1m, m is the mean vector that was subtracted in the preprocessing

29. Preprocessing - whitening
Another preprocessing is to whiten the observed variables.
Whitening means to transform the variable x linearly so that the new variable x~ is white, i.e., its components are uncorrelated, and their variances equal unity.
In other words, variable x~ is white means the covariance matrix of x~ equals identity matrix:

30. Preprocessing - whitening
The correlation  between two variables x and y is
The covariance between x and y is
The covariance Cov (x, y) can be computed by
If two variable are uncorrelated then (x, y)= Cov(x, y) =0
Covariance matrix = I means that if x not equal to y, then Cov(x,y)=0.
if a matrix’s covariance matrix is white, then it is uncorrelated.

31. Preprocessing - whitening
Although uncorrelated variables are only partly independent, decorrelation (using second-order information) can be used to reduce the problem to a simpler form.
Unwhitened matrix A needs n2 parameters, but whitened matrix needs lesser (about half) parameters

32. Fig 5
Fig 6
Fig 10
The graph to the right shows data in Fig 6 has been whitened. The square depicts the distribution is clearly a rotated version of original square in Fig 5. All that is left is the estimation of a single angle that gives rotation.

33. Preprocessing - whitening
Whitening can be computed by eigenvalue decomposition (EVD) of the covariance matrix E{xxT}=EDET
E is the orthogonal matrix of eigenvectors of E{xxT}
D is a diagonal matrix of its eigenvalues, D=diag(d1,…,dn)
note that E{xxT} can be estimated in a standard way from the available sample of x(1), …, x(T).

34. Preprocessing - whitening
Whitening can now be computed by
Where D-1/2 can be computed by D-1/2 =diag(d1-1/2 ,…, dn-1/2 ).
It is easy to show ,
Using (34) and E{xxT}=EDET, then QED.
According to x=As, thus whitening transform the mixing matrix to a new , and
Since
the new mixing matrix is orthogonal

35. The FastICA Algorithm - FastICA for one unit
The FastICA learning rule finds a direction, i.e., a unit vector w such that the projection wTx maximizes nongaussianity, which is measured by the approximation of negentropy J(wTx).
The variance of y=wTx must be constained to unity, for the whitened data, this is equivalent to constraining the norm of w to unity, i.e., E{(wTx)}2}=||w||2 =1.
In the following algorithm, g denotes the derivative of the derivative of the nonquadratic function G.

36. FastICA for one unit The FastICA algorithm
1) choose an initial (e.g., random) weight vector w.
2) Let W+ =E{xg(wTx)}-h E{xg’ (wTx)}w
3) Let W = W+ / || W+ || , normalization improve stability
4) if not converged, go back to 2.
The derivation is as follows:
the optima of E{G(wTx)} under the constraint E{(wTx)}2}=||w||2 =1 are obtained at points, where F=E{xg(wTx)}-bw = (40)
Solving this equation by Newton’s method w+=w–h (F/2F)
The Jacobian matrix is F = ∂F/∂w = E{xxTg’ (wTx)}-bI
And the Hessian matrix of F(w) is

37. FastICA for one unit
In order to simplify the inversion of the Hessin matrix, the first part of the the Hessian is aproximated, as
Since the data is in a unit sphere, thus
E{xxTg’ (wTx)}  E{xxT} E{g’ (wTx)}= E{g’ (wTx)}I
thus the Hessian matrix becomes diagonal, and it can be easily inverted,
Then, the vector w can be approximated by Newton
method: w+ = w – h (F/2F) =
By multiplying b -E{g’(wTx)} on both side, then after algebraic simplification, it gives the FastICA iteration

38. FastICA for one unit (c)
Discussion:
Expectations must be replaced as estimates, which are sample means
to compute sample mean, ideally all the data available should be used, but for the computational complexity, only part of or small size of samples are used,
If convergence is not satisfactory, one may then increase the sample size.

39. FastICA for several units
To etsimate several independent components, we need to run FastICA algorithm using several units, with weight vectors w1,…, wn.
To prevent different vectors from converging to the same maxima we need to decorrelate the outputs w1T x,…, wnT x after every iteration.
A simple way of decorrelation is to estimate the independent components one by one.
when p independent components are estimated, i.e., w1,…, wp,
run the one-unit fixed-point algorithm for wp+1 , and
subtract from wp+1 the “projection” wTp+1 Cwj wj , j=1,…,p of the previously estimated p vectors, and then renormalize wp+1 :

40. FastICA for several units
The covariance matrix C=E{xxT} is equal to I, if the data is sphered.

41. Applications of ICA - Finding Hidden Factors in Financial Data
Some financial data such as currency exchange rates, or daily returns of stocks, that may have some common underling factors.
ICA might reveal some driving mechanisms that otherwise remain hidden.
In a recent study of a stock portfolio, it was found that ICA is a complementary tool to PCA, allowing the underlying structure of the data to be more readily observed.

42. Term project
Using PCA, JADE, and FastICA to analyzes the Taiwan stocks returns for underling factors.
JADE and FastICA packages can be found by searching on the www.
Data are available at course web site.
Due: