1. Unsupervised Learning - PCA The neural approach->PCA; SVD; kernel PCA Hertz chapter 8 Presentation based on Touretzky + various additions

28. Use a set of pictures of faces to construct a PCA space. Shown are first 25 principal components. (C. DeCoro)

30. Variance as function of number of PC

31. Each iteration, from left to right, corresponds to addition of 8 principal components

32. Matching general images to faces using eigenfaces space

34. Singular Value Decomposition Singular Value Decomposition This can be rewritten in the matrix representation where ∑is a (non-square) diagonal matrix, and U,V are orthogonal matrices. Ordering the non-zero elements of ∑ in descending order, we can get an approximation of lower rank r of the matrix X by choosing =0 for j>r leading to SVD involves expanding an mxn matrix X of rank k=min(m,n) into a sum of k unitary matrices of rank 1, in the following way:

35. Singular Value Decomposition continued This is the best approximation of rank r to X, i.e. it leads to the minimal sum of square deviations

36. Relation between SVD and PCA SVD uses the same unitary transformations as PCA performed on the rows or columns of X (using the columns or rows as feature spaces). The singular values of SVD are the square toots of the eigenvalues of PCA.

37. SVD – Singular Value Decomposition

38. Applications of SVD  Dimensionality reduction, compression  Noise reduction  Pattern search, clustering Example: microarray of expression data, “ DNA chips ”

39. Affymetrix GeneChip ® probe array Image courtesy of Affymetrix

40. Hybridization of tagged probes Image courtesy of Affymetrix

41. Microarray Experiment Result

42. SVD example: gene expressions of nine rats

43. SVD – Singular Value Decomposition

44. Alter et al 2000: identify eigengenes that are responsible for yeast cell cycle

45. Processing: SVD

46. Processing: SVD – 1 dimension

47. Processing: SVD – 10 dimensions

48. Processing: SVD – 100 dimension

49. Kernel PCA