CS621 : Artificial Intelligence Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 28: Principal Component Analysis; Latent Semantic Analysis

Desired Features of the Search Engines Meaning based –More relevant results Multilingual –Query in English, e.g. –Fetch document in Hindi, e.g. –Show it in English

Precision (P) and Recall (R) Tradeoff between P and R Actual (A) Obtained (O) Intersection: shaded area (S) P= S/O R= S/A

Impediments to Good P and R Synonymy: A word in the document will not match its synonym in the query, bringing down Recall –E.g., “Planes for Bangkok”: query –“Flights to Bangkok”: text in the document Polysemy: A word in the query will bring up documents containing the same word, but used in a different sense, bringing down Precision –E.g., “Planes for Bangkok”: query –“Cartesian Planes”: text in the document

Principal Component Analysis

Eaample: IRIS Data (only 3 values out of 150) IDPetal Length (a 1 ) Petal Width (a 2 ) Sepal Length (a 3 ) Sepal Width (a 4 ) Classific ation Iris- setosa ,Iris- versicol or Iris- virginica

Training and Testing Data Training: 80% of the data; 40 from each class: total 120 Testing: Remaining 30 Do we have to consider all the 4 attributes for classification? Do we have to have 4 neurons in the input layer? Less neurons in the input layer may reduce the overall size of the n/w and thereby reduce training time It will also likely increase the generalization performance (Occam Razor Hypothesis: A simpler hypothesis (i.e., the neural net) generalizes better

The multivariate data X 1 X 2 X 3 X 4 X 5 … X p x 11 x 12 x 13 x 14 x 15 … x 1p x 21 x 22 x 23 x 24 x 25 … x 2p x 31 x 32 x 33 x 34 x 35 … x 3p x 41 x 42 x 43 x 44 x 45 … x 4p … x n1 x n2 x n3 x n4 x n5 … x np

Some preliminaries Sample mean vector: For the i th variable: µ i = (Σ n j=1 x ij )/n Variance for the i th variable: σ i 2 = [Σ n j=1 (x ij - µ i ) 2 ]/ [n-1] Sample covariance: c ab = [Σ n j=1 ((x aj - µ a )(x bj - µ b ))]/ [n-1] This measures the correlation in the data In fact, the correlation coefficient r ab = c ab / σ a σ b

Standardize the variables For each variable x ij Replace the values by y ij = (x ij - µ i )/σ i 2 Correlation Matrix

Short digression: Eigenvalues and Eigenvectors AX=λX a 11 x 1 + a 12 x 2 + a 13 x 3 + … a 1p x p =λx 1 a 21 x 1 + a 22 x 2 + a 23 x 3 + … a 2p x p =λx 2 … a p1 x 1 + a p2 x 2 + a p3 x 3 + … a pp x p =λx p Here, λs are eigenvalues and the solution For each λ is the eigenvector

Short digression: To find the Eigenvalues and Eigenvectors Solve the characteristic function det(A – λI)=0 Example: Characteristic equation (-9-λ)(-6- λ)-28=0 Real eigenvalues: -13, -2 Eigenvector of eigenvalue -13: (-1, 1) Eigenvector of eigenvalue -2: (4, 7) Verify: = λ 0 I= 0 λ

Next step in finding the PCs Find the eigenvalues and eigenvectors of R

Example 49 birds: 21 survived in a storm and 28 died. 5 body characteristics given X 1 : body length; X 2 : alar extent; X 3 : beak and head length X 4 : humerus length; X 5 : keel length Could we have predicted the fate from the body charateristic

Eigenvalues and Eigenvectors of R ComponentEigen value First Eigen- vector: V 1 V2V2 V3V3 V4V4 V5V

Which principal components are important? Total variance in the data= λ 1 + λ 2 + λ 3 + λ 4 + λ 5 = sum of diagonals of R= 5 First eigenvalue= ≈ 72% of total variance 5 Second ≈ 10.6%, Third ≈ 7.7%, Fourth ≈ 6.0% and Fifth ≈ 3.3% First PC is the most important and sufficient for studying the classification

Forming the PCs Z 1 = 0.452X X X X X 5 Z 2 = X X X X X 5 For all the 49 birds find the first two principal components This becomes the new data Classify using them

For the first bird X 1 =156, X 2 =245, X 3 =31.6, X 4 =18.5, X 5 =20.5 After standardizing Y 1 =( )/3.65=-0.54, Y 2 =( )/5.1=0.73, Y 3 =( )/0.8=0.17, Y 4 =( )/0.56=0.05, Y 5 =( )/0.99=-0.33 PC 1 for the first bird= Z 1 = 0.45X(-0.54)+ 0.46X(0.725)+0.45X(0.17)+0.47X(0.05)+0.39X(- 0.33) =0.064 Similarly, Z 2 = 0.602

Reduced Classification Data Instead of Use X1X1 X2X2 X3X3 X4X4 X5X5 49 rows Z1Z1 Z2Z2 49rows

Other Multivariate Data Analysis Procedures Factor Analysis Discriminant Analysis Cluster Analysis

Latent Semantic Analysis and Singular Value Decomposition Slides based on “Introduction to Information Retrieval”, Manning, Raghavan and Schutze, Cambridge University Press, 2008.

Term Document Matrix Terms as rows Docs as columns

Singular value Decomposition of Term vs. Document Matrix

Low Rank Approximation Given an M× N matrix C and a positive integer k, we wish to find an M× N matrix C k of rank at most k, so as to minimize the Frobenius norm of the matrix difference X = C − C k, defined to be

Example: Term-Document Matrix

Singular Values

Trucated SVD Matrix Retain only the first two Singular Values

Pros and Cons A kind of soft clustering on terms Documents pass through the LSA processing So do the queries No known efficient method of computation currently (billions of documents!) IMP: tries to capture association, a recurring notion in AI