Dimension Reduction via PCA (Principal Component Analysis)

Motivation Derivation/calculation of PCA PCA in practice Things to note Examples in R and Python

Motivation In “big data” we often discuss how to handle big n (number of rows). However, often there are issues with big p (number of variables/parameters) Additionally, variables are often correlated which causes some to be redundant at some level. e.g. Web traffic data has visits and page views. Often correlated. We would like a way to scale down p in order to analyze a smaller, uncorrelated subset of data. We can use this smaller data set in other algorithms like linear regression, clustering, etc.

Geometric Introduction We begin with a sample dataset of 20 observations We have two dimensions, or variables, called x and y X and Y are correlated We’d like to somehow project the data onto 1 dimension in an intelligent way Note: y is not a dependent variable here. Think of x and y as something like height and weight and we want to predict basketball performance

Geometric Introduction Projecting raw data onto the x-axis

Geometric Introduction Projecting the raw data onto the y-axis

Geometric Introduction We’d like to rotate and stretch our data with a linear combination so that our data is uncorrelated and “stretched” as much as possible Rotating the data, we get the new points in red. Data in red has more separation amongst the points.

Geometric Introduction Our new X and Y are now uncorrelated.

Geometric Introduction Now if we project our rotation onto one dimension, we still have a lot of variation amongst our points. Doing a similar projection onto one dimension for the raw data does not yield as much variation.

Algebraic motivation/explanation We want to find a linear combination, a, that maximizes the variation of x Maximize 𝑣𝑎𝑟( 𝒂’𝒙) = 𝒂’𝑺𝒂 where 𝑆 is the variance of 𝒙 Since a is arbitrary and can increase by scaling a, we normalize 𝒂’𝑺𝒂 and maximize λ where 𝝀= 𝒂 ′ 𝑺𝒂 𝒂 ′ 𝒂 𝝀 𝒂 ′ 𝒂= 𝒂 ′ 𝑺𝒂 𝒂 ′ 𝑺𝒂 −𝝀 𝒂 ′ 𝒂 = 0 𝒂 ′ 𝑺𝒂 −𝝀𝒂 = 0 𝑺 −𝝀 𝑰 a = 0 Implies 𝒂 is an eigenvector with eigenvalue λ

Algebraic motivation/explanation (cont) For each parameter, p, we continue to maximize the variation such that 𝒂𝒌 is orthogonal to the other 𝒂 𝒚𝒊𝟏= 𝒂𝟏 ′ 𝒙𝒊 𝒚𝒊𝟐= 𝒂𝟐 ′ 𝒙𝒊 𝒚𝒊𝒑= 𝒂𝒑 ′ 𝒙𝒊 This gives us 𝑌 𝑛 𝑥 𝑝 = 𝑋 𝑛 𝑥 𝑝 ∗ 𝐴 𝑝 𝑥 𝑝 where Y are the transformation or “scores”. We can use just a few rows of 𝒀 to approximate 𝑿

Algebraic motivation/explanation (cont) If we take a subset of A we can use a lower dimension 𝑌 𝑛 𝑥 1 = 𝑋 𝑛 𝑥 𝑝 ∗ 𝐴 𝑝 𝑥 1 And since A is orthogonal, A’ = A-1 𝑌 𝑛 𝑥 1 ∗ 𝐴′ 1 𝑥 𝑝 = 𝑋 𝑛 𝑥 𝑝

PCA in Practice Run principal component analysis algorithm Determine number of components to use: Three common ways to do this: Retain enough components to account for k% of the total variability (where k is 80 or 90 or . . .) Accounted variability for 𝑖𝑡ℎ component can be found by 𝜆 𝑖 𝑖 𝜆 𝑖 Retain components whose variance (eigenvalue) is greater than the average eigenvalue (Σλi /p) Use scree plot to find natural break between “large/important” components and “small/unimportant” components Note: none of the 3 above methods have theoretical justification so sometimes, use what makes the most sense Use the “scores” with the corresponding number of dimensions in new algorithm

Things to Note PCA is NOT scale invariant!!! Changing the units of each column will change the results of PCA. Parameters with high variance will influence the principal components more. A good best practice is to center and scale your variables (prcomp in R does not scale automatically) Because PCA is based off of orthogonal vectors, the signs for each vector are arbitrary. Meaning you could run PCA on the same data set and get all the same numbers except the sign would be flipped. This doesn’t really change anything but it is good to know. One downfall of PCA is we usually lose the interpretability of variables.

Examples See code