Techniques for studying correlation and covariance structure Principal Components Analysis (PCA) Factor Analysis

Principal Component Analysis

Then where are eigenvectors of  of length 1 and are eigenvalues of  Lethave a p-variate Normal distribution with mean vector

or and have covariance matrix The Principal Components are defined by are independent with Var(C j ) = j

Many times for large value of j, Var(C j ) = j, is small and contributes little to the total variance In this case the number of variables can be reduced to the small number of principal components. In regression analysis it is sometimes useful to transform the independent variables into their principal components

Scree Plot Proportion of variance Principal Components

Scree Plot Cumulative Proportion of variance Principal Components

Example In this example wildlife (moose) population density was measured over time (once a year) in three areas.

picture Area 3 Area 1 Area 2

The Sample Statistics The mean vector The covariance matrix The correlation matrix

Principal component Analysis The eigenvalues of S The eigenvectors of S The principal components

The Example

Scree Plots

More Examples

Computation of the eigenvalues and eigenvectors of  Recall:

continuing we see that: For large values of n

The algorithm for computing the eigenvector 1.Compute rescaling so that the elements do not become to large in value. i.e. rescale so that the largest element is 1. 2.Computeusing the fact that: 3.Compute 1 using

4.Repeat using the matrix 5.Continue with i = 2, …, p – 1 using the matrix Example – Using Excel - EigenEigen

Factor Analysis An Alternative technique for studying correlation and covariance structure

Let The Factor Analysis Model: have a p-variate Normal distribution with mean vector Let F 1, F 2, …, F k denote independent standard normal observations (the Factors) Suppose that there exists constants ij (the loadings) such that: Let  1,  2, …,  p denote independent normal random variables with mean 0 and var(  i ) =  p x 1 = 11 F F 2 + … +  k F k +   x 2 = 21 F F 2 + … +  k F k +   … x p = p1 F 1 + p2 F 2 + … + pk F k +  p

Using matrix notation where and with

Note: hence i.e. the component of variance of x i that is due to the common factors F 1, F 2, …, F k. and i.e. the component of variance of x i that is specific only to that observation.

Determine cov(x i,F j ) Recall

Also where Thus Now, if also then ij is the correlation between x i and F j.

Rotating Factors Recall the factor Analysis model This gives rise to the vectorhaving covariance matrix: Let P be any orthogonal matrix, then and

Hence if is a Factor Analysis model then so also is with where P is any orthogonal matrix. with

The process of exploring other models through orthogonal transformations of the factors is called rotating the factors There are many techniques for rotating the factors VARIMAX Quartimax Equimax VARIMAX rotation attempts to have each individual variables load high on a subset of the factors

Example: Olympic decathlon Scores Data was collected for n = 160 starts (139 athletes) for the ten decathlon events (100-m run, Long Jump, Shot Put, High Jump, 400-m run, 110-m hurdles, Discus, Pole Vault, Javelin, 1500-m run). The sample correlation matrix is given on the next slide

Correlation Matrix

Identification of the factors