Exploratory Factor Analysis

Principal components analysis seeks linear combinations that best capture the variation in the original variables. Factor analysis seeks linear combinations that best capture the correlations among the original variables. Review: FA vs. PCA

Factor Analysis: Seeks to identify common factors that influence correlations among items, it is not a correlation matrix that is analyzed  thus, the main diagonal is replaced by the communalities - the variances in X that are due only to the common factors. But, communalities are not known in advance of the factor analysis, giving rise to the communality problem and the need to solve for the common factors iteratively. Review: FA vs. PCA Principal Components Analysis: The correlation matrix contains ones on the main diagonal, accounting for all of the variance in X—the goal of principal components analysis.

In the principal axes approach to factor analysis, the only difference compared to principal components is that the matrix being analyzed is a correlation matrix (which is also a variance- covariance matrix for standardized variables) in which the main diagonal contains the communalities rather than the variances. FA: Principal Axes Factoring

Most of the loadings on any given factor are small and a few loadings are large in absolute value Most of the loadings for any given variable are small, with ideally only one loading being large in absolute value. Any pair of factors have dissimilar patterns of loadings. Simple Structure

The ideal loadings are the simple structure, which might look like this for the factor loadings: To get closer to the ideal, we rotate the factors F1F1 F2F2 F3F3 X1X1 100 X2X2 100 X3X3 100 X4X4 010 X5X5 010 X6X6 010 X7X7 001 X8X8 001 X9X9 001

Varimax One way to approach this ideal pattern is to find the rotation that maximizes the variance of the loadings in the columns of the factor structure matrix. This approach was suggested by Kaiser and is called varimax rotation. F1F1 F2F2 F3F3 X1X1 100 X2X2 100 X3X3 100 X4X4 010 X5X5 010 X6X6 010 X7X7 001 X8X8 001 X9X9 001

Quartimax A second way to approach this ideal pattern is to find the rotation that maximizes the variance of the loadings in the rows of the factor structure matrix. This approach is called quartimax rotation. F1F1 F2F2 F3F3 X1X1 100 X2X2 100 X3X3 100 X4X4 010 X5X5 010 X6X6 010 X7X7 001 X8X8 001 X9X9 001

Equimax For those who want the best of both worlds, equimax rotation attempts to satisfy both goals. Varimax is the most commonly used and the three rarely produce results that are very discrepant. F1F1 F2F2 F3F3 X1X1 100 X2X2 100 X3X3 100 X4X4 010 X5X5 010 X6X6 010 X7X7 001 X8X8 001 X9X9 001

Data on drug use reported by 1634 students in Los Angeles. Participants rated their use on a 5-point scale: 1 = never tried, 2 = only once, 3 = a few times, 4 = many times, 5 = regularly. Example: Principal Components Analysis

FACTOR /VARIABLES cigs beer wine liquor cocaine tranqs drugstr heroin marijuan hashish inhale hallucin amphets /PRINT INITIAL KMO EXTRACTION ROTATION /PLOT EIGEN ROTATION /CRITERIA MINEIGEN(1) ITERATE(25) /EXTRACTION PC /CRITERIA ITERATE(25) /ROTATION VARIMAX /METHOD=CORRELATION. Syntax for Factor Analysis PC = Principal Components PAF = Principal Axis Factoring KMO asks for a test of multicollinearity – this will tell whether there is enough shared variance to warrant a factor analysis

Should the matrix be analyzed? This can vary between 0 and 1, indicating whether there is sufficient multicollinearity to warrant an analysis. Higher values indicate the desirability of a principal components analysis. Bartlett's test indicates the correlation matrix is clearly not an identity matrix

Two principal components account for nearly half of the information in the original variables

Initial Extraction All principal components are extracted so all of the variance in the variables is accounted for.

Part of the loading matrix. How should the first two principal components be interpreted?

Only two principal components are indicated by the scree test.

Only two principal components are indicated by the Kaiser rule as well.

With only two principal components, much less of the variance in each variable is accounted for.

The loadings for the first two components do not change when they are the only ones extracted.

Drug use data by 1634 students. Participants rated their use on a 5-point scale: 1 = never tried, 2 = only once, 3 = a few times, 4 = many times, 5 = regularly. Common Factor Analysis

The analysis begins in the same way as principal components analysis. It would make little sense to search for common factors in an identity matrix:

Unlike principal components analysis, factor analysis will not attempt to explain all of the variance in each variable. Only common factor variance is of interest. This creates the need for some initial estimates of communalities.

The number of factors to extract is guided by the size of the eigenvalues, as it was in principal components analysis. But, not all of the variance can be accounted for in the variables.

To the extent there is random error in the measures, the eigenvalues for factor analysis will be smaller than the corresponding eigenvalues in principal components analysis. Remember, this is because PCA assumes no error of measurement, whereas FA assumes there is measurement error.

The location of the factors might be rotated to a position that allows easier interpretation. This will shift the variance, but preserve the total amount accounted for.

Two factors appear to be sufficient. The attenuated eigenvalues will generally tell the same story.

The loadings will be reduced in FA compared to PCA because not all of the variance in X is due to common factor variance. PCA FA

Rotating the factors to simple structure makes the interpretation easier. The first factor appears to be minor “recreational” drug use. The second factor appears to be major “abusive” drug use.

A sample of 303 MBA students were asked to evaluate different makes of cars using 16 different adjectives rated on a 5-point agreement scale (1 = strongly disagree, 5 = strongly agree): “This car is an exciting car.” New Example: FA

Are all 16 individual ratings required to understand product evaluation, or, is there a simpler measurement model?

The analysis must begin with some idea of the proportion of variance in each variable that can be attributed to the common factors. The most common initial estimate is the squared multiple correlation between a given measure and all the remaining measures.

Three common factors appear to underlie the 16 evaluative ratings.

The three common factors account for two-thirds of the common factor variance.

Without rotation, the interpretation of the factors is not readily apparent, especially the second and third factors.

The varimax rotation makes the interpretation a bit easier. What would you name these factors? Factor 1? Factor 2? Factor 3?

It is rare for the different rotational criteria to produce different results

Advocates of factor analysis often claim that it is inappropriate to apply principal components procedures in the search for meaning or latent constructs. But, does it really matter all that much? To the extent that the communalities for all variables are high  the two procedures should give very similar results. When commonalities are very low  factor analysis results may depart from principal components. PCA vs. FA

PCAFA

PCAFA

PCAFA

Factor analysis offers a more realistic model of measurement than principal components analysis by admitting the presence of random and systematic error. Another way to make the model more realistic is to relax the restriction that factors be orthogonal. Allowing oblique factors has two potential benefits: It allows the model to better match the actual data It allows the possibility of higher order factors FA: Oblique Rotation

A sample of 303 MBA students were asked to evaluate different makes of cars using 16 different adjectives rated on a 5-point agreement scale (1 = strongly disagree, 5 = strongly agree): “This car is an exciting car.” FA Example, Oblique Rotation

Oblique rotation relaxes the requirement that factors be independent. This requires the addition of a new matrix— the factor pattern matrix. With orthogonal rotation, because the factors are independent, the weights are simply correlations---just the elements of the factor loading (structure) matrix. When the factors are not orthogonal, the correlations are not the only weights considered (structure matrix) and a separate matrix containing partial weights is necessary (pattern matrix).

Because the factor pattern matrix represents the unique contribution of each factor to the reconstruction of any variable in X, it provides a better basis for judging simple structure. Available techniques for achieving simple structure for oblique rotation (e.g., promax, direct oblimin) confront an additional problem---the specification for the amount of correlation among the factors. Unlike orthogonal rotation in which these correlations are fixed at 0, in oblique rotation, simple structure can be sought for any correlations among the factors.

Hypothetical data (N = 500) were created for individuals completing a 12-section test of mental abilities. All variables are in standard form.

The correlation matrix is not an identity matrix:

The scree test clearly shows the presence of three factors:

On average, the three factors extracted can account for about half of the variance in the individual subtests.

Factor analysis accounts for less variance than principal components and rotation shifts the variance accounted for by the factors. Why does the sum of the squared rotated loadings not equal the sum of the squared unrotated loadings?

The initial extraction...

Oblique rotation is much clearer in the pattern matrix than in the structure matrix:

The correlations among the factors suggest the presence of a higher order factor: Why are some of the correlations negative when one would expect all mental abilities to be positively correlated?

An alternative oblique rotation---Promax---provides much the same answer: The order of the factors may vary, and one gets reflected, but the essential interpretation is the same.

The correlations among the factors are similar for the two procedures.

Exploratory factor analytic methods are sometimes used as a crude way to confirm hypotheses about the latent structure underlying a data set. As a first pass, these methods do just fine. But, more powerful confirmatory factor analytic procedures exist that can better address questions about data that are strongly informed by theory.