1. 1 Peter Fox Data Analytics – ITWS-4963/ITWS-6965 Week 10a, April 7, 2015 Factor Analysis (v. PCA), Fischer Linear Discriminant

2. Factor Analysis Exploratory factor analysis (EFA) is a common technique in qualitative sciences for explaining the (shared) variance among several measured variables as a smaller set of latent (hidden/not observed) variables. EFA is often used to consolidate survey data by revealing the groupings (factors) that underlie individual questions. A large number of observable variables can be aggregated into a model to represent an underlying concept, making it easier to understand the data. 2

3. Examples E.g. business confidence, morale, happiness and conservatism - variables which cannot be measured directly. E.g. Quality of life. Variables from which to “infer” quality of life might include wealth, employment, environment, physical and mental health, education, recreation and leisure time, and social belonging. Others? Tests, questionnaires, visual imagery… 3

4. Relation among factors “correlated” (oblique) or “orthogonal” factors? E.g. wealth, employment, environment, physical and mental health, education, recreation and leisure time, and social belonging Relations? 4

5. Factor Analysis

6. PCA and FA? CFA analyzes only the reliable common variance of data, while PCA analyzes all the variance of data. An underlying hypothetical process or construct is involved in CFA but not in PCA. PCA tends to increase factor loadings especially in a study with a small number of variables and/or low estimated communality. Thus, PCA is not appropriate for examining the structure of data. 6

7. FA vs. PCA conceptually FA produces factors; PCA produces components Factors cause variables; components are aggregates of the variables

8. Conceptual FA and PCA

9. PCA and FA? If the study purpose is to explain correlations among variables and to examine the structure of the data, FA provides a more accurate result. If the purpose of a study is to summarize data with a smaller number of variables, PCA is the choice. PCA can also be used as an initial step in FA because it provides information regarding the maximum number and nature of factors. –Scree plots (Friday) More on this later… 9

10. The Relationship between Variables Multiple Regression –Describes the relationship between several variables, expressing one variable as a function of several others, enabling us to predict this variable on the basis of the combination of the other variables Factor Analysis –Also a tool used to investigate the relationship between several variables –Investigates whether the pattern of correlations between a number of variables can be explained by any underlying dimensions, known as ‘factors’ From: Laura McAvinue School of Psychology Trinity College Dublin

11. Uses of Factor Analysis  Test / questionnaire construction oFor example, you wish to design an anxiety questionnaire… oCreate 50 items, which you think measure anxiety oGive your questionnaire to a large sample of people oCalculate correlations between the 50 items & run a factor analysis on the correlation matrix oIf all 50 items are indeed measuring anxiety… All correlations will be high One underlying factor, ‘anxiety’  Verification of test / questionnaire structure oHospital Anxiety & Depression Scale oExpect two factors, ‘anxiety’ & ‘depression’

12. How does it work? Correlation Matrix –Analyses the pattern of correlations between variables in the correlation matrix –Which variables tend to correlate highly together? –If variables are highly correlated, likely that they represent the same underlying dimension Factor analysis pinpoints the clusters of high correlations between variables and for each cluster, it will assign a factor

13. Correlation Matrix Q1-3 correlate strongly with each other and hardly at all with 4-6 Q4-6 correlate strongly with each other and hardly at all with 1-3 Two factors! Q1Q2Q3Q4Q5Q6 Q11 Q2.9871 Q3.801.7651 Q4-.003-.08801 Q5-.051.044.213.9681 Q6-.190-.1110.102.789.8641

14. Factor Analysis Two main things you want to know… –How many factors underlie the correlations between the variables? –What do these factors represent? Which variables belong to which factors?

15. Steps of Factor Analysis 1. Suitability of the Dataset 2. Choosing the method of extraction 3. Choosing the number of factors to extract 4. Interpreting the factor solution

16. 1. Suitability of Dataset  Selection of Variables  Sample Characteristics  Statistical Considerations

17. Selection of Variables  Are the variables meaningful? Factor analysis can be run on any dataset ‘Garbage in, garbage out’ (Cooper, 2002)  Psychometrics The field of measurement of psychological constructs Good measurement is crucial in Psychology Indicator approach Measurement is often indirect Can’t measure ‘depression’ directly, infer on the basis of an indicator, such as questionnaire  Based on some theoretical / conceptual framework, what are these variables measuring?

18. Selection of Variables, Example

19. How would you group these faces?

20. Sample Characteristics  Size  At least 100 participants  Participant : Variable Ratio  Estimates vary  Minimum of 5 : 1, ideal of 10 : 1  Characteristics  Representative of the population of interest?  Contains different subgroups?

21. Statistical Considerations  Assumptions of factor analysis regarding data  Continuous  Normally distributed  Linear relationships  These properties affect the correlations between variables  Independence of variables  Variables should not be calculated from each other  e.g. Item 4 = Item 1 + 2 + 3

22. Statistical Considerations  Are there enough significant correlations (>.3) between the variables to merit factor analysis?  Bartlett Test of Sphericity  Tests H o that all correlations between variables = 0  If p <.05, reject H o and conclude there are significant correlations between variables so factor analysis is possible

23. Statistical Considerations  Are there enough significant correlations (>.3) between the variables to merit factor analysis?  Kaiser-Meyer-Olkin Measure of Sampling Adequacy  Quantifies the degree of inter-correlations among variables  Value from 0 – 1, 1 meaning that each variable is perfectly predicted by the others  Closer to 1 the better  If KMO >.6, conclude there is a sufficient number of correlations in the matrix to merit factor analysis

24. Statistical Considerations, Example All variables Continuous Normally Distributed Linear relationships Independent Enough correlations? Bartlett Test of Sphericity (χ 2 ; df ; p <.05) KMO

25. 2. Choosing the method of extraction  Two methods  Factor Analysis  Principal Components Analysis  Differ in how they analyse the variance in the correlation matrix

26. Variable Specific Variance Error Variance Common Variance Variance unique to the variable itself Variance due to measurement error or some random, unknown source Variance that a variable shares with other variables in a matrix When searching for the factors underlying the relationships between a set of variables, we are interested in detecting and explaining the common variance

27. Principal Components Analysis Ignores the distinction between the different sources of variance Analyses total variance in the correlation matrix, assuming the components derived can explain all variance Result: Any component extracted will include a certain amount of error & specific variance Factor Analysis Separates specific & error variance from common variance Attempts to estimate common variance and identify the factors underlying this Which to choose? Different opinions Theoretically, factor analysis is more sophisticated but statistical calculations are more complicated, often leading to impossible results Often, both techniques yield similar solutions V

28. 3. Choosing the number of factors to extract Statistical Modelling –You can create many solutions using different numbers of factors An important decision –Aim is to determine the smallest number of factors that adequately explain the variance in the matrix –Too few factors Second-order factors –Too many factors Factors that explain little variance & may be meaningless

29. Criteria for determining Extraction  Theory / past experience  Latent Root Criterion  Scree Test  Percentage of Variance Explained by the factors

30. Latent Root Criterion (Kaiser-Guttman) Eigenvalues –Expression of the amount of variance in the matrix that is explained by the factor –Factors with eigenvalues > 1 are extracted –Limitations Sensitive to the number of variables in the matrix More variables… eigenvalues inflated… overestimation of number of underlying factors

31. Scree Test (Cattell, 1966) Scree Plot –Based on the relative values of the eigenvalues –Plot the eigenvalues of the factors –Cut-off point The last component before the slope of the line becomes flat (before the scree)

32. Elbow in the graph Take the components above the elbow

33. Percentage of Variance Percentage of variance explained by the factors –Convention –Components should explain at least 60% of the variance in the matrix (Hair et al., 1995)

34. 3. Choosing the number of factors to extract Three components with eigenvalues > 1 Explained 67.26% of the variance

35. BFI data in psych (R) 35

36. 4. Interpreting the Factor Solution Factor Matrix –Shows the loadings of each of the variables on the factors that you extracted –Loadings are the correlations between the variables and the factors –Loadings allow you to interpret the factors Sign indicates whether the variable has a positive or negative correlation with the factor Size of loading indicates whether a variable makes a significant contribution to a factor –≥.3

37. Component 1 – Visual imagery tests Component 2 – Visual imagery questionnaires Component 3 – ? VariablesComponent 1Component 2Component 3 Vividness Qu-.198-.805.061 Control Qu.173.751.306 Preference Qu.353.577-.549 Generate Test-.444.251.543 Inspect Test-.773.051-.051 Maintain.734-.003.384 Transform (P&P) Test.759-.155.188 Transform (Comp) Test -.792.179.304 Visual STM Test.792-.102.215

38. Factor Matrix Interpret the factors Communality of the variables –Percentage of variance in each variable that can be explained by the factors Eigenvalues of the factors –Helps us work out the percentage of variance in the correlation matrix that the factor explains

39. Communality of Variable 1 (Vividness Qu) = (-.198) 2 + (-.805) 2 + (.061) 2 =. 69 or 69% Eigenvalue of Comp 1 = ( [-.198] 2 + [.173] 2 + [.353] 2 + [-.444] 2 + [-.773] 2 +[.734] 2 + [.759] 2 + [-.792] 2 + [.792] 2 ) = 3.36 3.36 / 9 = 37.3% VariablesComponent 1Component 2Component 3 Communality Vividness Qu-.198-.805.061 69% Control Qu.173.751.306 69% Preference Qu.353.577-.549 76% Generate Test-.444.251.543 55% Inspect Test-.773.051-.051 60% Maintain.734-.003.384 69% Transform (P&P) Test.759-.155.188 64% Transform (Comp) Test -.792.179.304 75% Visual STM Test.792-.102.215 69% Eigenvalues3.361.6771.018/ % Variance37.3%18.6%11.3%/

40. Factor Matrix Unrotated Solution –Initial solution –Can be difficult to interpret –Factor axes are arbitrarily aligned with the variables Rotated Solution –Easier to interpret –Simple structure –Maximises the number of high and low loadings on each factor

41. Factor Analysis through Geometry It is possible to represent correlation matrices geometrically Variables –Represented by straight lines of equal length –All start from the same point –High correlation between variables, lines positioned close together –Low correlation between variables, lines positioned further apart –Correlation = Cosine of the angle between the lines

42. 60º 30º V1 V2 V3 The smaller the angle, the bigger the cosine and the bigger the correlation V1 & V3 90º angle Cosine = 0 No relationship V1 & V2 30º angle Cosine =.867 r =.867 V2 & V3 60º angle Cosine =.5 R =.5

43. V1 V5 V4 Factor Loading Cosine of the angle between each factor and the variable Factor Analysis Fits a factor to each cluster of variables Passes a factor line through the groups of variables V2 V3 V6 F1 F2

44. V1 V5 V4 V2 V3 V6 F1 F2 V1 V5 V4 V2 V3 V6 F1 F2 Two Methods of fitting Factors Orthogonal Solution Factors are at right angles Uncorrelated Oblique Solution Factors are not at right angles Correlated

45. V1 V5 V4 V2 V3 V6 F1 F2 Two Step Process Factors are fit arbitrarily Factors are rotated to fit the clusters of variables better V1 V5 V4 V2 V3 V6 F1 F2

46. VariablesC1C2C3 Vividness Qu-.198-.805.061 Control Qu.173.751.306 Preference Qu.353.577-.549 Generate Test-.444.251.543 Inspect Test-.773.051-.051 Maintain Test.734-.003.384 Transform (P&P) Test.759-.155.188 Transform (Comp) Test -.792.179.304 Visual STM Test.792-.102.215 VariablesC1C2C3 Vividness Qu-.029-.831.008 Control Qu.174.744.323 Preference Qu-.010.679-.547 Generate Test-.197.112.709 Inspect Test-.717-.103.279 Maintain Test.819.116.043 Transform (P&P) Test.779-.013-.166 Transform (Comp) Test -.599-.01.626 VisualSTM Test.813.045-.147 Unrotated Solution Solution following Orthogonal Rotation For example…

47. Factor Rotation Changes the position of the factors so that the solution is easier to interpret Achieves simple structure –Factor matrix where variables have either high or low loadings on factors rather than lots of moderate loadings

48. Evaluating your Factor Solution Is the solution interpretable? –Should you re-run and extract a bigger or smaller number of factors? What percentage of variance is explained by the factors? –>60%? Are all variables represented by the factors? –If the communality of one variable is very low, suggests it is not related to the other variables, should re-run and exclude

49. VariablesC1C2C3 Vividness Qu-.029-.831.008 Control Qu.174.744.323 Preference Qu-.010.679-.547 Generate Test-.197.112.709 Inspect Test-.717-.103.279 Maintain Test.819.116.043 Transform (P&P) Test.779-.013-.166 Transform (Comp) Test -.599-.01.626 VisualSTM Test.813.045-.147 First SolutionSecond Solution For example… VariablesComponent 1Component 2 Vividness Qu.013-.829 Control Qu-.023.770 Preference Qu.195.648 Generate Test-.493.130 Inspect Test-.760-.146 Maintain Test.711.183 Transform (P&P) Test.773.042 Transform (Comp) Test -.811-.028 Visual STM Test.792.103 Component 3 = ? C1 = Efficiency of objective visual imagery C2 = Self-reported imagery efficacy

54. We are looking for an eigenvalue above 1.0. Cumulative percent of variance explained.

58. Expensive Exciting Luxury Distinctive Not Conservative Not Family Not Basic Appeals to Others Attractive Looking Trend Setting Reliable Latest Features Trust

59. Expensive Exciting Luxury Distinctive Not Conservative Not Family Not Basic Appeals to Others Attractive Looking Trend Setting Reliable Latest Features Trust What shall these components be called?

60. Expensive Exciting Luxury Distinctive Not Conservative Not Family Not Basic Appeals to Others Attractive Looking Trend Setting Reliable Latest Features Trust EXCLUSIVE TRENDY RELIABLE

61. = (Expensive + Exciting + Luxury + Distinctive – Conservative – Family – Basic)/7 = (Appeals to Others + Attractive Looking + Trend Setting)/3 = (Reliable + Latest Features + Trust)/3 EXCLUSIVE TRENDY RELIABLE Calculate Component Scores

63. Not much differing on this dimension.

66. References Some… Cooper, C. (1998). Individual differences. London: Arnold. Kline, P. (1994). An easy guide to factor analysis. London: Routledge.

67. Assignment to come… Assignment 7: Predictive and Prescriptive Analytics. Due ~ week ~12. 20%.. 67