1. Psych 706: stats II
Class #7

2. Today: more regression!
Example of how ANOVA = Regression
Review last week’s class
More options for outlier detection and influence
More options to check for multicollinearity of predictors
Statistical power: How many subjects do I need?
Hierarchical linear regression
SPSS tutorial: Hierarchical linear regression

3. ANOVA = Regression example
IV = “jobcat” Employment Group (1 = clerical, 2 = custodial, 3 = manager)
DV = “prevexp” previous job experience (months)
Analyze  Compare Means  One Way ANOVA

4. Difference between each score and the grand mean One-Way ANOVA: Group
SS Total
Difference between each score and the grand mean
One-Way ANOVA: Group
SS Model
Difference between each group mean and the grand mean
CUSTODIAL
MEAN
SS Residual
Difference between each score and its group mean
GRAND MEAN
CLERICAL
MEAN
MANAGER
MEAN

5. ANOVA = Regression example
ANOVA Output

6. ANOVA = Regression example
IV = “jobcat” job group (1 = clerical, 2 = custodial, 3 = manager)
Regression: To run, need to DUMMY CODE groups
# dummy code columns = # groups minus 1, so in this case we need TWO
Clerical dummy code column: Clerical = 1, other 2 groups = 0
Custodial dummy code column: Custodial = 1, other 2 groups = 0
Since Managerial is 0 in both columns, it is known as the reference group to the other two groups

7. instead of the group variable “jobcat” we used in the ANOVA
Here are the two new dummy-coded groups (Clerical and Custodial) that we’ll use as IVs in the regression…
instead of the group variable “jobcat” we used in the ANOVA

8. Regression: Two dummy coded group variables

9. ANOVA = Regression example
Analyze Regression Linear
Our IV is now 2 dummy variables: Clerical and Custodial, entered in the box at the same time

10. Clerical Group GRAND MEAN
SS total = differences between each data point and GRAND mean
SS model = differences between slope and GRAND mean
GRAND MEAN
SS residual = differences between each data point and slope

11. Custodial Group GRAND MEAN
SS total = differences between each data point and GRAND mean
SS model = differences between slope and GRAND mean
GRAND MEAN
SS residual = differences between each data point and slope

12. ANOVA = Regression example
TEST STATISTIC = VARIANCE EXPLAINED BY MODEL = EFFECT
VARIANCE NOT EXPLAINED BY MODEL ERROR
ANOVA
Model Effect is comparing the mean difference for each group to the GRAND mean
Model Error is comparing individual data points to its GROUP mean
Regression
Model Effect is comparing mean value of clerical and custodial slopes to GRAND mean
Model Error is comparing individual data points to each slope

13. ANOVA = Regression example
Overall Regression ANOVA output
Overall One-Way ANOVA output

14. ANOVA = Regression example
Intercept (B0) = Mean of our reference group, Managers (assigned the value of 0 for both dummy-coded group variables)

15. ANOVA = Regression example
Intercept (B0) = Mean of our reference group, Managers
Clerical (B1) = difference between Clerical and Managers
85.04 – = 7.42

16. ANOVA = Regression example
Intercept (B0) = Mean of our reference group, Managers
Clerical (B1) = difference between Clerical and Managers
Custodial (B2) = difference between Custodial and Managers
– =

17. Any questions?

18. Regression review Example: Outcome Y = album sales
Two predictors X1 = (sqrt transformed) advertising; X2 = radio plays (let’s forget about band attractiveness)
Last week we ran a simple regression with advertising predicting album sales
Now we’ll run a multiple regression with advertising AND radio plays predicting album sales
Along the way, we’ll review concepts from last week

19. Regression review Example: Outcome Y = album sales
Two predictors X1 = (sqrt transformed) advertising; X2 = radio plays (let’s forget about band attractiveness)
Unstandardized regression:
Y = b0 + b1X1 + b2X2 + error
Need intercept to show where line starts for your particular data set
Standardized regression:
Y = b1X1 + b2X2 + error
Predictor and outcome variables go into the model z-scored (all on same scale)
Do not need the intercept; all betas are correlation coefficients (Pearson’s r)
More easily generalized to other populations

20. Multiple Regression Standardized Multiple Regression One DV
Album Sales
Two IVs (z-scored)
Adverts
# Plays on Radio
Note: they are entered at the same time

21. Regression review
Pearson’s r = How much predictors and outcomes covary
R² = How much variance in Y is accounted for by our predictor(s)
Durbin-Watson test = Amount of correlations among errors/residuals (Less than 1 or greater than 3 is a problem)
Adjusted R² = How much variance would be accounted for if the model had been derived from the population from which this sample was taken

22. Multiple regression

23. Regression review Test of overall regression model = F test
Overall, does the model you created predict (or share variance with) the outcome?
Test of individual predictors (betas) = t tests
Is the slope of each beta different from zero?

24. MULTIPLE REGRESSION

25. MULTIPLE REGRESSION

26. Regression review Outlier detection and their influence on the model
Multicollinearity
Homoscedasticity
Residuals independent of each other
Residuals normally distributed
Linearity/normality of predictors and outcome
Non-zero variances

27. Univariate vs multivariate
Univariate: involving one variable
Multivariate: involving two or more variables
Multiple Regression: need to deal with potential cases that are outliers on more then one dimension

28. Multivariate OUTLIER DETECTION AND INFLUENCE
Influence: A case is said to be influential if removing the case substantially changes the estimate of beta coefficients.
Mahalanobis Distance
Cook’s Distance
Leverage
Standardized Residuals
Standardized df Betas
Standardized df Fit

29. distances
Calculating distances between individual cases and the mean of the data to identify any cases that have excessive influence on results of your model
(1) Mahalanobis Distance
(2) Cook’s Distance
(3) Leverage
These are each saved as individual new variables in your SPSS file so you can graph them and look for influential outliers

30. Distances
We sometimes measure "nearness" or "farness" in terms of the scale of the data (standard deviation)
Individual score 1 SD from mean is closer than another score that is 2 SD
You could measure the distance between scores in SD units!
How far away you are from the mean in SD units also tells you something about the probability of you seeing this distance in a normal distribution (z-scores)
Two variables: X and Y
(Univariate Example)

31. Mahalanobis distance
For multivariate data (2+ predictors) we instead graph prediction ellipses
Accounts for the fact that the variances in each direction are different
Accounts for the covariance between variables
The further out a particular case is, the more likely it’s an outlier for more than 1 predictor

32. Mahalanobis distance
FUN FACT: Initial motivation for developing this measure was to analyze and classify human skulls into groups, based on various properties (Mahalanobis, 1927) Mahalanobis, Prasanta Chandra (1927). Analysis of race blend in Bengal. Journal and Proceedings of the Asiatic Society of Bengal, 23, 301–333

33. Distance of individual cases from the mean of the predictor variable
Mahalanobis distance
Distance of individual cases from the mean of the predictor variable

34. Cook’s distance
Say you were making a soup and deciding what spices that would go well together
Some examples:
Basil, thyme, oregano
Cumin, coriander, turmeric
For some reason you decide to throw in a really pungent spice that has a distinct flavor profile from the other ones you used (cloves, nutmeg or anise)
It screws up the otherwise harmonious blend of the soup and overwhelms it so that the only thing you can taste is that one bad spice
What if we could go back and see what our soup would be like without that nasty spice in there?

35. Cook’s distance
Measures the sum of squared deviations between the observed values and the hypothetical values we would get if we deleted a particular case
Shows you influential scores

36. Cook’s distance
Measure of overall influence of one case on the overall model
Values > 1 are problematic

37. leverage
One definition: Influence or power used to get a certain result
Example in real life: blackmail peeps to get dough

38. leverage
In a regression context, it’s one particular case or cases that significantly influence what the slope (beta) of the regression line looks like

39. leverage
Leverage points = cases made at extreme or outlying values of the independent variables
Lack of neighboring cases means that the fitted regression model will pass close to that particular case!

40. LEVERAGE Anything above this # is problematic:
3*[(#predictors +1) / sample size]
3[2+1/200] = .045
Here values > .045 might be a problem

41. residuals
What is left over of total sums of squares after you fit your model sums of squares
How well or poorly your individual data points predict the regression line

42. RESIDUALS
Standardized Residuals save the errors from your model as z-scores
Under the normal distribution (z), we would expect:
95% of errors to be between +/- 1.96
99% of errors to be between +/- 2.58
99.9% of errors to be between +/- 3.29
Typically look at scores of +/-3 or greater as a problem, meaning that our model including these scores is a POOR representation of our actual data

43. influence statistics Standardized df Beta
Shows how much predictors’ (Xs) beta coefficients would change if a particular case was dropped from the model
Absolute values greater than 2 suggest that the case has a PROBLEMATIC influence on the model
Standardized df Fit
Shows how much predicted value of the outcome Y would change if a particular case was dropped from the model

44. Checks for multicollinearity
When multicollinearity ↑
Beta coefficients in model become unstable
Standard errors for betas get wildly inflated!
Variance Inflation Factor (VIF)
How much variance increased because of multicollinearity of predictors
If VIF > 10, you have a problem
Tolerance
1-R²
% variance in predictor that cannot be accounted for by the other predictors, hence very small values indicate that a predictor is redundant
Lower than .10 is a PROBLEM

45. Checks for multicollinearity

46. Checks for multicollinearity

47. Regression: Power
Based on sample size, # of predictors, and effect size you want to detect
R² = .02 (small effect)
fuggedaboudit
R² = .13 (medium effect)
With N=160 and up to 20 predictors, you’ll have enough power
If 6 or less predictors, N=100 should work
R² = .26 (large effect)
With N=77 and up to 20 predictors, you’ll have enough power

48. Hierarchical regression
Theory, not math, driven (do not use FORWARD, BACKWARD, or STEPWISE options in SPSS)
Entering known predictors first into model (based on literature) using ENTER option in SPSS
Then ENTERING (adding) new predictors at each block to see if they predict incremental/unique variance to your model
R² will always increase with the more predictors you add to the model, but we can test and see if it’s significantly bigger
F –test and R² change calculated for each block in the model

49. Hierarchical regression example
Why do you like your lecturers?
Hypothesis: students like lecturers who are like themselves
Predictors (X’s)
Gender (women = -1, men = + 1) and age of student
Student’s NEO-FFI Conscientiousness score
Student’s NEO-FFI Neuroticism score
Outcome variable (Y): Conscientiousness of lecturer (scale goes from -5 through 0 to 5)
-5 = I don’t want this characteristic at all in my lecturers
0 = this characteristic is not important
5 = I really want this characteristic in my lecturers

50. Hierarchical regression example
Z-scored Age and Gender together in first block

51. Hierarchical regression example
Z-scored student conscientiousness in second block

52. Hierarchical regression example
Z-scored student neuroticism in third block

53. Hierarchical regression example
NOTE: we are clicking the R² change box (NEW!) because we have multiple blocks of predictors to evaluate

54. Hierarchical regression example
Getting distances, residuals, and influence stats

55. Hierarchical regression example
You could create plots, etc. but we’re not going to do that here for sake of time…
Let’s get it started!!

56. Three blocks of predictors
Regression output
Three blocks of predictors

57. Looking at R² Change Scores and their corresponding ANOVAs
Gender and age contribute significant unique variance to preference for lecturer conscientiousness, R² change = .026, F(2, 385) = 5.202, p=.006.
Student conscientiousness adds significant unique variance to preference for lecturer conscientiousness, R² change = .034, F(1, 384) = , p<.001.

63. At this point, What do we conclude?
Across models, age and neuroticism are not significant predictors of how much conscientiousness students want their lecturers to have
Gender (women > men) (1st/2nd model) and higher student conscientiousness (2nd/3rd model) predict higher levels of conscientiousness students want their lecturers to have (you can use standardized betas to relate to SD relationships)
Potential issues: Mahalanobis/Cooks distances and Leverage values may be problematic; 2 residuals outside 3 SD
Need to check for normality, residuals assumptions, etc. before making any firm conclusions about the data

64. Next up, SPSS tutorial