1. The State of Mediation Analysis in Psychological Science Issues and Solutions Rick Hoyle Duke University

2. York Psychology Department Colloquium, 9 November 2015 2 putative cause explanatory mechanism (behavioral) outcome

3. York Psychology Department Colloquium, 9 November 2015 3 2006 review of articles in psychology reporting mediation analysis: N = 291 distribution across areas of psychology 34% social 24% clinical 10% health remainder in developmental I/O cognitive methods program evaluation

4. York Psychology Department Colloquium, 9 November 2015 4

5. York Psychology Department Colloquium, 9 November 2015 5 review of social psychology articles published from 2005- 2009: at least one mediation test 59% of articles in JPSP 65% of articles in PSPB

6. York Psychology Department Colloquium, 9 November 2015 6 JPSP (latest issue)

7. York Psychology Department Colloquium, 9 November 2015 7 JCCP (latest issue)

8. York Psychology Department Colloquium, 9 November 2015 8 Psych Science (latest issue)

9. York Psychology Department Colloquium, 9 November 2015 9 JEP: General (latest issue)

10. York Psychology Department Colloquium, 9 November 2015 10

11. York Psychology Department Colloquium, 9 November 2015 11

12. York Psychology Department Colloquium, 9 November 2015 12

13. York Psychology Department Colloquium, 9 November 2015 13 Origins and Early Developments

14. York Psychology Department Colloquium, 9 November 2015 14 counter- attitudinal statement cognitive dissonance attitude change social psychologists, in particular, have long theorized about and drawn inferences with reference to mediators without directly observing them or accounting for them in hypothesis tests

15. York Psychology Department Colloquium, 9 November 2015 15 counter- attitudinal statement cognitive dissonance attitude change self-image threat self- perception self- inconsistency

16. York Psychology Department Colloquium, 9 November 2015 16 “When what a person does [is] attributed to what is going on inside him, investigation is brought to an end.” (Skinner, 1974)

17. York Psychology Department Colloquium, 9 November 2015 17 “The time seems to have come when psychology must discard all reference to consciousness; when it need no longer delude itself into thinking that it is making mental states the object of observation.” Watson (1913), “Psychology as the Behaviorist Views It” vs. “Who can possibly care about a psychology that is silent on such topics as thinking, motivation, and volition?” Kimble (1989), “Psychology from the Standpoint of a Generalist”

18. York Psychology Department Colloquium, 9 November 2015 18 “First psychology lost its soul, then it lost its mind, then it lost consciousness; it still has behavior, of a kind.” “In order to predict the response, we must know not only the stimulus, but also the organism stimulated” “Behavior we can observe, consciousness we can observe with some difficulty, but the inner dynamics of the mental processes must be inferred rather than observed.... A dynamic psychology must utilize the observations of consciousness and behavior as indications of the 'workings of the mind'... ” Woodworth (1918), Dynamic Psychology

19. York Psychology Department Colloquium, 9 November 2015 19 Tolman’s (1938) intervening variable framework “Some of us, psychologically, just demand theories. Even if we had all the million and one concrete facts, we would still want theories to, as we would say, “explain” those facts. Theories just seem to be necessary to some of us to relieve our inner tensions.” “A theory, as I shall conceive it, is a set of “intervening variables.” These to-be-inserted variables are “constructs” which we, the theorists, evolve....”

20. York Psychology Department Colloquium, 9 November 2015 20 “Mentalistic concepts enter psychology as inferences from behavior. The observations that define them often suggest causes.” (Kimble, 1989)

21. York Psychology Department Colloquium, 9 November 2015 21 Hyman (1955) Step 1: establish simple relationship between two variables Step 2: introduce additional variables to address problems of spuriousness Step 3: engage in process of elaboration; within general schema of elaboration interpret the relationship “When the analyst interprets a relationship, he [or she] determines the process through which the assumed cause is related to what we take to be its effect.... What are the “links” between the two variables?”

22. York Psychology Department Colloquium, 9 November 2015 22 two challenges for informative mediation analyses: 1.estimation and testing 2.inference and interpretation

23. York Psychology Department Colloquium, 9 November 2015 23 Statistical Tests

24. York Psychology Department Colloquium, 9 November 2015 24 X M Y c' a b

25. York Psychology Department Colloquium, 9 November 2015 25 X M Y c' a b X Y c difference in coefficients method c - c' > 0

26. York Psychology Department Colloquium, 9 November 2015 26 X M Y c' a b joint significance method a > 0 and b > 0

27. York Psychology Department Colloquium, 9 November 2015 27 X M Y c' a b product of coefficients method ab > 0

28. York Psychology Department Colloquium, 9 November 2015 28 issue with product of coefficients method: problem: which standard error to use to construct the test statistic; which reference distribution for tests of significance test-statistic options: Sobel approximation; normal distribution empirical distribution of ab; tabled critical values based on empirical distributions for different values of a and b distribution of product of two normal variables (z a z b ); critical values (or confidence interval) based on distribution of product of random variables

29. York Psychology Department Colloquium, 9 November 2015 29 issue with product of coefficients method: problem: which standard error to use to construct the test statistic; which reference distribution to test for significance resampling/simulation options: percentile bootstrap confidence interval bias corrected bootstrap Monte Carlo simulation

30. York Psychology Department Colloquium, 9 November 2015 30 issue with product of coefficients method: problem: which standard error to use to construct the test statistic; which reference distribution to test for significance solutions: test statistic: best option for power and Type 1 error is empirical distribution of ab resampling: percentile bootstrap if Type 1 error is primary consideration; bias corrected bootstrap if power is primary consideration

31. York Psychology Department Colloquium, 9 November 2015 31 Inference and Interpretation

32. York Psychology Department Colloquium, 9 November 2015 32 “If the mediational model is wrong (i.e., misspecified), the results from a mediational analysis are not so much meaningless, but rather they are misleading” (Kenny, 2008) Inferences about mediation are causal inferences. 1.The cause (X) precedes the mediator (M) in time, and M precedes the effect (Y) in time (i.e., temporal precedence). 2.In each case, the cause and effect are related to one another (i.e., covariation). 3.There are no rival explanations of the observed relations between the causes and effects (i.e., absence of confounds).

33. York Psychology Department Colloquium, 9 November 2015 33 evaluation of approaches to explicit testing of mediation hypotheses: findings are... misleading uninformative informative definitive current desired

34. York Psychology Department Colloquium, 9 November 2015 34 X M Y c' a b

35. York Psychology Department Colloquium, 9 November 2015 35 necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if experimental method is feasible randomize to levels of a manipulated X measure or observe Y at a process-informed interval following X

36. York Psychology Department Colloquium, 9 November 2015 36 necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if randomization to levels of a manipulated X isn’t feasible or successful use statistical means to isolate X by including covariates in the model

37. York Psychology Department Colloquium, 9 November 2015 37 X M Y c' a b CiCi C1C1......

38. York Psychology Department Colloquium, 9 November 2015 38 necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if X is not manipulated longitudinal design to address temporal order inclusion of covariates to achieve some measure of isolation of X

39. York Psychology Department Colloquium, 9 November 2015 39 the problem of equivalent models in data from nonexperimental studies X Y X Y X Y X Y C 1. 2. 3. 4.

40. York Psychology Department Colloquium, 9 November 2015 40 X M Y c' a b

41. York Psychology Department Colloquium, 9 November 2015 41 necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if experimental method is feasible randomize to levels of a manipulated X measure or observe M at a process-informed interval following X

42. York Psychology Department Colloquium, 9 November 2015 42 necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if manipulation is not possible longitudinal design to address temporal order inclusion of covariates to achieve some measure of isolation of X

43. York Psychology Department Colloquium, 9 November 2015 43 X M Y c' a b CiCi C1C1......

44. York Psychology Department Colloquium, 9 November 2015 44 X M Y c' a b

45. York Psychology Department Colloquium, 9 November 2015 45 necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions manipulation of M is not possible longitudinal design to address temporal order inclusion of covariates to achieve some measure of isolation of M

46. York Psychology Department Colloquium, 9 November 2015 46 X M Y c' a b CiCi C1C1......

47. York Psychology Department Colloquium, 9 November 2015 47 critical issue for informative/definitive M-Y inference—and X-M inference in nonexperimental data—is temporal order inability to draw firm directional inferences leaves open the possibility that other equivalent models provide an equally compelling account of the data equivalent models: two or more models that, because they offer statistically equivalent accounts of the data, cannot be distinguished on statistical grounds

48. York Psychology Department Colloquium, 9 November 2015 48

49. York Psychology Department Colloquium, 9 November 2015 49 X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12..23.36.26.11.42.30.10.39.34.23.42.26.23.36.26.23.36.26.12.39.30.10.39.34.10.39.34.12.30.39.11.42.30.11.42.30

50. York Psychology Department Colloquium, 9 November 2015 50 “... that fact that a study’s results are consistent with an assumed mediation model does not allow for any valid conclusion about the correctness of the model, unless the results stem from a study that allows for ruling out rival models.” (Stone-Romero & Raposa, 2008)

51. York Psychology Department Colloquium, 9 November 2015 51 X1X1 a b M2M2 Y3Y3

52. York Psychology Department Colloquium, 9 November 2015 52 X1X1 a b M1M1 M2M2 Y2Y2 Y3Y3 b

53. York Psychology Department Colloquium, 9 November 2015 53 X1X1 a b M1M1 Y1Y1 X2X2 M2M2 Y2Y2 X3X3 M3M3 Y3Y3 b a

54. York Psychology Department Colloquium, 9 November 2015 54 X1X1 a b M1M1 Y1Y1 X2X2 M2M2 Y2Y2 X3X3 M3M3 Y3Y3 a b

55. York Psychology Department Colloquium, 9 November 2015 55 X1X1 a b M1M1 Y1Y1 X2X2 M2M2 Y2Y2 X3X3 M3M3 Y3Y3 X-MM-Y a b Covariates 1 Covariates 2

56. York Psychology Department Colloquium, 9 November 2015 56 X a b M2M2 Y2Y2 M3M3 Y3Y3 X-MM-Y Covariates 1 Covariates 2

57. York Psychology Department Colloquium, 9 November 2015 57 issues with inferences about the extent of mediation: problem: under certain conditions the simple direct effect (c) might not be significant when the indirect effect (ab) is significant solution: focus testing and inference on the indirect effect in the full model; avoid extent of mediation inference

58. York Psychology Department Colloquium, 9 November 2015 58 issues with inferences about the extent of mediation: problem: c' may be nonsignificant but mediation not full solution: under certain conditions, proportion of full effect that is attributable to the indirect effect can be informative standardized c (r xy in simple mediation model) should be at least |.20| proportion of effect that is through the mediator or this value is descriptive (i.e., not subject to formal inference)

59. York Psychology Department Colloquium, 9 November 2015 59 issues with inferences about the extent of mediation: problem: when there is measurement error in the mediator, c' might be significant despite full mediation solutions: (1) use highly reliable measure of the mediator; (2) obtain multiple measures of the mediator and model it as a latent variable

60. York Psychology Department Colloquium, 9 November 2015 60 proposals for tests of mediation going forward: they should not be done unless the results will be informative (i.e., criterion is not statistical significance) they should not be required for circumspect, theory- driven inferences about causal mechanisms when they are done, they should be done better! randomization and manipulation when possible careful and thorough accounting for confounds consideration for time passage between assessment of variables in the causal sequence use of longitudinal designs appropriate for observing the focal causal sequence

61. York Psychology Department Colloquium, 9 November 2015 61 Pek, J., & Hoyle, R. H. (in press). On the (in)validity of tests of simple mediation: Threats and solutions. Social and Personality Psychology Compass. Thank you! rhoyle@duke.edu