The State of Mediation Analysis in Psychological Science Issues and Solutions Rick Hoyle Duke University

York Psychology Department Colloquium, 9 November putative cause explanatory mechanism (behavioral) outcome

York Psychology Department Colloquium, 9 November review of articles in psychology reporting mediation analysis: N = 291 distribution across areas of psychology 34% social 24% clinical 10% health remainder in developmental I/O cognitive methods program evaluation

York Psychology Department Colloquium, 9 November

York Psychology Department Colloquium, 9 November review of social psychology articles published from : at least one mediation test 59% of articles in JPSP 65% of articles in PSPB

York Psychology Department Colloquium, 9 November JPSP (latest issue)

York Psychology Department Colloquium, 9 November JCCP (latest issue)

York Psychology Department Colloquium, 9 November Psych Science (latest issue)

York Psychology Department Colloquium, 9 November JEP: General (latest issue)

York Psychology Department Colloquium, 9 November

York Psychology Department Colloquium, 9 November

York Psychology Department Colloquium, 9 November

York Psychology Department Colloquium, 9 November Origins and Early Developments

York Psychology Department Colloquium, 9 November counter- attitudinal statement cognitive dissonance attitude change social psychologists, in particular, have long theorized about and drawn inferences with reference to mediators without directly observing them or accounting for them in hypothesis tests

York Psychology Department Colloquium, 9 November counter- attitudinal statement cognitive dissonance attitude change self-image threat self- perception self- inconsistency

York Psychology Department Colloquium, 9 November “When what a person does [is] attributed to what is going on inside him, investigation is brought to an end.” (Skinner, 1974)

York Psychology Department Colloquium, 9 November “The time seems to have come when psychology must discard all reference to consciousness; when it need no longer delude itself into thinking that it is making mental states the object of observation.” Watson (1913), “Psychology as the Behaviorist Views It” vs. “Who can possibly care about a psychology that is silent on such topics as thinking, motivation, and volition?” Kimble (1989), “Psychology from the Standpoint of a Generalist”

York Psychology Department Colloquium, 9 November “First psychology lost its soul, then it lost its mind, then it lost consciousness; it still has behavior, of a kind.” “In order to predict the response, we must know not only the stimulus, but also the organism stimulated” “Behavior we can observe, consciousness we can observe with some difficulty, but the inner dynamics of the mental processes must be inferred rather than observed.... A dynamic psychology must utilize the observations of consciousness and behavior as indications of the 'workings of the mind'... ” Woodworth (1918), Dynamic Psychology

York Psychology Department Colloquium, 9 November Tolman’s (1938) intervening variable framework “Some of us, psychologically, just demand theories. Even if we had all the million and one concrete facts, we would still want theories to, as we would say, “explain” those facts. Theories just seem to be necessary to some of us to relieve our inner tensions.” “A theory, as I shall conceive it, is a set of “intervening variables.” These to-be-inserted variables are “constructs” which we, the theorists, evolve....”

York Psychology Department Colloquium, 9 November “Mentalistic concepts enter psychology as inferences from behavior. The observations that define them often suggest causes.” (Kimble, 1989)

York Psychology Department Colloquium, 9 November Hyman (1955) Step 1: establish simple relationship between two variables Step 2: introduce additional variables to address problems of spuriousness Step 3: engage in process of elaboration; within general schema of elaboration interpret the relationship “When the analyst interprets a relationship, he [or she] determines the process through which the assumed cause is related to what we take to be its effect.... What are the “links” between the two variables?”

York Psychology Department Colloquium, 9 November two challenges for informative mediation analyses: 1.estimation and testing 2.inference and interpretation

York Psychology Department Colloquium, 9 November Statistical Tests

York Psychology Department Colloquium, 9 November X M Y c' a b

York Psychology Department Colloquium, 9 November X M Y c' a b X Y c difference in coefficients method c - c' > 0

York Psychology Department Colloquium, 9 November X M Y c' a b joint significance method a > 0 and b > 0

York Psychology Department Colloquium, 9 November X M Y c' a b product of coefficients method ab > 0

York Psychology Department Colloquium, 9 November issue with product of coefficients method: problem: which standard error to use to construct the test statistic; which reference distribution for tests of significance test-statistic options: Sobel approximation; normal distribution empirical distribution of ab; tabled critical values based on empirical distributions for different values of a and b distribution of product of two normal variables (z a z b ); critical values (or confidence interval) based on distribution of product of random variables

York Psychology Department Colloquium, 9 November issue with product of coefficients method: problem: which standard error to use to construct the test statistic; which reference distribution to test for significance resampling/simulation options: percentile bootstrap confidence interval bias corrected bootstrap Monte Carlo simulation

York Psychology Department Colloquium, 9 November issue with product of coefficients method: problem: which standard error to use to construct the test statistic; which reference distribution to test for significance solutions: test statistic: best option for power and Type 1 error is empirical distribution of ab resampling: percentile bootstrap if Type 1 error is primary consideration; bias corrected bootstrap if power is primary consideration

York Psychology Department Colloquium, 9 November Inference and Interpretation

York Psychology Department Colloquium, 9 November “If the mediational model is wrong (i.e., misspecified), the results from a mediational analysis are not so much meaningless, but rather they are misleading” (Kenny, 2008) Inferences about mediation are causal inferences. 1.The cause (X) precedes the mediator (M) in time, and M precedes the effect (Y) in time (i.e., temporal precedence). 2.In each case, the cause and effect are related to one another (i.e., covariation). 3.There are no rival explanations of the observed relations between the causes and effects (i.e., absence of confounds).

York Psychology Department Colloquium, 9 November evaluation of approaches to explicit testing of mediation hypotheses: findings are... misleading uninformative informative definitive current desired

York Psychology Department Colloquium, 9 November X M Y c' a b

York Psychology Department Colloquium, 9 November necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if experimental method is feasible randomize to levels of a manipulated X measure or observe Y at a process-informed interval following X

York Psychology Department Colloquium, 9 November necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if randomization to levels of a manipulated X isn’t feasible or successful use statistical means to isolate X by including covariates in the model

York Psychology Department Colloquium, 9 November X M Y c' a b CiCi C1C

York Psychology Department Colloquium, 9 November necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if X is not manipulated longitudinal design to address temporal order inclusion of covariates to achieve some measure of isolation of X

York Psychology Department Colloquium, 9 November the problem of equivalent models in data from nonexperimental studies X Y X Y X Y X Y C

York Psychology Department Colloquium, 9 November X M Y c' a b

York Psychology Department Colloquium, 9 November necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if experimental method is feasible randomize to levels of a manipulated X measure or observe M at a process-informed interval following X

York Psychology Department Colloquium, 9 November necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions if manipulation is not possible longitudinal design to address temporal order inclusion of covariates to achieve some measure of isolation of X

York Psychology Department Colloquium, 9 November X M Y c' a b CiCi C1C

York Psychology Department Colloquium, 9 November X M Y c' a b

York Psychology Department Colloquium, 9 November necessary for informative/definitive findings: temporal precedence covariation absence of confounds strategies and solutions manipulation of M is not possible longitudinal design to address temporal order inclusion of covariates to achieve some measure of isolation of M

York Psychology Department Colloquium, 9 November X M Y c' a b CiCi C1C

York Psychology Department Colloquium, 9 November critical issue for informative/definitive M-Y inference—and X-M inference in nonexperimental data—is temporal order inability to draw firm directional inferences leaves open the possibility that other equivalent models provide an equally compelling account of the data equivalent models: two or more models that, because they offer statistically equivalent accounts of the data, cannot be distinguished on statistical grounds

York Psychology Department Colloquium, 9 November

York Psychology Department Colloquium, 9 November X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M X Y M

York Psychology Department Colloquium, 9 November “... that fact that a study’s results are consistent with an assumed mediation model does not allow for any valid conclusion about the correctness of the model, unless the results stem from a study that allows for ruling out rival models.” (Stone-Romero & Raposa, 2008)

York Psychology Department Colloquium, 9 November X1X1 a b M2M2 Y3Y3

York Psychology Department Colloquium, 9 November X1X1 a b M1M1 M2M2 Y2Y2 Y3Y3 b

York Psychology Department Colloquium, 9 November X1X1 a b M1M1 Y1Y1 X2X2 M2M2 Y2Y2 X3X3 M3M3 Y3Y3 b a

York Psychology Department Colloquium, 9 November X1X1 a b M1M1 Y1Y1 X2X2 M2M2 Y2Y2 X3X3 M3M3 Y3Y3 a b

York Psychology Department Colloquium, 9 November X1X1 a b M1M1 Y1Y1 X2X2 M2M2 Y2Y2 X3X3 M3M3 Y3Y3 X-MM-Y a b Covariates 1 Covariates 2

York Psychology Department Colloquium, 9 November X a b M2M2 Y2Y2 M3M3 Y3Y3 X-MM-Y Covariates 1 Covariates 2

York Psychology Department Colloquium, 9 November issues with inferences about the extent of mediation: problem: under certain conditions the simple direct effect (c) might not be significant when the indirect effect (ab) is significant solution: focus testing and inference on the indirect effect in the full model; avoid extent of mediation inference

York Psychology Department Colloquium, 9 November issues with inferences about the extent of mediation: problem: c' may be nonsignificant but mediation not full solution: under certain conditions, proportion of full effect that is attributable to the indirect effect can be informative standardized c (r xy in simple mediation model) should be at least |.20| proportion of effect that is through the mediator or this value is descriptive (i.e., not subject to formal inference)

York Psychology Department Colloquium, 9 November issues with inferences about the extent of mediation: problem: when there is measurement error in the mediator, c' might be significant despite full mediation solutions: (1) use highly reliable measure of the mediator; (2) obtain multiple measures of the mediator and model it as a latent variable

York Psychology Department Colloquium, 9 November proposals for tests of mediation going forward: they should not be done unless the results will be informative (i.e., criterion is not statistical significance) they should not be required for circumspect, theory- driven inferences about causal mechanisms when they are done, they should be done better! randomization and manipulation when possible careful and thorough accounting for confounds consideration for time passage between assessment of variables in the causal sequence use of longitudinal designs appropriate for observing the focal causal sequence

York Psychology Department Colloquium, 9 November Pek, J., & Hoyle, R. H. (in press). On the (in)validity of tests of simple mediation: Threats and solutions. Social and Personality Psychology Compass. Thank you!