Joint Statistical Meetings, Vancouver, August 1, 2018 Weighting-Based Sensitivity Analysis in Causal Mediation Studies: Interactive Tools for Analysts Guanglei Hong University of Chicago Xu Qin University of Pittsburgh Fan Yang University of Colorado Denver

What Is Sensitivity Analysis? A causal conclusion is valid only when the identification assumptions hold. Sensitivity Analysis: An attempt to determine whether a plausible violation of an identification assumption could qualitatively change an analytic conclusion Logical Reasoning: Analytic conclusions that are harder to alter by such a violation are expected to add a higher value to scientific knowledge

Did You Conduct a Sensitivity Analysis? Scientists always rightfully voice skepticisms toward causal claims. A study is incomplete without a sensitivity analysis. However, Preach  Practice Sensitivity analysis is rarely done (except in economics) Discussions mostly limited to the statistics community Never in guidelines for grant application or publication

WHY? Sensitivity analysis is challenging Often requires additional assumptions like a tail-chasing game that never ends…

Weighting-based sensitivity analysis The new approach is consistent with the logic of using propensity score-based weighting to reduce selection bias In a nutshell The discrepancy between a new weight that adjusts for an omitted confounder and an initial weight that omits the confounder captures the role of the confounder that contributes to the bias

A Re-evaluation of the National Welfare-to-Work Strategies (Hong, Deutsch, & Hill, 2015) Employment M Under LFA + Under control + Treatment Z Depression Y Did the treatment increase employment rate? -- YES (40%  65%) Did the treatment alleviate or aggravate maternal depression two years later? -- APPARENTLY NO Did employment mediate the treatment impact on depression? -- SEEMINGLY

Definition of the Causal Effects Definition of the Causal Effects . (Pearl, 2001; Robins & Greenland, 1992) 𝑀 𝑍 𝑌 Z Effect on M: 𝐸 𝑀 1 −𝑀 0 Z Effect on Y: 𝐸 𝑌 1 −𝑌 0 Natural Indirect Effect: 𝐸 𝑌 1,𝑀 1 −𝑌 1,𝑀 0 Natural Direct Effect: 𝐸 𝑌 1,𝑀 0 −𝑌 0,𝑀 0

Major advantage: No need to specify an outcome model! Weighting-Based Identification of the Causal Effects (Hong, 2010, 2015) 𝐸 𝑌 1,𝑀 0 =𝐸 𝑊𝑌|𝑍=1 where 𝑊= 𝑝𝑟 𝑀=𝑚|𝑍=0,𝑋=𝑥 𝑝𝑟 𝑀=𝑚|𝑍=1,𝑋=𝑥 NIE: 𝐸 𝑌 1,𝑀 1 −𝑌 1,𝑀 0 =𝐸 𝑌|𝑍=1 −𝐸 𝑊𝑌|𝑍=1 NDE: 𝐸 𝑌 1,𝑀 0 −𝑌 0,𝑀 0 =𝐸 𝑊𝑌|𝑍=1 −𝐸 𝑌|𝑍=0 Major advantage: No need to specify an outcome model!

Identification under Sequential Ignorability (Imai et al, 2010a, 2010b) X X: Observed pretreatment covariates 𝑀 𝑍 𝑌 Assignment of Z is strongly ignorable given X Assignment of M is strongly ignorable given Z=z and X (i.e., there are no omitted pretreatment and post-treatment confounders of the M-Y relationship)

Potential Sources of Omitted Confounding Three Types of Omission in causal mediation analysis: Observed pretreatment covariates and/or their higher order functions Observed posttreatment covariates Unobserved pretreatment or posttreatment covariates (Sensitivity analysis for type C omission requires scientific reasoning and empirical evidence indicating the comparability of an unobserved covariate to some of the observed ones)

Bias due to an Omitted Pretreatment Confounder P 𝑋 𝑀 𝑍 𝑌 𝑃 e.g., preference for taking care of family rather than working

𝛿 𝑁𝐼𝐸 ≡𝐸 𝑌 1,𝑀 1 −𝑌 1,𝑀 0 Weighting-based adjustment for X 𝛿 𝑁𝐼𝐸 ≡𝐸 𝑌 1,𝑀 1 −𝑌 1,𝑀 0 Weighting-based adjustment for X 𝑑 𝑁𝐼𝐸 =𝐸 𝑌|𝑍=1 −𝐸 𝑊𝑌|𝑍=1 𝑊= 𝑝𝑟 𝑀=𝑚|𝑍=0,𝑋=𝑥 𝑝𝑟 𝑀=𝑚|𝑍=1,𝑋=𝑥 To additionally remove bias associated with the omitted P 𝑑 𝑃.𝑁𝐼𝐸 =𝐸 𝑌|𝑍=1 −𝐸 𝑊 𝑃 𝑌|𝑍=1 𝑊 𝑃 = 𝑝𝑟 𝑀=𝑚|𝑍=0,𝑋=𝑥,𝑃=𝑝 𝑝𝑟 𝑀=𝑚|𝑍=1,𝑋=𝑥,𝑃=𝑝

Bias due to an omitted pretreatment confounder P 𝐵𝑖𝑎𝑠 𝑁𝐼𝐸 = 𝑑 𝑁𝐼𝐸 − 𝑑 𝑃.𝑁𝐼𝐸 =𝑐𝑜𝑣 𝑊 𝑃 −𝑊,𝑌|𝑍=1 𝐵𝑖𝑎𝑠 𝑁𝐷𝐸 = −1 × 𝐵𝑖𝑎𝑠 𝑁𝐼𝐸 𝐸𝑆 of 𝐵𝑖𝑎𝑠 𝑁𝐼𝐸 =𝜎𝜌 𝜎= 𝑣𝑎𝑟 𝑊 𝑃 −𝑊|𝑍=1 ; 𝜌=𝑐𝑜𝑟𝑟 𝑊 𝑃 −𝑊,𝑌|𝑍=1 𝜎=0 if P does not conditionally predict M; monotonic between 𝜎 and the P-M association 𝜌=0 if P does not conditionally predict Y; monotonic between 𝜌 and the P-Y association

Bias due to an omitted posttreatment confounder Q 𝑋 𝑀 𝑄 𝑍 𝑌 e.g., ever been out of welfare and out of work during the first year after randomization

𝑁𝐼𝐸≡𝐸 𝑌 1,𝑄 1 ,𝑀 1,𝑄 1 −𝑌 1,𝑄 1 ,𝑀 0,𝑄 0 𝑋 𝑀 𝑄 𝑍 𝑌

𝑁𝐷𝐸≡𝐸 𝑌 1,𝑄 1 ,𝑀 0,𝑄 0 −𝑌 0,𝑄 0 ,𝑀 0,𝑄 0 𝑋 𝑀 𝑄 𝑍 𝑌

Q and M as Consecutive Mediators Identification Assumptions Assignment of Z is strongly ignorable given X Assignment of Q is strongly ignorable given Z=z and X Assignment of M is strongly ignorable given Z=z, X, and Q 𝐸 𝑌 1,𝑄 1 , 𝑀 0,𝑄 0 =𝐸 𝑊 𝑄 𝑌|𝑍=1 where 𝑊 𝑄 = 𝑝𝑟 𝑀=𝑚|𝑍=0,𝑋=𝑥,𝑄=𝑞 𝑝𝑟 𝑀=𝑚|𝑍=1,𝑋=𝑥,𝑄=𝑞

𝐵𝑖𝑎𝑠 𝑁𝐼𝐸 = 𝑑 𝑁𝐼𝐸 − 𝑑 𝑄.𝑁𝐼𝐸 =𝑐𝑜𝑣 𝑊 𝑄 −𝑊,𝑌|𝑍=1 𝑁𝐼𝐸≡𝐸 𝑌 1,𝑄 1 ,𝑀 1,𝑄 1 −𝑌 1,𝑄 1 ,𝑀 0,𝑄 0 Weighting-based adjustment for X 𝑑 𝑁𝐼𝐸 =𝐸 𝑌|𝑍=1 −𝐸 𝑊𝑌|𝑍=1 𝑊= 𝑝𝑟 𝑀=𝑚|𝑍=0,𝑋=𝑥 𝑝𝑟 𝑀=𝑚|𝑍=1,𝑋=𝑥 To additionally remove bias associated with Q 𝑑 𝑄.𝑁𝐼𝐸 =𝐸 𝑌|𝑍=1 −𝐸 𝑊 𝑄 𝑌|𝑍=1 𝑊 𝑄 = 𝑝𝑟 𝑀=𝑚|𝑍=0,𝑋=𝑥,𝑄=𝑞 𝑝𝑟 𝑀=𝑚|𝑍=1,𝑋=𝑥,𝑄=𝑞 𝐵𝑖𝑎𝑠 𝑁𝐼𝐸 = 𝑑 𝑁𝐼𝐸 − 𝑑 𝑄.𝑁𝐼𝐸 =𝑐𝑜𝑣 𝑊 𝑄 −𝑊,𝑌|𝑍=1 𝐵𝑖𝑎𝑠 𝑁𝐷𝐸 = −1 × 𝐵𝑖𝑎𝑠 𝑁𝐼𝐸

Two Sensitivity Parameters Use 𝑊 # as a general form of the new weight, 𝑊 as the initial weight Effect Size of Bias in Identifying NIE: 𝜎𝜌 Effect Size of Bias in Identifying NDE: −𝜎𝜌 𝜎= 𝑣𝑎𝑟 𝑊 # −𝑊|𝑍=1 (reflecting the degree to which the omitted confounder is associated with M given Z and X) 𝜌=𝑐𝑜𝑟𝑟 𝑊 # −𝑊,𝑌|𝑍=1 (reflecting the degree to which the omitted confounder predicts Y in the experimental group given X)

Weighting-Based Sensitivity Analysis Procedure Obtain the initial weight Obtain a new weight If type A or type B omission, then include the omitted observed covariates or their high-order forms If type C omission, then exclude a comparable observed covariate Compute the values of the sensitivity parameters 𝜎 and 𝜌 each as a function of the weight discrepancy Compute the effect size of bias 𝜎𝜌 and −𝜎𝜌 Compute the adjusted point estimate and interval estimate of the effect size of NIE and NDE

Sensitivity Analysis for NIE Initial estimate of effect size of NIE: -0.111; 95% CI: (-0.227, 0.006) Reference values from X1, X2, X3, X1X2, X1X3, X2X3, X1X2X3, X4 (posttreatment)

Sensitivity Analysis for NDE Initial estimate of effect size of NDE: 0.158; 95% CI: (-0.054, 0.371)

Relative Advantages of Weighting-Based Sensitivity Analysis Necessary and coherent for weighting-based causal investigations No additional simplifying assumptions (nearly a free lunch!) No increase in sensitivity parameters despite changes in data generation functions Unconstrained by measurement scales of M, Y, P, and Q Easy to assess the aggregate bias due to omitting multiple confounders For assessing nonresponse bias in addition to mediation selection bias Easy to combine the bias from different sources Easy to assess potential bias associated with posttreatment as well as pretreatment covariates

Extensions to ATT, ATE, Multisite studies… Hong, G., Qin, X., & Yang, F. (2018). Weighting-based sensitivity analysis in causal mediation studies. Journal of Educational and Behavioral Statistics, 43(1), 32-56. “rmpw” package in R: https://cran.r-project.org/web/packages/rmpw/index.html. Make the tool “interactive” with users for assisting with analytic decision-making by asking for: Omitted pretreatment confounders Omitted posttreatment confounders Observed confounders comparable to unobserved ones in terms of plausible values of 𝜎 and 𝜌 Extensions to ATT, ATE, Multisite studies…