Optimal Dynamic Treatment Regimes Julia Wang, Kwan Lee, and Xiwu Lin RWE Analytics at Janssen Quantitative Sciences Sept. 7th, 2017

What is Optimal DTR? Dynamic Treatment Regimes (DTR) are individually tailored treatment sequences based on subject-level information. A DTR algorithm provides a mapping from the space of subject-level information (including baseline, treatments, and responses) to the treatment action space. Optimal DTRs maximize overall average treatment effects for a population following such regimes.

bmiData from r Package iqLearn > head(bmiData) gender race parent_BMI baseline_BMI month4_BMI month12_BMI A1 A2 1 0 1 31.59683 35.84005 34.22717 34.27263 CD MR 2 1 0 30.17564 37.30396 36.38014 36.38401 CD MR 3 1 0 30.27918 36.83889 34.42168 34.41447 MR CD 4 1 0 27.49256 36.70679 32.52011 32.52397 CD CD 5 1 1 26.42350 34.84207 33.72922 33.73546 CD CD 6 0 0 29.30970 36.68640 32.06622 32.15977 MR MR

Baseline Characteristics

Baseline Characteristics Observations On-study Baseline Characteristics

Baseline Characteristics Observations On-study Baseline Characteristics Randomized Groups

Two-Stage Randomized Treatment Set-up MR or CD Stage 2: MR or CD Baseline

Subject-level information at Stage 1: Gender, Race, Baseline BMI, and Parents BMI Subject-level information at Stage 2: Gender, Race, Baseline BMI, Parents BMI and Month 4 BMI Stage 1 Baseline BMI vs. Parent's BMI Stage 2 Month 4 BMI vs. Parent's BMI 50 50 45 45 Baseline BMI Month 4 BMI 40 40 35 35 30 30 25 25 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Randomized Regime Stage 1 Baseline BMI vs. Parent's BMI Stage 2 Month 4 BMI vs. Parent's BMI MR CD 50 50 45 45 Baseline BMI Month 4 BMI 40 40 35 35 30 30 MR CD 25 25 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

A partition? Optimized Regime Stage 1 Baseline BMI vs. Parent's BMI Month 4 BMI vs. Parent's BMI MR CD 50 50 45 45 Baseline BMI Month 4 BMI 40 40 35 35 30 30 MR CD 25 25 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Data Generation Process Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space Data Generation Process Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD Y Baseline Characteristics This process can be either randomized or observational, how do we estimate the optimal DTR from the data obtained?

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 )))

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 ))) Maximize by partition the subject-level space defined by S1, S2, S3

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 ))) M-EY Maximize by partition the subject-level space defined by S1, S2, S3

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 ))) M-EY M-EY Maximize by partition the subject-level space defined by S1, S2, S3

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 ))) M-EY

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 ))) M-EY Maximize by partition the subject-level space defined by S1 and S2

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 ))) MM-EY M-EY Maximize by partition the subject-level space defined by S1 and S2

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 ))) MM-EY MM-EY M-EY Maximize by partition the subject-level space defined by S1, and S2

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 ))) MM-EY MM-EY M-EY

A Conditional Expectation Property The Goal: Maximize E ( Y ) E ( Y ) = E ( E ( Y | S1 )) = E ( E ( E ( Y | S1 , S2 ))) = E ( E ( E ( E ( Y | S1 , S2, S3 ))) MM-EY MM-EY M-EY Maximize by partition the subject-level space defined by S1

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD Y Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD Y Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD Y Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD Y Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD M-EY Y Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD M-EY Y Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD M-EY Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD M-EY Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD M-EY Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD MM-EY M-EY Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD MM-EY Baseline Characteristics

Estimation of DTR as a Recursive Partitioning of the Subject-level Information Space History Stage 1 TRT OBS MR CD Stage 2 TRT OBS MR CD Stage 3 TRT OBS MR CD Stage 4 TRT OBS MR CD MMM-EY Baseline Characteristics We need to be able to link the outcome Y with the history space. One such method of optimal DTR estimation is Q-learning

Q-learning Recursive Linear Model Fitting, with treatment by covariates interaction

Q-learning Recursive Linear Model Fitting, with treatment by covariates interaction

Q-learning Recursive Linear Model Fitting, with treatment by covariates interaction MMM-EY MM-EY M-EY

r DTR Packages iqLearn DynTxRegime DTRreg DTR DTRlearn

Illustration 1 using bmiData and Q-learning Stage 1 only and Parent’s BMI only

Month 4 and Month 12 Outcome Variable y = -100*(Month12_BMI – Baseline_BMI)/Baseline_BMI y4 = -100*(Month14_BMI – Baseline_BMI)/Baseline_BMI

Weight Reduction Outcome at Month 4 vs. Parental BMI MR CD 20 20 Negative Percent Change in BMI at 4 Month Negative Percent Change in BMI at 4 Month 10 10 −10 −10 −20 −20 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Weight Reduction Outcome at Month 4 vs. Parental BMI MR CD 20 20 Negative Percent Change in BMI at 4 Month Negative Percent Change in BMI at 4 Month 10 10 −10 −10 −20 −20 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Treatment by Parent’s BMI interaction Expected Weight Reduction Outcome at Month 4 vs. Parental BMI Treatment by Parent’s BMI interaction 20 Negative Percent Change in BMI at 4 Month ( 30.95 , 7.96 ) ● 10 −10 MR CD −20 25 30 35 40 45 50 Parent's BMI

●● Expected Weight Reduction Outcome at Month 4 vs. Parental BMI 20 Negative Percent Change in BMI at 4 Month ●●● ●● ● ●●● ●●●●●● ( 30.95 , 7.96 ) 10 ●●●●●● ● ●● ●●●●● ● ● ● ●● ● ●● ●●● ●●●●●● ●● ●●●●●●●●●●●●●●●● ●● ●●●● ●●●● ●● ● ● ●●●●●●●●●● ●● ●● ●●● ●●●●●●●● ●●●● ● ● ● ● ●●●●●● ●●●● ●●● ● ● ● ● ●● ●●● ●● ● ● ● ●●● ● ●● ●●●● ● ● −10 ●● ● MR CD −20 25 30 35 40 45 50 Parent's BMI

●●● Expected Weight Reduction Outcome at Month 4 vs. Parental BMI ● MR CD 20 20 Negative Percent Change in BMI at 4 Month ●● ● Negative Percent Change in BMI at 4 Month ● ● ●● ●●● ●●● 10 ● ●●● 10 ● ●●● ●● ●●● ● ● ● ● ●●●●●●●●●●●● ●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●● ●●●●●● ●●●●●● ●● ●● ● ●● ●● ● ● ●● ● ●● ● ● ●● ●● ● ●● ●● ●● ● ● −10 −10 ● −20 −20 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

●● Expected Weight Reduction Outcome at Month 4 vs. Parental BMI ● MR CD 20 20 ●● Negative Percent Change in BMI at 4 Month ●● ● Negative Percent Change in BMI at 4 Month ● ● ● ●● ● ●● ●● ●●● ● ● ●●● ●● 10 ● ●●● 10 ●● ●● ● ●● ●●● ●● ●● ● ●● ●● ● ●● ● ●●●●● ● ● ●●●● ●●●●●●●●●●●●● ●● ●●●●●● ●● ●●●●●●● ●●●● ● ● ● ● ●●●●●● ●●●●● ● ● ● ● ● ●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●● ●● ● ●● ●● ● ● ●● ● ●● ● ● ●● ●● ● ●● ●● ●● ● ● −10 −10 ● −20 −20 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Optimal Expected Weight Reduction Outcome at Month 4 vs. Parental BMI MR CD 20 20 ●● Negative Percent Change in BMI at 4 Month ●● ● Negative Percent Change in BMI at 4 Month ● ● ● ●● ● ●● ●● ●●● ● ● ●●● ●● 10 ● ●●● 10 ●● ●● ● ●● ●●● ●● ● ●● ●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●● ●● ● ● ● −10 −10 −20 −20 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Optimal Expected Weight Reduction Outcome at Month 4 vs. Parental BMI MR CD 20 20 ●● Negative Percent Change in BMI at 4 Month ●● ● Negative Percent Change in BMI at 4 Month ● ● ● ●● ● ●● ●● ●●● ● ● ●●● ●● 10 ● ●●● 10 ●● ●● ● ●● ●●● ●● ● ●● ●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●● ●● ● ● ● −10 −10 −20 −20 25 35 30 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

A Classification Problem Expected Weight Reduction Outcome at Month 4 vs. Parental BMI MR CD Loss 20 20 ●● Negative Percent Change in BMI at 4 Month ●● ● Negative Percent Change in BMI at 4 Month ● ● ● ●● ● ●● ●● ●●● ● ● ●●● ●● 10 ● ●●● 10 ●● ●● ● ●● ●●● ●● ●● ● ●● ●●●●●● ●● ●●●●●● ●● ●●●●●●● ●●●● ● ● ●● ● ●●●●● ●● ● ● ● ●●●● ●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ● ●●●●●● ●●●●● ● ● ● ● ● ● ●● ●● ● ●● ●● ● ● ●● ● ●● ● ● ●● ●● ● ●● ●● ●● ● ● −10 −10 ● A Classification Problem −20 −20 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Illustration 2 using bmiData Stage 1 only and Parent’s BMI and Baseline BMI

Model Fitted MR (A1=1) and CD (A1=-1)

Predicted Stage 1 Optimal Treatment based on Month 4 Outcome MR Group Demarcation Line MR better CD better 50 45 Baseline BMI 40 35 30 25 30 35 40 45 50 Parent's BMI

Predicted Stage 1 Optimal Treatment based on Month 4 Outcome CD Group Demarcation Line MR better CD better 50 45 Baseline BMI 40 35 30 25 30 35 40 45 50 Parent's BMI

Expected Weight Reduction Outcome at Month 4 vs. Parental BMI Parent’s BMI only 20 Negative Percent Change in BMI at 4 Month ( 30.95 , 7.96 ) ● 10 −10 MR CD −20 25 30 35 40 45 50 Parent's BMI

Predicted Stage 1 Optimal Treatment based on Month 4 Outcome MR Group Demarcation Line MR better CD better 50 45 Baseline BMI 40 35 30 30.95 25 30 35 40 45 50 Parent's BMI

Predicted Stage 1 Optimal Treatment based on Month 4 Outcome MR Group Demarcation Line MR better CD better 50 45 Baseline BMI 40 35 30 30.95 25 30 35 40 45 50 Parent's BMI

●● Expected Weight Reduction Outcome at Month 4 vs. Parental BMI ● MR CD 20 20 ●● Negative Percent Change in BMI at 4 Month ●● ● Negative Percent Change in BMI at 4 Month ● ● ● ●● ● ●● ●● ●●● ● ● ●●● ●● 10 ● ●●● 10 ●● ●● ● ●● ●●● ●● ●● ● ●● ●● ● ●● ● ●●●●● ● ● ●●●● ●●●●●●●●●●●●● ●● ●●●●●● ●● ●●●●●●● ●●●● ● ● ● ● ●●●●●● ●●●●● ● ● ● ● ● ●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●● ●● ● ●● ●● ● ● ●● ● ●● ● ● ●● ●● ● ●● ●● ●● ● ● −10 −10 ● −20 −20 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Optimal vs. Predicted Month 4 Outcome MR CD 18 20 Optimal Predicted Month 4 Outcome Optimal Predicted Month 4 Outcome 16 10 14 12 10 −10 8 −20 6 −20 −10 10 20 6 8 10 12 14 16 18 Predicted Month 4 Outcome Predicted Month 4 Outcome

Illustration 3 using bmiData Both Stages Stage 1: Parents BMI and Baseline BMI Stage 2: Parents BMI and Month 4 BMI

Models Fitted Stage 2: y ~ gender + parent_BMI + month4_BMI + A2*(parent_BMI + month4_BMI) A2 coded as 1 and -1 Stage 1: Opt_y~ gender + race + parent_BMI + baseline_BMI + A1*(parent_BMI + baseline_BMI) A1 coded as 1 and -1

Predicted Stage 2 Optimal Treatment based on Month 12 Outcome MR Group Demarcation Line MR better CD better 45 40 Month 4 BMI 35 30 25 30 35 40 45 50 Parent's BMI

Predicted Stage 2 Optimal Treatment based on Month 12 Outcome CD Group Demarcation Line MR better CD better 45 40 Month 4 BMI 35 30 25 30 35 40 45 50 Parent's BMI

Predicted Stage 1 Optimal Treatment based on Optimal Month 12 Outcome MR Group Demarcation Line MR better CD better 50 45 Baseline BMI 40 35 30 25 30 35 40 45 50 Parent's BMI

Predicted Stage 1 Optimal Treatment based on Optimal Month 12 Outcome CD Group Demarcation Line MR better CD better 50 45 Baseline BMI 40 35 30 25 30 35 40 45 50 Parent's BMI

Models Fitted Stage 2: y ~ gender + parent_BMI + month4_BMI + A2*(parent_BMI + month4_BMI) A2 coded as 1 and -1 Contrast Term Stage 1: Opt_y~ gender + race + parent_BMI + baseline_BMI + A1*(parent_BMI + baseline_BMI) A1 coded as 1 and -1

Estimated Contrast Term vs. Parent's BMI in Stage 2 MR (coded as 1) CD (coded as −1) 4 4 2 2 Estimated Contrast Term Estimated Contrast Term −2 −2 −4 −4 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Switch!!! Estimated Contrast Term vs. Parent's BMI in Stage 2 MR (coded as 1) CD (coded as −1) 4 4 2 2 Estimated Contrast Term Estimated Contrast Term Switch!!! −2 −2 −4 −4 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

Switch!!! Estimated Contrast Term vs. Parent's BMI in Stage 2 MR (coded as 1) CD (coded as −1) 4 4 2 2 Estimated Contrast Term Estimated Contrast Term Switch!!! −2 −2 −4 −4 25 30 35 40 45 50 25 30 35 40 45 50 Parent's BMI Parent's BMI

The Goal of Optima DTR maximize overall average treatment effects for a population following such regimes The outcome of a fixed treatment regime can be estimated from the population of subjects whose observe regime is the same as the optimal regime.

Estimated Mean Y 9.9284

7.9175 8.0631 3.5236 6.2016 Estimated Mean Y 9.9284

What if the data is observational? Use Inverse Probability Weighting, at each stage It creates an approximate pseudo-randomized population at each stage Why?

IPW creates an approximate pseudo-randomized population Assuming there are N subjects in the study, and the probability of being treated is p We will have (on average) N*p treated subjects. If we weight each of these subjects by 1/p, then these N*p subjects will represent N*p*(1/p)=N subjects. The same reasoning applies to the control subjects weighted by 1/(1-p), and they will represent on average N*(1-p)*(1/(1-p))=N subjects.

An IPW Illustration Assuming there are N=90m (m is an integer multiplier) subjects in the study, evenly spread within a single confounder with 9 subgroups; The probability of being treated varies from p = 0.1, 0.2, to 0.9 within each subgroup of the confounder. Within each subgroup, we will have (on average) n1=10m*p treated and n2=10m*(1-p) controls. This leads to: n1*(1/p) = n2* (1/(1-p)) = 10m Conclusion: IPW balances the distribution of the confounders. However, in practice, these probabilities, also called propensity scores, need to be estimated.

Distribution of Confonders before IPW ● ● 0.1 0.2 ● 0.3 ● 0.4 0.5 ● 0.6 ● 0.7 0.8 ● 0.9 Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Subjects Confonder Subgroups

● Distribution of Confonders for Treated Group before IPW Gr. 1 Gr. 2 Subjects Confonder Subgroups

Distribution of Confonders for Treated Group before IPW ● 1/ 0.9 ● 1/ 0.8 ● 1/ 0.7 ● 1/ 0.6 ● 1/ 0.5 ● 1/ 0.4 ● 1/ 0.3 ● 1/ 0.2 ● 1/ 0.1 Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Subjects Confonder Subgroups

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Distribution of Confonders for Treated Group before IPW ● ● ● ● ● ● ● ● ● ● Subjects ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 / 0.1 ● 2 / 0.2 ● 3 / 0.3 ● 4 / 0.4 ● 5 / 0.5 ● 6 / 0.6 ● 7 / 0.7 ● 8 / 0.8 ● 9 / 0.9 Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Confonder Subgroups

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Distribution of Confonders for Treated Group before IPW ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Subjects ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Confonder Subgroups

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Distribution of Confonders for Treated Group after IPW ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Subjects ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Confonder Subgroups

Distribution of Confonders before IPW ● ● 0.1 0.2 ● 0.3 ● 0.4 0.5 ● 0.6 ● 0.7 0.8 ● 0.9 Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Subjects Confonder Subgroups

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Distribution of Confonders for Control Group before IPW ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Subjects ● ● ● ● ● ● ● ● ● ● Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Confonder Subgroups

Distribution of Confonders for Control Group before IPW ● 1/ 0.9 ● 1/ 0.8 ● 1/ 0.7 ● 1/ 0.6 ● 1/ 0.5 ● 1/ 0.4 ● 1/ 0.3 ● 1/ 0.2 ● 1/ 0.1 ● 1/ 0.2 Subjects Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Confonder Subgroups

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Distribution of Confonders for Control Group before IPW ● 9 / 0.9 ● 8 / 0.8 ● 7 / 0.7 ● 6 / 0.6 ● 5 / 0.5 ● 4 / 0.4 ● 3 / 0.3 ● 2 / 0.2 ● 1 / 0.1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Subjects ● ● ● ● ● ● ● ● ● ● Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Confonder Subgroups

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Distribution of Confonders for Control Group before IPW ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Subjects ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 9 Confonder Subgroups

● Distribution of Confonders for Control Group after IPW Gr. 1 Gr. 2 Subjects Confonder Subgroups

● ● Distribution of Confonders after IPW Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5 Subjects Confonder Subgroups

Propensity Scores The more accurately a propensity score can be estimated, the better it is at reducing confounding and bias. How to estimate propensity scores properly is a very big topic in itself.

7.9175 8.0631 3.5236 6.2016 Estimated Mean Y 9.9284 dwols: 10.0805

Comparing Q-learning vs. dwols using bmiData Stage 2 dwols Q-learning CD MR 94 1 4 111 Stage 1 dwols Q-learning CD MR 111 1 2 96