Secondary Aims Using Data Arising from a SMART Module 6—Day 2 Getting SMART About Developing Individualized Adaptive Health Interventions Methods Work, Chicago, Illinois, June Daniel Almirall & Susan A. Murphy

Secondary Aims, Outline Discuss what is a “more deeply-tailored ATS” Review of auxiliary data typical of SMARTs Q-Learning: Using backward-induction logic – S(a): Analysis of time-varying tailoring variables of second-stage treatment There is one step to this: Step (i) – S(b): Analysis of baseline tailoring variables of first- stage (assuming an optimal second-stage treatment) There are two steps to this: Steps (ii) and (iii)

What is a more deeply-tailored ATS? To understand this, we first review what are the “embedded ATSs” within the ADHD SMART study Recall there are 4 SMART-design embedded ATSs.

Recall the Prototypical SMART Design: ADHD SMART Example Continue Medication Responders Medication Increase Medication Dose Add Behavioral Intervention R Continue Behavioral Intervention Behavioral Intervention Increase Behavioral Intervention Add Medication Non-Responders R Responders Non-Responders R

Continue Medication Responders Medication Increase Medication Dose Add Behavioral Intervention R Continue Behavioral Intervention Behavioral Intervention Increase Behavioral Intervention Add Medication Non-Responders R Responders Non-Responders R These are the 2 embedded ATS that employ the “add other treatment” tactic

Continue Medication Responders Medication Increase Medication Dose Add Behavioral Intervention R Continue Behavioral Intervention Behavioral Intervention Increase Behavioral Intervention Add Medication Non-Responders R Responders Non-Responders R These are the 2 embedded ATSs that employ the “intensify initial treatment” tactic

So why consider a more deeply- tailored ATS? First, it may be that some participants (e.g., those who have used MED in the past) may benefit more from MED vs BMOD. Second, certain types of non-responders (e.g., those who do not adhere to initial treatment) may benefit more from INTENSIFY vs ADD.

What is a more deeply-tailored ATS? It may look like this… If the child used MED in prior year, then begin with MED; otherwise, begin with BMOD. If the child is non-responsive and non-adherent to either first-line treatment, then AUGMENT with the other treatment option. If the child is non-responsive but adherent to either first-line treatment, then it is better to INTENSIFY first-line treatment. If the child is responsive to first-line treatment, then CONTINUE first-line treatment.

Secondary Aims, Outline Discuss what is a “more deeply-tailored ATS” Review of auxiliary data typical of SMARTs Q-Learning: Using backward-induction logic – S(a): Analysis of time-varying tailoring variables of second-stage treatment There is one step to this: Step (i) – S(b): Analysis of baseline tailoring variables of first- stage (assuming an optimal second-stage treatment) There are two steps to this: Steps (ii) and (iii)

Other Measures Collected in a SMART Continue Medication Responders Medication Increase Medication Dose Add Behavioral Intervention R Continue Behavioral Intervention Behavioral Intervention Increase Behavioral Intervention Add Medication Non-Responders R Responders Non-Responders R O1 = Demog., Pre-txt Medication Hx, Pre-txt ADHD scores, Pre-txt school performance, ODD Dx, … O2 = Month of non-response, adherence to first-stage txt, … O1A1O2 / R StatusA2Y

How are O1 and O2 used? We can use the auxiliary data O1 to help decide which is best MED v BMOD for certain individuals We can use the auxiliary data O1 and O2 to help decide which is best INTENSIFY v ADD for certain individuals

Secondary Aims, Outline Discuss what is a “more deeply-tailored ATS” Review of auxiliary data typical of SMARTs Q-Learning: Using backward-induction logic – S(a): Analysis of time-varying tailoring variables of second-stage treatment There is one step to this: Step (i) – S(b): Analysis of baseline tailoring variables of first- stage (assuming an optimal second-stage treatment) There are two steps to this: Steps (ii) and (iii)

Using Q-Learning to develop a more richly-individualized ATS Q-Learning is an extension of regression to sequential treatments. Q-Learning results in a proposal for an adaptive treatment strategy with greater individualization. A subsequent trial would evaluate the proposed adaptive treatment strategy versus usual care.

Three Steps in Q-Learning Regression Work backwards (as you would for a project time-line) i.Do a regression to learn how to more deeply- tailor second-stage treatment using O1 and O2 (in ADHD SMART, this is only with non-responders) ii.Assign each non-responder the value Ŷ i, an estimate of the outcome under the second-line treatment that yields best outcome for that person i. Responders get their observed Y i. iii.Using Ŷ i do a regression to learn how to more deeply-tailor first-stage treatment using O1

Continue Medication Responders Medication Increase Medication Dose Add Behavioral Intervention R Continue Behavioral Intervention Behavioral Intervention Increase Behavioral Intervention Add Medication Non-Responders R Responders Non-Responders R Step (i) of Q-Learning: Second-stage treatment tailoring? Step (i)a. Step (i)b. Adherence to initial MED Adherence to initial BMOD O1A1O2 / R StatusA2Y

Step (i) of Q-Learning: Second-stage treatment tailoring? To accomplish Step (i) of Q-Learning, you may fit a regression model such as this: Y = b1 + b2 o11c + b3 o12c + b4 o13c + b5 o14c + b6 o21c + b7 a1 + b8 o22 + b9 a2*a1 + b10 a2*o22 + b11 a2 + error From such a model we would learn if, for example, o22 (adherence to first stage txt) is a strong moderator of the effect of a2.

Step (i) of Q-Learning: Learn best second- stage treatment for non-responders ≈ -1.3 to -2.1 Among non-adherers to either first- stage treatment, better to add txt. This analysis is with simulated data. INT –ADD

Step (i) of Q-Learning: Learn best second- stage treatment for non-responders ≈ Among adherers to MED, better to intensify MED. This analysis is with simulated data. INT –ADD

Step (i) of Q-Learning: Learn best second- stage treatment for non-responders ≈ Among adherers to BMOD, better to intensify but not by much (NS). This analysis is with simulated data. INT –ADD

Three Steps in Q-Learning Regression Work backwards (as you would for a project time-line) i.Do a regression to learn how to more deeply- tailor second-stage treatment using O1 and O2 (in ADHD SMART, this is only with non-responders) ii.Assign each non-responder the value Ŷ i, an estimate of the outcome under the second-line treatment that yields best outcome for that person i. Responders get their observed Y i. iii.Using Ŷ i do a regression to learn how to more deeply-tailor first-stage treatment using O1

Step (ii) of Q-Learning: Graphically This analysis is with simulated data. For patients who would initially begin MED and don’t respond, we assign them their optimal outcome under the treatment that is best for them based on adherence or non- adherence. For those who would begin on BMOD and don’t respond, assign them their optimal outcome based on adherence or non- adherence.

Three Steps in Q-Learning Regression Work backwards (as you would for a project time-line) i.Do a regression to learn how to more deeply- tailor second-stage treatment using O1 and O2 (in ADHD SMART, this is only with non-responders) ii.Assign each non-responder the value Ŷ i, an estimate of the outcome under the second-line treatment that yields best outcome for that person i. Responders get their observed Y i. iii.Using Ŷ i do a regression to learn how to more deeply-tailor first-stage treatment using O1

Step (iii) of Q-Learning: First-stage treatment tailoring? To accomplish Step (iii) of Q-Learning, you may fit a regression model such as this: Ŷ = b1 + b2 o11c + b3 o12c + b4 o14c + b5 o13 + b6 o13*a1 + b7 a1 + error From such a model we would learn if, for example, o13 (prior MED) is a strong moderator of the effect of a1 such that it would make a good tailoring variable.

Step (iii) Q-Learning: Learn optimal first- treatment for all given optimal future txt Among kids using MED in prior year, it is better to start with MED. = BMOD – MED This analysis is with simulated data.

Q-Learning Step (iii): Learn optimal first- treatment for all given optimal future txt Among kids not using MED in prior year, it is better to start with BMOD = BMOD – MED This analysis is with simulated data.

SAS software to perform Q-Learning We next show you how to do Step (i) using regression Steps (ii) and (iii) using a SAS add-on known as PROC QLEARN We will use the two example regression models shown previously.

SAS code for Step (i) of Q-Learning: Second-stage treatment tailoring * use only non-responders; data dat10; set dat1; if R=0; run; * comparison of add vs intensify given first line txt and adherence; proc genmod data = dat10; model y = o11c o12c o13c o14c a1 o21c o22 a2 a2*a1 a2*o22; * effect of add vs intensify given first-line = MED x ADH status; estimate 'INT vs ADD for NR MED ADH' a2 2 a2*a1 -2 a2*o22 2 ; estimate 'INT vs ADD for NR MED Non-ADH' a2 2 a2*a1 -2 a2*o22 0 ; * effect of add vs intensify given first-line = BMOD x ADH status; estimate 'INT vs ADD for NR BMOD ADH' a2 2 a2*a1 2 a2*o22 2 ; estimate 'INT vs ADD for NR BMOD Non-ADH' a2 2 a2*a1 2 a2*o22 0 ; run; Contrast Estimate Results 95% Conf Limits Label Estimate Lower Upper P-value INT vs ADD for NR MED ADH <.0001 INT vs ADD for NR MED Non-ADH <.0001 INT vs ADD for NR BMOD ADH INT vs ADD for NR BMOD Non-ADH <.0001 This analysis is with simulated data.

Try it yourself in SAS Go to the file: sas_code_modules_4_5_and_6_ADHD.doc Copy the SAS code on Page 11 Paste into SAS Enhanced Editor window Press F8 or click the Submit button (the little running guy)

SAS code for Step (i) of Q-Learning: Second-stage treatment tailoring * use only non-responders; data dat10; set dat1; if R=0; run; * comparison of add vs intensify given first line txt and adherence; proc genmod data = dat10; model y = o11c o12c o13c o14c a1 o21c o22 a2 a2*a1 a2*o22; * effect of add vs intensify given first-line = MED x ADH status; estimate 'INT vs ADD for NR MED ADH' a2 2 a2*a1 -2 a2*o22 2 ; estimate 'INT vs ADD for NR MED Non-ADH' a2 2 a2*a1 -2 a2*o22 0 ; * effect of add vs intensify given first-line = BMOD x ADH status; estimate 'INT vs ADD for NR BMOD ADH' a2 2 a2*a1 2 a2*o22 2 ; estimate 'INT vs ADD for NR BMOD Non-ADH' a2 2 a2*a1 2 a2*o22 0 ; run; Contrast Estimate Results 95% Conf Limits Label Estimate Lower Upper P-value INT vs ADD for NR MED ADH <.0001 INT vs ADD for NR MED Non-ADH <.0001 INT vs ADD for NR BMOD ADH INT vs ADD for NR BMOD Non-ADH <.0001 This analysis is with simulated data.

SAS add-on: PROC QLEARN What do we provide PROC QLEARN? 1.Data set with O1, A1, R, O2, A2, Y 2.The second stage regression model Y ~ O1, A1, O2, A2 Specify among who to do this regression (e.g., non-responders in ADHD SMART) 3.The first stage regression model Ŷ ~ O1, A1

SAS add-on: PROC QLEARN What does PROC QLEARN do? 1.It implements Step (i): Second-stage Regression Provides us with second-stage regression parameter estimates 2.It implements Step (ii): Obtains Ŷ It assigns non-responders the outcome under the best second-stage treatment based on Step (i). It assigns responders their observed outcome. 3.It implements Step (iii): First-stage Regression Provides first-stage regression parameter estimates Provides appropriate confidence intervals for the first-stage regression parameter estimates

SAS add-on: PROC QLEARN What does the syntax look like? PROC QLEARN ; MAIN1 variables ; TAILOR1 variables ; MAIN2 variables ; TAILOR2 variables ; RESPONSE variable ; STG1TRT variable ; STG2TRT variable ; STG2SAMPLE variable ; ALPHA value ; RUN;

SAS add-on: PROC QLEARN Model Specification PROC QLEARN ; MAIN1 variables ; TAILOR1 variables ; MAIN2 variables ; TAILOR2 variables ; RESPONSE variable ; STG1TRT variable ; STG2TRT variable ;... RUN; STAGE2 REGRESSION RESULTS ARE OUTPUT LIKE THIS: INT + MAIN2 + TAILOR2 + TAILOR2*STG2TRT + STG2TRT STAGE1 REGRESSION RESULTS ARE OUTPUT LIKE THIS: INT + MAIN1 + TAILOR1 + TAILOR1*STG1TRT + STG1TRT

Recall our second-stage regression model data dat10; set dat1; if R=0; run; * use only non-responders; proc genmod data = dat10; model y = o11c o12c o13c o14c a1 o21c o22 a2 a2*a1 a2*o22; run; This is how that same model is specified in PROC QLEARN data dat11; set dat1; S = 1-R; run; * use only non-responders; proc qlearn data=dat11;...[next slide] main2 o11c o12c o13c o14c o21c; tailor2 a1 o22; stg2sample s; response y; stg2trt a2;...[next slide] run; Y = b1 + b2 o11c + b3 o12c + b4 o13c + b5 o14c + b6 o21c + b7 a1 + b8 o22 + b9 a1*a2 + b10 o22*a2 + b11 a2 + error

PROC QLEARN Full Example, ADHD Data data dat11; set dat1; S = 1-R; run; proc qlearn data=dat11 contrasts1=contrasts1 deriveci; main1 o11c o12c o14c; tailor1 o13; main2 o11c o12c o13c o14c o21c; tailor2 a1 o22; stg2sample s; response y; stg1trt a1; stg2trt a2; run; This will ask SAS to fit the following two regressions: Stage 2: Y = b1 + b2 o11c + b3 o12c + b4 o13c + b5 o14c + b6 o21c + b7 a1 + b8 o22+ b9 a1*a2 + b10 o22*a2 + b11 a2 + error (fit only with non-responders, S=1) Stage 1: Ŷ = b1 + b2 o11c + b3 o12c + b4 o14c + b5 o13 + b6 o13*a1 + b7 a1 + error This requests confidence intervals by Laber and Murphy (2012) This requests contrasts of interest (in GENMOD they are call “estimates”. See next slide for details.

PROC QLEARN: Contrast Matrix data contrasts1; input M1 M2 M3 M4 M5 M6 M7; *no. of mtrx cols = no. of stage 1 parameters ; * the cols correspond to the parameters in stage 1 model: ; * b1 + b2 o11c + b3 o12c + b4 o14c + b5 o13 + b6 a1 o13 + b7 a1 ; * each row corresponds to a different linear comb of the b's. some linear ; * combos can be used to obtain mean outcomes for different children under ; * initial BMOD vs initial MED. whereas other linear combos can be used to ; * compare mean outcomes between MED vs BMOD for different children. ; datalines; ; * taking into account optimal future decisions (to intensify vs augment) ; * contrast 1 = mean outcome under BMOD for children w/yes med in prior year; * contrast 2 = mean outcome under MED for children w/yes med in prior year ; * contrast 3 = mean diff (BMOD - MED) for children w/yes med in prior year ; * contrast 4 = mean outcome under BMOD for children w/no med in prior year ; * contrast 5 = mean outcome under MED for children w/no med in prior year ; * contrast 6 = mean diff (BMOD - MED) for children w/no med in prior year ; run;

Try it yourself in SAS Go to the file: sas_code_modules_4_5_and_6_ADHD.doc Copy the SAS code on Page 14 – This code defines the contrast matrix and runs PROC QLEARN Paste into SAS Enhanced Editor window Press F8 or click the Submit button (the little running guy)

PROC QLEARN: Results PROC QLEARN -- Q-Learning and Adaptive Confidence Intervals Number of observations in dataset: 150 Number of observations in stage 1: 150 Number of observations in stage 2: 99 First Stage Regression Result Parameter Confidence Interval Variable Estimates Upper Lower intercept o11c o12c o14c o o13 :a a Parameter Confidence Interval Contrasts Estimates Upper Lower Contrast Contrast Contrast Contrast Contrast Contrast

What did we learn with Q-learning? Adaptive Treatment Strategy Proposal If the child used MED in prior year, then begin with MED; otherwise, begin with BMOD. If the child is non-responsive and non-adherent to either first-line treatment, then AUGMENT with the other treatment option. If the child is non-responsive but adherent to either first-line treatment, then it is better to INTENSIFY first-line treatment. If the child is responsive to first-line treatment, then CONTINUE first-line treatment. This Q-learning analysis was done with simulated/altered data.

What did we learn with Q-learning? Adaptive Treatment Strategy Proposal The mean Y, school performance, under the more deeply individualized ATS obtained via Q- learning is estimated to be As expected, this is larger than the value of the ATS which started with BMOD and augmented with MED for non-responders (mean = 3.51) Recall (BMOD, MED) was the ATS with the largest mean among the 4 embedded ATSs. This Q-learning analysis was done with simulated/altered data.

Citations to Technical Reports Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, B., Fabiano, G., Waxmonsky, J., et al. (2010). Q- Learning: A data analysis method for constructing adaptive interventions. Technical Report, The Methodology Center, Penn State University. Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, B., Fabiano, G., Waxmonsky, J., Yu, J., & Murphy, S. (2010). Experimental design and primary data analysis for informing sequential decision making processes. Technical Report, The Methodology Center, Penn State University.

Practicum Autism Exercises: As before, we will go through the Autism Starter File to continue practicing/working through these primary data analyses using the Autism data set.

Extra Slides: For Statistical Aficionados! In the next two slides, we actually do Steps (ii) and (iii) of Q-Learning by hand. This is what the PROC QLEARN software automatically does. The issue, however, is that the standard errors here are incorrect. PROC QLEARN calculates the appropriate standard errors.

SAS code for Step (ii) of Q-Learning: Assign optimal outcome based on Step (i) data dat11; set dat1; * first, everyone gets their observed outcome; yhat = y; * second, re-assign the outcome for non-responders; if R=0 then yhat = *o11c *o12c *o13c *o14c *a *o21c *o22 + abs( *a *a1*a *o22*a2) ; run; proc means data=dat11; var y yhat; run; The MEANS Procedure Variable N Mean Std Dev Y yhat This analysis is with simulated data.

SAS code for Step (iii) of Q-Learning: First-stage treatment tailoring * Step (iii) regression using the new outcome; proc genmod data=dat11; model yhat = o11c o12c o14c o13 a1*o13 a1; estimate 'BMOD vs MED given MED prior yr ' a1*o13 2 a1 2; estimate 'BMOD vs MED given NO MED prior yr' a1*o13 0 a1 2; run; * medication in the year prior appears to be a tailoring variable ; * however, statistical inferences (p-values, confidence intervals) ; * should not be based on this output. ; Contrast Estimate Results Naïve 95% Conf Limits Label Estimate Lower Upper P-value BMOD vs MED given MED prior yr BMOD vs MED given MED prior yr <0.001 This analysis is with simulated data.