1. Dynamic Treatment Regimes, STAR*D & Voting D. Lizotte, E. Laber & S. Murphy LSU ---- Geaux Tigers! April 2009

2. 2 Outline Dynamic Treatment Regimes Constructing Regimes from Data A Measure of Confidence: Voting STAR*D

3. 3 Dynamic treatment regimes are individually tailored treatments, with treatment type and dosage changing according to patient outcomes. Operationalize clinical practice. k Stages for one individual Observation available at j th stage Action at j th stage (usually a treatment)

4. 4 Goal : Construct decision rules that input information available at each stage and output a recommended decision; these decision rules should lead to a maximal mean Y. Y is a known function of The dynamic treatment regime is the sequence of two decision rules: k=2 Stages

5. 5 Deriving the Optimal Dynamic Regime: Move Backwards Through Stages. You know multivariate distribution

6. 6 Optimal Dynamic Treatment Regime satisfies

7. 7 Data for Constructing the Dynamic Treatment Regime: Subject data from sequential, multiple assignment, randomized trials. At each stage subjects are randomized among alternative options. A j is a randomized treatment with known randomization probability.

8. 8 STAR*D

9. 9 STAR*D Analyses X 1 includes site, preference for future treatment and can include other baseline variables. X 2 can include measures of symptoms (Qids), side effects, preference for future treatment Y is (reverse-coded) the minimum of the time to remission and 30 weeks.

10. 10 Outline Dynamic Treatment Regimes Constructing Regimes from Data A Measure of Confidence: Voting STAR*D

11. 11 Regression-based methods for constructing decision rules Q-Learning (Watkins, 1989) (a popular method from computer science) Optimal nested structural mean model (Murphy, 2003; Robins, 2004) The first method is equivalent to an inefficient version of the second method, if we use linear models and each stages’ covariates include the prior stages’ covariates and the actions are centered to have conditional mean zero.

12. 12 There is a regression for each stage. A Simple Version of Q-Learning – Stage 2 regression: Regress Y on to obtain Stage 1 regression: Regress on to obtain

13. 13 for patients entering stage 2: is the average outcome conditional on patient history (no remission in stage 1; includes past treatment and variables affected by stage 1 treatment). is the estimated average outcome assuming the “best” treatment is provided at stage 2 (note max in formula). is the dependent variable in the stage 1 regression for patients moving to stage 2

14. 14 Optimal Dynamic Treatment Regime satisfies

15. 15 A Simple Version of Q-Learning – Stage 2 Q function, (Y was dependent variable) yields Stage 1 Q function, ( was dependent variable) yields

16. 16 Decision Rules:

17. 17 Outline Dynamic Treatment Regimes Constructing Regimes from Data A Measure of Confidence: Voting STAR*D

18. 18 Measures of Confidence Classical –Confidence/Credible intervals and/or p- values concerning the β 1, β 2. –Confidence/Credible intervals concerning the average response if is used in future to select the treatments.

19. 19 A Measure of Confidence for use in Exploratory Data Analysis Replication Probability –Estimate the chance that a future trial would find a particular stage j treatment best for a given s j. The vote for treatment a j * is

20. 20 A Measure of Confidence for use in Exploratory Data Analysis Replication Probability –If stage j treatment a j is binary, coded in {-1,1}, then

21. 21 Bootstrap Voting Use bootstrap samples to estimate by

22. 22 The Vote: Intuition If has a normal distribution with variance matrix then is

23. 23 Bootstrap Voting The naïve bootstrap vote estimator is inconsistent.

24. 24 Bootstrap Voting A consistent bootstrap vote estimator of is where is smooth and

25. 25 Bootstrap Voting In our simple example is approximately

26. 26 What does the vote mean? is similar to the p-value for the hypothesis in that it converges, as n increases, to 1 or 0 depending on the sign of If then the limiting distribution is not uniform; instead converges to a constant.

27. 27 Outline Dynamic Treatment Regimes Constructing Regimes from Data A Measure of Confidence: Voting STAR*D

28. 28 STAR*D

29. 29 STAR*D Regression formula at stage 2:

30. 30

31. 31 STAR*D Regression formula at stage 1:

32. 32 STAR*D Decision Rule for subjects preferring a switch at stage 1 if offer VEN if offer SER if offer BUP

33. 33 STAR*D Level 2, Switch

34. 34

35. 35

36. 36 Truth in Advertising: STAR*D Missing Data + Study Drop-Out 1200 subjects begin level 2 (e.g. stage 1) 42% study dropout during level 2 62% study dropout by 30 weeks. Approximately 13% item missingness for important variables observed after the start of the study but prior to dropout.

37. 37 Truth in Advertising: STAR*D Multiple Imputation within Bootstrap 1000 bootstrap samples of the 1200 subjects Using the location-scale model we formed 25 imputations per bootstrap sample. The stage j Q-function (regression function) for a bootstrap sample is the average of the 25 Q-functions over the 25 imputations.

38. 38 Discussion We consider the use of voting to provide a measure of confidence in exploratory data analyses. Our method of adapting the bootstrap voting requires a tuning parameter, γ. It is unclear how to best select this tuning parameter. We ignored the bias in estimators of stage 1 parameters due to the fact that these parameters are non-regular. The voting method should be combined with bias reduction methods.

39. 39 This seminar can be found at: http://www.stat.lsa.umich.edu/~samurphy/ seminars/LSU2009.ppt Email me with questions or if you would like a copy! samurphy@umich.edu