1. Dynamic Treatment Regimes, STAR*D & Voting D. Lizotte, E. Laber & S. Murphy Psychiatric Biostatistics Symposium May 2009

2. 2 Outline Dynamic Treatment Regimes Constructing Regimes from Data A Measure of Confidence: Voting STAR*D

3. 3 Dynamic treatment regimes are individually tailored treatments, with treatment type and dosage changing according to patient outcomes. Operationalize clinical practice. k Stages for one individual Observation available at j th stage Action at j th stage (usually a treatment)

4. 4 A dynamic treatment regime is the sequence of two decision rules: for selecting the actions in future. Goal : Construct decision rules that lead to a maximal mean Y. Y is a known function of k=2 Stages

5. 5 Deriving the Optimal Dynamic Regime: Move Backwards Through Stages. You know multivariate distribution

6. 6 Optimal Dynamic Treatment Regime satisfies

7. 7 Data for Constructing the Dynamic Treatment Regime: Subject data from sequential, multiple assignment, randomized trials. At each stage subjects are randomized among alternative options. A j is a randomized treatment with known randomization probability.

8. 8 STAR*D

9. 9 STAR*D Observations and Reward X 1 includes site, preference for future treatment and can include other baseline variables. X 2 can include measures of symptoms (Qids), side effects, preference for future treatment Y is (reverse-coded) the minimum of the time to remission and 30 weeks.

10. 10 Outline Dynamic Treatment Regimes Constructing Regimes from Data A Measure of Confidence: Voting STAR*D

11. 11 Regression-based methods for constructing decision rules Q-Learning (Watkins, 1989) (a popular method from computer science) Optimal nested structural mean model (Murphy, 2003; Robins, 2004) The first method is equivalent to an inefficient version of the second method, if we use linear models and each stages’ covariates include the prior stages’ covariates and the actions are centered to have conditional mean zero.

12. 12 There is a regression for each stage. A Simple Version of Q-Learning – Stage 2 regression: Regress Y on to obtain Stage 1 regression: Regress on to obtain

13. 13 for patients entering stage 2: is the average outcome conditional on patient history (no remission in stage 1; includes stage 1 treatment and stage 2 variables affected by stage 1 treatment). is the estimated average outcome assuming the “best” treatment is provided at stage 2 (note max in formula). is the dependent variable in the stage 1 regression for patients moving to stage 2

14. 14 Optimal Dynamic Treatment Regime satisfies

15. 15 A Simple Version of Q-Learning – Stage 2 Q function, (Y was dependent variable) yields Stage 1 Q function, ( was dependent variable) yields

16. 16 Decision Rules:

17. 17 Outline Dynamic Treatment Regimes Constructing Regimes from Data A Measure of Confidence: Voting STAR*D

18. 18 Measures of Confidence Classical –Confidence/Credible intervals and/or p- values concerning the β 1, β 2. –Confidence/Credible intervals concerning the average response if is used in future to select the treatments.

19. 19 A Measure of Confidence for use in Exploratory Data Analysis Voting –Estimate the chance that a future trial would find a particular stage j treatment best for a given s j. The vote for treatment a j * is

20. 20 A Measure of Confidence for use in Exploratory Data Analysis Voting –If stage j treatment a j is binary, coded in {-1,1}, then

21. 21 Bootstrap Voting Use bootstrap samples to estimate Should we use ?

22. 22 The Vote: Intuition If has a normal distribution with variance matrix then is

23. 23 Bootstrap Voting The naïve bootstrap vote estimator is inconsistent.

24. 24 Bootstrap Voting A consistent bootstrap vote estimator of is where is a shrinkage term. is smooth in u and

25. 25 Bootstrap Voting In our simple example is approximately

26. 26 How does the vote behave? is similar to the p-value for the hypothesis in that it converges, as n increases, to 1 or 0 depending on the sign of If then the limiting distribution is not uniform; instead converges to.5.

27. 27 What does the vote mean? The vote is an e stimate of the chance that a future trial would find a particular stage j treatment best for a given s j. The vote for treatment a j * is

28. 28 Outline Dynamic Treatment Regimes Constructing Regimes from Data A Measure of Confidence: Voting STAR*D

29. 29 STAR*D

30. 30 STAR*D Regression formula at stage 2:

31. 31

32. 32 STAR*D Regression formula at stage 1:

33. 33 STAR*D Decision Rule for subjects preferring a switch at stage 1 if offer VEN if offer SER if offer BUP

34. 34 STAR*D Level 2, Switch

35. 35

36. 36

37. 37 Truth in Advertising: STAR*D Missing Data + Study Drop-Out 1200 subjects begin level 2 (e.g. stage 1) 42% study dropout during level 2 62% study dropout by 30 weeks. Approximately 13% item missingness for important variables observed after the start of the study but prior to dropout.

38. 38 Truth in Advertising: STAR*D Multiple Imputation within Bootstrap 1000 bootstrap samples of the 1200 subjects Using the location-scale model we formed 25 imputations per bootstrap sample. The stage j Q-function (regression function) for a bootstrap sample is the average of the 25 Q-functions over the 25 imputations.

39. 39 Discussion We consider the use of voting to provide a measure of confidence in exploratory data analyses. Our method of adapting the bootstrap voting requires a tuning parameter, γ. It is unclear how to best select this tuning parameter (Empirical Bayes?). We ignored the bias in estimators of stage 1 parameters due to the fact that these parameters are non-regular. The voting method should be combined with bias reduction methods.

40. 40 This seminar can be found at: http://www.stat.lsa.umich.edu/~samurphy/ seminars/PsychWorkshop2009.ppt Email me with questions or if you would like a copy! samurphy@umich.edu