1. Adaptive Treatment Strategies: Challenges in Data Analysis S.A. Murphy NY State Psychiatric Institute February, 2009

2. 2 Outline What are Adaptive Treatment Strategies? Myopic Decision Making Constructing Strategies Q-Learning & Voting Example using CATIE

3. 3 Adaptive Treatment Strategies operationalize multi- stage decision making. These are individually tailored sequences of interventions, with intervention type and dosage adapted to the individual. Generalization from a one-time decision to a sequence of decisions concerning interventions Operationalize clinical practice. Each decision corresponds to a stage of intervention

4. 4 Example of an Adaptive Treatment Strategy Treatment of depression. Goal is to achieve and maintain remission. Provide Citalopram for up to 12 weeks gradually increasing dose as required. If, there is no remission yet either the maximum dose has been provided for two weeks, or 12 weeks have occurred, then if there has been a 50% improvement in symptoms, augment with Mirtazapine. else switch treatment to Bupropion. Else (remission is achieved) maintain on Citalopram and provide web-based disease management.

5. 5 Adaptive Treatment Strategy At each stage of intervention –Use tailoring variables (symptoms, adherence, side effects, social support, family support…), –To select actions/interventions such as medications, behavioral therapies, augmentations,…. –In order to maximize long term rewards (maximize functionality, symptom-free time).

6. 6 Why use an Adaptive Treatment Strategy? –High heterogeneity in response to any one intervention What works for one person may not work for another What works now for a person may not work later –Improvement often marred by relapse Remitted or few current symptoms is not the same as cured. –Co-occurring disorders/adherence problems are common

7. 7 Outline What are Adaptive Treatment Strategies? Myopic Decision Making Constructing Strategies Q-Learning Example using CATIE

8. 8 Myopic Decision Making In myopic decision making, decision makers use strategies that seek to maximize immediate rewards. Problems: –Ignore longer term consequences of present actions. –Ignore the range of feasible future actions/interventions –Ignore the fact that immediate responses to present actions may yield information that pinpoints best future actions (A strategy tells us how to use the observations to choose the actions/interventions.)

9. 9 Treatment of Schizophrenia Myopic action: Offer patients a treatment that reduces schizophrenia symptoms for as many people as possible. The result: Some patients are not helped and/or experience abnormal movements of the voluntary muscles (TDs). The class of subsequent medications is greatly reduced. The mistake: We should have taken into account the variety of treatments available to those for whom the first treatment is ineffective. The message: Use an initial medication that may not have as large a success rate but that will be less likely to cause TDs.

10. 10 Treatment of Opioid Dependence Myopic action: Choose an intensive multi-component treatment (methadone + counseling + behavioral contingencies) that immediately reduces opioid use for as many people as possible. The result: Behavioral contingencies are burdensome/expensive to implement and many people may not need the contingencies to improve. The mistake: We should allow the patient to exhibit poor adherence prior to implementing the behavioral contingencies. The message: Use an initial treatment that may not have as large an immediate success rate but will allow us to ascertain whether behavioral contingencies are required.

11. 11 Outline What are Adaptive Treatment Strategies? Myopic Decision Making Constructing Strategies Q-Learning & Voting Example using CATIE

12. 12 Basic Idea for Constructing an Adaptive Treatment Strategy: Move Backwards Through Stages. (Pretend you are “All-Knowing”)

13. 13 2 Stages for each individual Observations available at j th stage Action at j th stage

14. 14 2 Stages History available at each stage Primary Outcome/Reward:

15. 15 An Adaptive Treatment Strategy is the sequence of decision rules: A simple decision rule is: given weights β, switch treatment at stage j if otherwise maintain on current treatment; S j is a vector summary of the history, H j.

16. 16 Goal: Use data to construct decision rules that input information in the history at each stage and output a recommended decision; these decision rules should lead to a maximal mean Y. In the future we employ the actions recommended by the decision rules:

17. 17 Idealized Data for Constructing an Adaptive Treatment Strategy: Data from sequential, multiple assignment, randomized trials in which at each stage subjects are randomized among alternative options. That is, A j is a randomized action with known randomization probability. Binary actions with P[A j =1]=P[A j =-1]=.5

18. 18 Outline What are Adaptive Treatment Strategies? Myopic Decision Making Constructing Strategies Q-Learning & Voting Example using CATIE

19. 19 Regression-based methods for constructing decision rules Q-Learning (Watkins, 1989) (a popular method from computer science) A-Learning or optimal nested structural mean model (Murphy, 2003; Robins, 2004) The first method is an inefficient version of the second method when each stages’ covariates include the prior stages’ covariates and the actions are centered to have conditional mean zero.

20. 20 Basic Idea for Constructing a Regime: Move Backwards Through Stages.

21. 21 (k=2) Dynamic Programming

22. 22 Approximate for S', S vector summaries of the history, A Simple Version of Q-Learning –binary actions Stage 2 regression: Use least squares with outcome, Y, and covariates to obtain Set Stage 1 regression: Use least squares with outcome, and covariates to obtain

23. 23 Approximate for S', S vector summaries of the history, A Simple Version of Q-Learning –binary actions Stage j decision rule: Select treatment = 1 if Otherwise select treatment = -1

24. 24 Voting (exploratory analysis) Estimate the chance that if we were to rerun the trial we would see treatment A j =1 perform better than treatment A j =-1 for patients with history h j. That is, Use bootstrap

25. 25 Voting (exploratory analysis) Use bootstrap samples to estimate percentage of the time that treatment A 1 =1 is favored: Natural approach will not work, e.g. is not necessarily consistent. We use an adaptive bootstrap

26. 26 Voting (exploratory analysis) Use an “adaptive” bootstrap method to estimate percentage of the time that treatment A 1 =1 is favored: Adaptive bootstrap estimator:

27. 27 Outline What are Dynamic Treatment Regimes? Myopic Decision Making Constructing Regimes Q-Learning & Voting Example using CATIE

28. 28 Clinical Antipsychotic Trials of Intervention Effectiveness (Schizophrenia) Multi-stage trial of 18 months duration Relaxed entry criteria A large number of sites representing a broad array of clinical settings (state mental health, academic, Veterans’ Affairs, HMOs, managed care) Approximately 1500 patients

29. 29 CATIE Randomizations (simplified) Phase 1 Randomized Treatments OLAN QUET RISP ZIPR PERP Phase 2 Treatment preference Efficacy Tolerability Randomized Treatments CLOZ OLAN QUET RISP OLAN QUET RISP ZIPR Phase 3 Treatments selected many options by preference

30. 30 Exploratory Analyses Reward: Time to Treatment Dropout Phase 1 analysis: –Controls: TD, recent exacerbation, site –Tailoring variable: pretreatment PANSS Phase 2 analysis: –Controls: TD, recent exacerbation, site –Tailoring variables: “treatment preference,” phase 1 treatment, end of phase 1 PANSS Constructing Dynamic Treatment Regimes using CATIE

31. 31

32. 32

33. 33

34. 34 Challenges It is extremely challenging to provide measures of confidence that possess “good frequentist properties.” Clinical Decision Support Systems –We need to be able construct dynamic treatment regimes that recommend a group of treatment actions when there is no evidence that a particular treatment action is best. Even in a randomized trial setting, the most straightforward analyses are subject to confounding bias. Some methods to avoid confounding bias are available.

35. 35 Acknowledgements: This presentation is based on work with many individuals including Eric Laber, Dan Lizotte, John Rush, Scott Stroup, Joelle Pineau, Daniel Almirall and Bibhas Chakraborty,. Email address: samurphy@umich.edu Slides with notes at: http://www.stat.lsa.umich.edu/~samurphy/ Click on seminars > health science seminars