1. Using Clinical Trial Data to Construct Policies for Guiding Clinical Decision Making S. Murphy & J. Pineau American Control Conference Special Session June, 2009

2. 2 Outline –Sequential Clinical Decision Making –Clinical Trials –Challenges Incomplete, primitive, mechanistic models Measures of Confidence –Illustration Long Term Goal: Improving Clinical Decision Making Using Data

3. 3

4. 4 Critical Decisions Which treatments should be offered first? How long should we wait for these treatments to work? How long should we wait before offering a transition to a maintenance stage? Which treatments should be offered next? All of these questions relate to the formulation of a policy.

5. 5 Examples of Clinical Trials Sequenced RBT: Goal is to improve neonatal outcomes STAR*D: Goal is to achieve depression remission.

6. 6 Jones’ Study for Drug-Addicted Pregnant Women rRBT 2 wks Response rRBT tRBT Random assignment: rRBT Nonresponse tRBT Random assignment: aRBT 2 wks Response Random assignment: eRBT tRBT rRBT Nonresponse

7. 7 STAR*D

8. 8 Challenges Incomplete Mechanistic Models –non-causal “associations” in data occur due to the unknown causes of the observations Small, Expensive, Data Sets with High Noise to Signal Ratio –Measures of confidence are essential

9. 9 Conceptual Structure in the Behavioral Sciences (clinical trial data)

10. 10 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

11. 11 The problem: Even when treatments are randomized, non-causal associations occur in the data. Solutions: – Recognize that parts of the transition probabilities (“system dynamics”) can not be informed by domain expertise as these parts reflect non-causal associations –Or use methods for constructing policies that “average” over the non-causal associations between action and cost or reward. Unknown, Unobserved Causes (Incomplete Mechanistic Models)

12. 12 Measures of Confidence We would like measures of confidence for the following: –To assess if there is sufficient evidence that a particular observation (e.g. output of a biological test) should be part of the policy. –To assess if there is sufficient evidence that a subset of the actions lead to lower cost than the remaining actions. (reward=-cost)

13. 13 Measures of Confidence Traditional methods for constructing measures of confidence require differentiability (if frequentist properties are desired). Optimal policies are constructed via non- differentiable operations (e.g. minimization/maximization).

14. 14 STAR*D

15. 15 STAR*D Stage 1 Observation: QIDS: low score is desirable Preference for type of Stage 1 treatment: Switch or Augment Stage 1Treatment Action: If Stage 1 preference is Switch then randomize switch to either Ser, Bup or Ven; if Stage 1 preference is Augment then randomize to augment with Bup or Bus. Stage 2 Observation: QIDS: low score is desirable Preference for type of Stage 2 treatment: Switch or Augment Stage 2 Treatment Action: If Stage 2 preference is Switch then randomize switch to either Mirt or Ntp: if Stage 2 preference is Augment then randomize to augment with Li or Thy Patients exit to follow-up if remission is achieved (QIDS ≤ 5).

16. 16 Construct the policy to minimize cost (or maximize reward) Cost: minimum of time to remission and 30 weeks. Construct policy so as to minimize average cost

17. 17 Algorithm Fitted Q-iteration with linear function approximation. One estimates the “state-action cost” function at stages 1,2 via a linear model. Use voting across bootstrap samples (approximate double bootstrap) to assess confidence that a particular action is best. (cost=-value=-benefit-to-go)

18. 18

19. 19 Conclusion for Stage 1(level 2) If QIDS is >13 then both Ven and Bup are best treatment actions If QIDS is <9 then Ser is best treatment action. If QIDS is around 10-13 then no real winner(s).

20. 20 Discussion If modern control methods are to be used with clinical trial data then these methods must accommodate the existence of unknown, unobserved variables influencing observations at multiple stages, should provide measures of confidence and must be combined with modern missing data methods.

21. 21 This seminar can be found at: http://www.stat.lsa.umich.edu/~samurphy/ seminars/ACC06.09.ppt Email me with questions or if you would like a copy! samurphy@umich.edu

22. 22 The Problem Many patients dropout of the study. Stage 1Stage 2 Remit38336 Move to next stage 456260 Dropout362160 Sum1201456