1. 1 Machine/Reinforcement Learning in Clinical Research S.A. Murphy May 19, 2008

2. 2 Outline Goal: Improving Clinical Decision Making Using Data –Clinical Decision Making –Types of Training Data –Incomplete Mechanistic Models –Clinical Trials –Some Open Problems –Example

3. 3

4. 4

5. 5 Questions Patient Evaluation Screen with MSE

6. 6

7. 7 Policies are individually tailored treatments, with treatment type and dosage changing according to the patient’s outcomes. k Stages for each patient Observation available at j th stage Action at j th stage (usually a treatment)

8. 8 k Stages History available at j th stage Reward following j th stage (r j is a known function) Primary Outcome:

9. 9 Goal : Use training data to construct decision rules, d 1,…, d k that input information in the history at each stage and output a recommended action; these decision rules should lead to a maximal mean Y (cumulative reward). The policy is the sequence of decision rules, d 1,…, d k. In implementation of the policy the actions are set to:

10. 10 Some Characteristics of a Clinical Decision Making Policy The learned policy should be decisive only when warranted. The learned policy should not require excessive data collection in order to implement. The learned policy should be justifiable to clinical scientists.

11. 11 Types of Data Clinical Trial Data –Actions are manipulated (randomized) Large Databases or Observational Data Sets –Actions are not manipulated by scientist Bench research on cells/animals/humans

12. 12 Clinical Trial Data Sets Experimental trials conducted for research purposes –Scientists decide proactively which data to collect and how to collect this data –Use scientific knowledge to enhance the quality of the proxies for observation, reward –Actions are manipulated (randomized) by scientist –Short Horizon (less than 5) –Hundreds of subjects.

13. 13 Observational Data Sets Observational data collected for research purposes –Use scientific knowledge to pinpoint high quality proxies for observation, action, reward –Scientists decide proactively which proxies to collect and how to collect this data –Actions are not manipulated by scientist –Moderate Horizon –Hundreds to thousands of subjects.

14. 14 Observational Data Sets Clinical databases or registries– (an example in the US would be the VA registries) –Data was not collected for research purposes –Only gross proxies are available to define observation, action, reward –Moderate to Long Horizon –Thousands to Millions of subjects

15. 15 Mechanistic Models In many areas of RL, scientists can use mechanistic theory, e.g., physical laws, to model or simulate the interrelationships between observations and how the actions might impact the observations. Scientists know many (the most important) of the causes of the observations and know a model for how the observations relate to one another.

16. 16 Low Availability of Mechanistic Models Clinical scientists have recourse to only crude, qualitative models Unknown causes create problems. Scientists who want to use observational data to construct policies must confront the fact that non-causal “associations” occur due to the unknown causes of the observations.

17. 17 Conceptual Structure in the Clinical Sciences (observational data)

18. 18 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

19. 19 Unknown, Unobserved Causes (Incomplete Mechanistic Models) Problem: Non-causal associations between treatment (here counseling) and rewards are likely. Solutions: –Collect clinical trial data in which treatment actions are randomized. This breaks the non-causal associations yet permits causal associations. –Participate in the observational data collection; proactively brainstorm with domain experts to ascertain and measure the main determinants of treatment selection. Then take advantage of causal inference methods designed to utilize this information.

20. 20 Conceptual Structure in the Clinical Sciences (experimental trial data)

21. 21 STAR*D The statistical expertise relevant for policy construction was unavailable at the time the trial was designed. This trial is over and one can apply for access to this data One goal of the trial is construct good treatment sequences for patients suffering from treatment resistant depression. www.star-d.org

22. 22

23. 23 ExTENd Ongoing study at U. Pennsylvania Goal is to learn how best to help alcohol dependent individuals reduce alcohol consumption.

24. 24 Oslin ExTENd Late Trigger for Nonresponse 8 wks Response TDM + Naltrexone CBI Random assignment: CBI +Naltrexone Nonresponse Early Trigger for Nonresponse Random assignment: Naltrexone 8 wks Response Random assignment: CBI +Naltrexone CBI TDM + Naltrexone Naltrexone Nonresponse

25. 25 Clinical Trials Data from the --short horizon– clinical trials make excellent test beds for combinations of supervised/unsupervised and reinforcement learning methods. –In the clinical trial large amounts of data are collected at each stage of treatment –Small number of finite horizon patient trajectories –The learned policy can vary greatly from one training set to another.

26. 26 Open Problems 1)Equivalent Actions –Need to know when a subset of actions are equivalent–that is, when there is no or little evidence to contradict this equivalence. 2)Evaluation –Need to assess the quality of the learned policy (or compare policies) using training data

27. 27 Open Problems 3)Variable Selection –To reduce the large number of variables to those most useful for decision making –Once a small number of variables is identified, we need to know if there is sufficient evidence that a particular variable (e.g. output of a biological test) should be part of the policy.

28. 28 Measures of Confidence A statistician’s approach: use measures of confidence to address these three challenges –Pinpointing equivalent actions –Pinpointing necessary patient inputs to the policy –Evaluating the quality of a learned policy

29. 29 Evaluating the quality of a learned policy using the training data Traditional methods for constructing measures of conference require differentiability (to assess the variation in the policy from training set to training set). The mean outcome following use of a policy (the value of the policy) is a non-differentiable function of the policy.

30. 30 Example: Single Stage (k=1) Find a prediction interval for the mean outcome if a particular estimated policy (here one decision rule) is employed. Action A is binary in {-1,1}. Suppose the decision rule is of form We do not assume the Bayes decision boundary is linear.

31. 31 Single Stage (k=1) Mean outcome following this policy is is the randomization probability

32. 32 Prediction Interval for Two problems V(β) is not necessarily smooth in β. We don’t know V so V must be estimated as well. Data set is small so overfitting is a problem.

33. 33 Similar Problem in Classification Misclassification rate for a given decision rule (classifier) where V is defined by (A is the {-1,1} classification; O 1 is the observation; β T O 1 is a linear classification boundary)

34. 34 Jittering is non-smooth. Toy Example: The unknown Bayes classifier has quadratic decision boundary. We fit, by least squares, a linear decision boundary f(o)= sign(β 0 + β 1 o)

35. 35 Jittering of N=100 N=30

36. 36 Simulation Example Data Sets from the UCI repository Use squared error loss to form classification rule Sample 30 examples from each data set; for each sample construct prediction interval. Assess coverage using remaining examples. Repeat 1000 times

37. 37 “95% Prediction Intervals” Data SetPercentile Bootstrap Adjusted Bootstrap Ionosphere.61.93 Heart.41.98 Simulated.83.72 Confidence rate should be ≥.95

38. 38 Prediction Interval for Our method obtains a prediction interval for a smooth upper bound on is training error.

39. 39 Prediction Interval for where is the set of close to in terms of squared error loss. Form a percentile bootstrap interval for this smooth upper bound. This method is generally too conservative

40. 40 “95% Prediction Intervals” Data SetCUDCVInverse Binomial Ionosphere (width) 1.00 (.4).99 (.5).75 (.3) Heart (width) 1.00 (.5) 1.00 (.4).46 (.4) Simulated (width).99 (.5).76 (.5).95 (.4) Confidence rate should be ≥.95

41. 41 A Challenge! Methods for constructing the policy (or classifier) and providing an evaluation of the policy (or classifier) must use same small data set. How might you better address this problem?

42. 42 Discussion 1)Equivalent Actions: Need to know when a subset of actions are equivalent–that is, when there is no or little evidence to contradict this equivalence. 2)Evaluating the usefulness of a particular variable in the learned policy. 3)Methods for producing composite rewards. –High quality elicitation of functionality 4)Feature construction for decision making in addition to prediction

43. 43 This seminar can be found at: http://www.stat.lsa.umich.edu/~samurphy/ seminars/Benelearn08.ppt Email me with questions or if you would like a copy: samurphy@umich.edu

44. 44 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

45. 45 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

46. 46 Questions Patient Evaluation Screen with MSE