Substance Abuse, Multi-Stage Decisions, Generalization Error How are they connected?! S.A. Murphy Univ. of Michigan CMU, Nov., 2004

Outline Chronic Disorders Multi-Stage Decisions & Reinforcement Learning Statistical Challenges Generalization Error Discussion

Chronic Disorders

Why does the management of a chronic disorder such as drug dependence require multi-stage decisions? High variability across patients in response to any one treatment No Cure Relapse is likely without either continuous or intermittent treatment for a large proportion of people. What works now may not work later Exacerbations in disorder may occur if there are no alterations in treatment

Additional Issues in Managing the Chronic Disorder Treatment is often burdensome, especially over time Patient adherence is a critical issue Co-occurring problems are common

High-Level Questions What is the best sequencing of therapies? What is the best timings of alterations in therapies? What information do we use to make these decisions?

Multi-Stage Decisions & Reinforcement Learning

k Decisions on one individual Observation made prior to j th decision point Decision at j th decision point Primary outcome Y is a specified summary of decisions and observations

Goal : Construct decision rules that input data at each decision point and output a recommended decision; these decision rules should lead to a maximal mean Y. where eachis a low dimensional summary of for j=1,…., k

An example of a decision rule is: alter treatment if otherwise maintain on current treatment.

Conceptual Formulation

Statistical Challenges

Construction of summaries useful for decision making High dimensional noisy information In many cases summary must be meaningful Construction of decision rules Variety of data sources In many cases decision rule must be meaningful Experimental designs Evaluation versus construction/refinement of decision rules.

Experimental Design Adaptive Treatment Strategies are multi-component treatments Multiple decision points through time Different kinds of decisions Decision options for improving patients are often different from decision options for non-improving patients Services Research Delivery Mechanisms Training of Staff…….

Conceptual Formulation

Goal : Provide experimental designs for developing or refining adaptive treatment strategies Advocating for a series of developmental, experimental trials --- Collins, Murphy, Nair & Strecher (2004) Designs similar to full factorial designs --- Lavori & Dawson (2004), Lavori (2001) --- Murphy (2004) Need designs that are similar to balanced fractional factorials

Examples of sequential multiple assignment randomized trials: CATIE (2001) Treatment of Psychosis in Alzheimer’s Patients CATIE (2001) Treatment of Psychosis in Schizophrenia STAR*D (2003) Treatment of Depression Oslin (ongoing) Treatment of Alcoholism

Construction of decision rules Data: n (patient) trajectories of observations and decisions---from one patient we see: Decisions are randomized among feasible options Y is known summary of all decisions and observations Estimate decision rules so as to maximize mean of Y

Data Structure (k=2)

Some Methods Q-Learning (Watkins, 1989) (one of many……many methods from reinforcement learning) ---regression A-Learning (Murphy, 2003; Robins, 2004; Blatt et al. 2004) ---regression on a mean zero space Weighting (Murphy, van der Laan & Robins, 2002) ---weighted mean

Generalization Error

Goal : Find the decision rules that are best (maximize mean Y) within a restricted class of decision rules: e.g.,

One decision only! Data: is randomized with probability

The estimand is or equivalently

A Direct Approach But not feasible, particularly if more than one decision!!

Q-Learning Approximate Minimize

Even if our sample is infinite, is not necessarily close to The difference is the generalization error.

The message: In adding information we changed our goal. Our estimator will not (even asymptotically) achieve equivalently

Can we add information without changing our goal? –Can we “guide” Q-learning, A-learning and other methods closer to our goal? –Do we need a theory that compares biased estimators? –Is there a way to use a sieve without changing our goal?

Discussion

Open Problems How might we form summaries of high dimensional noisy data so as to make good decisions? (Prediction is not the goal) How might we use structural models to inform the design of randomized experiments? How should we design experiments if our goal is building or refining an adaptive treatment strategy?

Open Problems How might we use observational data to estimate good adaptive treatment strategies (e.g. decision rules)? How might we use data in which an adaptive treatment strategy was implemented to improve the decision rules? How might we “guide” Q-Learning or A-Learning so as to more closely achieve our goal? How might Bayesian methods be used here?

This seminar can be found at: cmu1104.ppt Further information on adaptive treatment strategies can be found at:

Q-Learning (k=2) Data:

Q-Learning Minimize