1 Dynamic Treatment Regimens S.A. Murphy PolMeth XXV July 10, 2008

2 Outline –Three apparently dissimilar problems –Myopic decision making –Constructing policies –Challenges Unknown, unobserved causes Small, expensive data sets –Discussion

3 Three Apparently Dissimilar Problems –Artificial Intelligence: Autonomous Helicopter Flight –Management of Substance Abuse/Mental Illness –Management of Welfare for Recipients

4 Artificial Intelligence Autonomous Helicopter Flight –Observations: characteristics of the helicopter (position, orientation, velocity, angular velocity, ….), characteristics of the environment (wind speed, wind angle, turbulence….) –Actions/treatments: cyclic pitch (causes forward/backward and sideways acceleration), tilt angle of main rotor blades (direction), tail rotor pitch control (turning) –Rewards: Closeness of helicopter’s flight path to the desired path; avoidance of crashes(!)

5 Andrew Ng’s Helicopter:

6 The Management of Substance Abuse/Mental Illness Treating Patients with Opioid Dependence (heroin) –Observations: individual characteristics (withdrawal symptoms, craving, attendance at counseling sessions, results of urine tests….), characteristics of the environment (family support, employment.…) –Actions/treatments: methadone dose, amount of weekly group counseling sessions, daily dosing time of methadone, individual counseling sessions, methadone taper –Rewards: minimize opioid use and maximize health/functionality, minimize cost

7

8 Management of a Welfare Program “Jobs First” Program in Connecticut –Observations: individual characteristics (assets, income, age, health, employment), characteristics of the environment (domestic violence, incapacitated family member, # children, living arrangements…) –Actions/treatments: child care, job search skills training, amount of cash benefit, medical assistance, education –Rewards: maximize employment/independence.

9

10 The Common Thread: Multi-Stage Decision Making Observation, action, observation, action, observation, action,……………………. A policy tells us how to use the observations to choose the actions. We’d like to develop policies that maximize the rewards.

11 Questions for the Methodologist What kinds of data are most useful for developing policies? How do we use limited and expensive data to construct good policies? How do we evaluate policies using the limited data? (A policy tells us how to use the observations to choose the actions.)

12 Outline –Three apparently dissimilar problems –Myopic decision making –Constructing policies –Challenges Unknown, unobserved causes Small, expensive data sets –Discussion

13 Myopic Decision Making In myopic decision making, decision makers use policies that seek to maximize immediate rewards. Problems: –Ignore longer term consequences of present actions. –Ignore the range of feasible future actions/treatments –Ignore the fact that immediate responses to present actions may yield information that pinpoints best future actions (A policy tells us how to use the observations to choose the actions.)

14 Autonomous Helicopter Flight The helicopter has veered from flight plan. Myopic action: Choose an acceleration and direction that will ASAP bring us back to the flight plan. The result: The myopic action results in the helicopter overshooting the planned flight path and in drastic situations may lead to the helicopter cycling out of control. The mistake: We did not consider the range of actions we can take following the initial action. The ability to slow down is mechanically limited. The message: Use an acceleration that will not return us as quickly to the planned flight path but will take into account the ability of the helicopter to slow down and reduce the overshoot.

15 Treatment of Psychosis Myopic action: Offer patients a treatment that reduces psychosis for as many people as possible. The result: Some patients are not helped and/or experience abnormal movements of the voluntary muscles (TDs). The class of subsequent medications is greatly reduced. The mistake: We should have taken into account the variety of treatments available to those for whom the first treatment is ineffective. The message: Use an initial medication that may not have as large a success rate but that will be less likely to cause TDs.

16 Treatment of Opioid Dependence Myopic action: Choose an intensive multi-component treatment (methadone + counseling + behavioral contingencies) that immediately reduces opioid use for as many people as possible. The result: Behavioral contingencies are burdensome/expensive to implement and many people may not need the contingencies to improve. The mistake: We should allow the patient to exhibit poor adherence prior to implementing the behavioral contingencies. The message: Use an initial treatment that may not have as large an immediate success rate but carefully monitor patient adherence to ascertain if behavioral contingencies are required.

17 Outline –Three apparently dissimilar problems –Myopic decision making –Constructing policies –Challenges Unknown, unobserved causes Small, expensive data sets –Discussion

18 Basic Idea for Constructing a Policy: Move Backwards Through Time. (Pretend you are “All-Knowing”)

19 Outline –Three apparently dissimilar problems –Myopic decision making –Constructing policies –Challenges Unknown, unobserved causes (e.g. how data might mislead you) Small, expensive data sets –Discussion

20 Artificial Intelligence Scientists who construct policies in autonomous helicopter flight use mechanistic theory (physical laws: momentum=m*v, W=F*d*cos(θ)…) to model the interrelationships between observations and how the actions might impact. –Scientists know many (most?) of the causes of the observations and know how the observations relate to one another. Scientists can quickly evaluate policies for selecting the actions (within a matter of months).

21 Behavioral/Social/Medical Sciences Incomplete mechanistic models Unknown causes Use data on individuals to combat the dearth of mechanistic models. –Drawback: non-causal “associations” occur due to the unknown causes of the observations. We must use the same data to construct a policy and to provide an evaluation of that policy

22 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

23 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

24 Unknown, Unobserved Causes (Incomplete Mechanistic Models) Problem: Non-causal associations between treatment (here counseling) and rewards are likely. A solution: Construct policies using data sets in which randomization is used to assign treatments to students. This breaks the non-causal associations yet permits causal associations.

25 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

26 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

27 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

28 The problem: Even when treatments are randomized, non-causal associations occur in the data. The solution: Methods for constructing policies must be conducted at each time point as opposed to “all-at-once.” Statistical methods should appropriately “average” over the non-causal associations between treatment and reward. Unknown, Unobserved Causes (Incomplete Mechanistic Models)

29 Review of Instrumental Variables

30 Review of Instrumental Variables Stage 1: Estimate mean of Z by

31 Review of Instrumental Variables Stage 1: Estimate α’s in E[Z|V]=α 0 + α 1 V

32 Stage 2: Solve for in Time 2: X 2 acts as its own instrument Stage 1: Estimate α’s in logistic regression E[X 2 |O 1,X 1 ]= expit(α 0 + α 1 O 1 +α 2 X 1 )

33 Construct Time 1 Response Variable: Output of Time 2 Analysis Time 2 Decision Rule: Select treatment X 2 =1 if otherwise select treatment X 2 =0.

34 Stage 2: Solve for in Time 1: X 1 acts as its own instrument Stage 1: Estimate α’s in logistic regression, E[X 1 |O 0 ]= expit(α 0 + α 1 O 0 )

35 Resulting (Simple) Policy Time 2 decision rule: Select treatment X 2 =1 if otherwise select treatment X 2 =0. Time 1 decision rule: Select treatment X 1 =1 if otherwise select treatment X 1 =0. In reality decision rules are most likely more complex.

36 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

37 Unknown, Unobserved Causes Problem: We recruit students via flyers posted in dormitories. Associations between observations and rewards are highly likely to be (due to the unknown causes) non- representative. Solution: Sample a representative group of college students.

38 Summary of Solutions To Causal Problems If possible randomize treatments (e.g. actions). Develop methods that avoid being influenced by non-causal associations yet help you construct the policy. Subjects in your data should be representative of population of subjects.

39 Outline –Three apparently dissimilar problems –Myopic decision making –Constructing policies –Challenges Unknown, unobserved causes Small, expensive data sets –Discussion

40 Expensive Data on a Limited Number of Individuals Scientists who want to use data on individuals to construct treatment policies must provide measures of confidence and also evaluations of alternative treatment policies. Above is challenging because methods for constructing policies are non-smooth.

41 Stage 2: Solve for in Time 1: X 1 acts as its own instrument Stage 1: Estimate α’s in logistic regression, E[X 1 |O 0 ]= expit(α 0 + α 1 O 0 ) Form

42 Expensive, Limited Data on Individuals In order to provide measures of confidence and comparisons of policies, the statistical methods for constructing policies must be regularized. Ideas include replacing the max by a “soft- max” or by the use of thresholding.

43 Outline –Three apparently dissimilar problems –Myopic decision making –Constructing policies –Challenges Unknown, unobserved causes Small, expensive data sets –Experiments & Discussion

44 ExTENd Ongoing study at U. Pennsylvania (D. Oslin) Goal is to learn how best to help alcohol dependent individuals reduce alcohol consumption.

45 Oslin ExTENd Late Trigger for Nonresponse 8 wks Response TDM + Naltrexone CBI Random assignment: CBI +Naltrexone Nonresponse Early Trigger for Nonresponse Random assignment: Naltrexone 8 wks Response Random assignment: CBI +Naltrexone CBI TDM + Naltrexone Naltrexone Nonresponse

46 Adaptive Treatment for ADHD Ongoing study at the State U. of NY at Buffalo (B. Pelham) Goal is to learn how best to help children with ADHD improve functioning at home and school.

47 ADHD Study B. Begin low dose medication 8 weeks Assess- Adequate response? B1. Continue, reassess monthly; randomize if deteriorate B2. Increase dose of medication with monthly changes as needed Random assignment: B3. Add behavioral treatment; medication dose remains stable but intensity of bemod may increase with adaptive modifications based on impairment No A. Begin low-intensity behavior modification 8 weeks Assess- Adequate response? A1. Continue, reassess monthly; randomize if deteriorate A2. Add medication; bemod remains stable but medication dose may vary Random assignment: A3. Increase intensity of bemod with adaptive modifi- cations based on impairment Yes No Random assignment:

48 Studies under review H. Jones study of drug-addicted pregnant women (goal is to reduce cocaine/heroin use during pregnancy and thereby improve neonatal outcomes) J. Sacks study of parolees with substance abuse disorders (goal is reduce recidivism and substance use)

49 Discussion The best management of chronic disorders (poverty, mental illness, other medical conditions) requires multi-stage decision making. Avoid myopic decision making! –Allow for longer term effects of the treatment –When comparing treatment options take into account the effect of future treatments –Appreciate the value of observing patients outcomes such as adherence Basic experimental designs and statistical methods are available.

50 This seminar can be found at: seminars/MethPolSci07.08.ppt me with questions or if you would like a copy:

51 Jones’ Study for Drug-Addicted Pregnant Women rRBT 2 wks Response rRBT tRBT Random assignment: rRBT Nonresponse tRBT Random assignment: aRBT 2 wks Response Random assignment: eRBT tRBT rRBT Nonresponse

52 Sack’s Study of Adaptive Transitional Case Management Standard Services Standard TCM Random assignment: 4 wks Response Standard TCM Augmented TCM Standard TCM Nonresponse

53 STAR*D This trial is over and the data is being analyzed (PI: J. Rush). One goal of the trial is construct good treatment sequences for patients suffering from treatment resistant depression.

54