1. 1 Meeting the Future in Managing Chronic Disorders: Individually Tailored Strategies S.A. Murphy Univ. of Michigan Oberlin College, Feb. 20, 2006

2. 2 Outline –Three apparently dissimilar problems –Myopic decision making –Unknown, unobserved causes –Discussion

3. 3 Three Apparently Dissimilar Problems –Artificial Intelligence: Autonomous Helicopter Flight –Management of Chronic Mental Illnesses –Management of a Welfare Program

4. 4 Artificial Intelligence Autonomous Helicopter Flight –Observations: characteristics of the helicopter (position, orientation, velocity, angular velocity, ….), characteristics of the environment (wind speed, wind angle, turbulence….) –Actions/treatments: cyclic pitch (causes forward/backward and sideways acceleration), tilt angle of main rotor blades (direction), tail rotor pitch control (turning) –Rewards: Closeness of helicopter’s flight path to the desired path; avoidance of crashes(!)

5. 5 Andrew Ng’s Helicopter: http://ai.stanford.edu/~ang/

6. 6 The Management of Chronic Mental Illnesses Treating Patients with Opioid Dependence (heroin) –Observations: characteristics of the individual (withdrawal symptoms, craving, attendance at counseling sessions, results of urine tests….), characteristics of the environment (housing, employment.…) –Actions/treatments: methadone dose, amount of weekly group counseling sessions, daily dosing time of methadone, individual counseling sessions, methadone taper –Rewards: minimizing opioid use and maximizing health, minimizing cost

7. 7 http://www.nida.nih.gov/perspectives/vol1no1.html

8. 8 Management of a Welfare Program “Jobs First” Program in Connecticut –Observations: characteristics of the individual (assets, income, age, health, employment), characteristics of the environment (domestic violence, incapacitated family member, # children, living arrangement…) –Actions/treatments: child care, job search skills training, amount of cash benefit, medical assistance, education –Rewards: maximizing employment/independence.

9. 9

10. 10 The Common Thread: Sequential Decision Making Observation, action, observation, action, observation, action,……………………. A strategy tells us how to use the observations to choose the actions. We’d like to develop strategies that maximize the rewards.

11. 11 Role of the Statistician What kinds of data are most useful for developing strategies? How do we design an experiment that will produce the most useful data? How do we use the data to construct good strategies? (A strategy tells us how to use the observations to choose the actions.)

12. 12 Myopic Decision Making

13. 13 Myopic Decision Making In myopic decision making, decision makers use strategies that seek to maximize immediate rewards. Problems: –Longer term consequences of present actions. –Ignore the range of feasible future actions/treatments (A strategy tells us how to use the observations to choose the actions.)

14. 14 Autonomous Helicopter Flight The helicopter has veered from flight plan. Myopic action: Choose an acceleration and direction that will ASAP bring us back to the flight plan. The result: The myopic action results in the helicopter overshooting the planned flight path and in drastic situations may lead to the helicopter cycling out of control. The mistake: We did not consider the range of actions we can take following the initial action. The ability to slow down is mechanically limited. The message: Use an acceleration that will not quickly return us to the planned flight path but will take into account the ability of the helicopter to slow down and reduce the overshoot.

15. 15 Myopic Decision Making The message: The fields of robotics and artificial intelligence teach us that we should try to construct strategies that are not myopic! –Pay attention to the longer term consequences of present actions. –Do not ignore the range of feasible future actions/treatments (A strategy tells us how to use the observations to choose the actions.)

16. 16 Treatment of Psychosis Myopic action: Choose a medication, say A, that reduces psychosis for as many people as possible. The result: Some patients are not helped and/or experience abnormal movements of the voluntary muscles (TDs). The class of subsequent medications is greatly reduced. The mistake: We should have taken into account the variety of treatments available to those for whom the first treatment is ineffective. The message: Use an initial medication that may not have as large a success rate but that will be less likely to cause TDs.

17. 17 Unknown, Unobserved Causes

18. 18 Artificial Intelligence Scientists who construct strategies that will be used for autonomous helicopter flights can use physical laws: momentum=m*v, W=F*d*cos(θ)……… Scientists know many (most?) of the causes of the observations and know how the observations relate to one another.

19. 19 Conceptual Structure in the Behavioral/Social/Medical Sciences

20. 20 Unknown, Unobserved Causes Scientists who want to use data on individuals to construct treatment strategies must confront the fact that non-causal “relationships” occur due to the unknown causes.

21. 21 Unknown, Unobserved Causes

22. 22 Unknown, Unobserved Causes

23. 23 Unknown, Unobserved Causes

24. 24 Unknown, Unobserved Causes Problem: Non-causal associations between observations and rewards are likely (due to the unknown causes). Solution: Construct strategies using data sets collected on representative students (representative of all college students).

25. 25 Unknown, Unobserved Causes

26. 26 Unknown, Unobserved Causes Problem: Non-causal associations between “treatments” and rewards are likely (due to the unknown causes). Solution: Construct strategies using data sets in which a coin was tossed in order to assign students to treatments. This breaks the non-causal associations yet permits causal associations.

27. 27 Unknown, Unobserved Causes

28. 28 Unknown, Unobserved Causes (Constructing Sequences of Treatment)

29. 29 Unknown, Unobserved Causes

30. 30 Unknown, Unobserved Causes

31. 31 Unknown, Unobserved Causes The problem: Even when treatments are randomized (flip coin to assign treatment) non- causal associations can occur in the data. The solution: Develop statistical and mathematical methods that construct strategies but are able to ignore the non-causal “associations” between treatment and reward.

32. 32 Summary of Solutions Subjects in your sample should be representative of population of subjects. Experiments should randomize actions. Use statistical methods that avoid being influenced by non-causal associations yet help you construct the strategy. Scientists in the fields of robotics and artificial intelligence should pay attention to our field!

33. 33 Some Experiments

34. 34 ExTENd Ongoing study at U. Pennsylvania (D. Oslin) Goal is to learn how best to help alcohol dependent individuals reduce alcohol consumption.

35. 35

36. 36 STAR*D (www.star-d.org) This trial is over and the data is being analyzed (J. Rush). One goal of the trial is construct good treatment sequences for patients suffering from treatment resistant depression.

37. 37

38. 38 Discussion When thinking how best to manage chronic disorders (poverty, mental illness, other medical conditions) we need to –Allow for longer term effects of the treatments –When comparing treatment options take into account the effect of future treatments –Use data and good statistical methods to develop the strategies.

39. 39 This seminar can be found at: http://www.stat.lsa.umich.edu/~samurphy/seminars/Oberlin 02-06.ppt Research samurphy@umich.edu