1. Total Clinical Outcomes Management: Theory and Method Richard A. Epstein, Ph.D., M.P.H. Michael J. Cull, Ph.D., M.S.N. Department of Psychiatry

2. Road Map The importance of shared vision Managing information asymmetry The role of standardized assessment Implementation examples 2

3. 3 “To organize the world’s information and make it universally accessible and useful.”

4. “Our investment in data-mining is part of our drive to deliver what our customers want: the item, at the right store, at the right time, at the right price.” 4 “To organize the world’s information and make it universally accessible and useful.”

5. “Our investment in data-mining is part of our drive to deliver what our customers want: the item, at the right store, at the right time, at the right price.” 5 “The behavioral health system, at all levels, should always make decisions based on the needs and well-being of the people served.” “To organize the world’s information and make it universally accessible and useful.”

6. 6 Step 1: Sit down Step 2: Hold on for 8 seconds

7. 7 Landing a passenger jet on the Hudson

8. 8 A Successful Landing Capt. Sullenberger credited teamwork, preparation and strict adherence to protocols for the successful landing.

9. 9 Success in Health Care

10. 10 International Comparison, 2004 Source: Commonwealth Fund National Scorecard on U.S. Health System Performance, 2008 Infant Mortality Rate Deaths per 1,000 births

11. 11 Ambulatory Care Visits for Adverse Drug Effects Visits per 1,000 population per year Source: Commonwealth Fund National Scorecard on U.S. Health System Performance, 2008 By Gender, Race, and Region, 2004Annual Averages, by Care Setting

12. 12

13. 13 Health Care Failures 1.Ignorance – science has given us incomplete information 2.Ineptitude – science has provided a solution, we fail to apply it correctly Atul Gwande, 2009

14. 14 Ignorance Evidence-based practice Practice-based evidence

15. 15 Ineptitude Organizing information Standardize practice Support rational decision making

16. The cost of ineptitude Poor Quality Inefficiency Poor resource allocation Excess costs 16

17. Combating ineptitude Bias Complexity Goal incongruence 17

18. Bias Stereotyping Moral hazard Treatment Framing Probability Confirmation bias 18

19. Complexity Co-morbidity Treatment options Governance and financing 19

20. Goal Incongruence Multiple perspectives Information asymmetry Motive/”agenda” 20

21. The Case for Decision Support Eliminates bias Manages complexity Supports creation of a shared vision 21

22. Shared Vision: Child/Family Child and family centric Strengths based Culturally and contextually sensitive 22

23. “12 Angry Men” Video Clip 23

24. “12 Angry Men” Multiple perspectives? Group think? Strong-arming? Disagreement? Bias? How will they discuss their disagreement? Can they reach a shared vision? 24

25. 25 Managing Complexity: Aviation Checklists developed in 1935 to manage complexities of the Boeing model 299 long-range bomber In 1968 NTSB recommended a re-evaluation of checklists after the crash of an American World Airways B-707 Pre-flight checklists are now “standard of care”

26. 26 Reducing Bias & Complexity: Medicine Critical care – Pronovost et al. (2003) – Use of “daily goals” checklists associated with improved communication and reduced length of ICU stay. Catheter related infections – Pronovost et al. (2006) – Use of checklists and team empowerment associated with 66% reduction in catheter- related infections. Surgery – Makary et al. (2007) – Use of operating room briefing tool improved team communication and reduced risk of “wrong- side” surgeries – Haynes et al. (2009) – Use of checklists associated with reduced risk of death and other surgery outcomes

27. 27 Example: Surgical Safety Surgery is integral to global health – 234 million operations each year Many complications are preventable Evaluated the effectiveness of improving team communication on outcomes

28. 28 Hospitals Toronto, Canada New Delhi, India Amman, Jordan Auckland, New Zealand Manila, Philippines Ifakara, Tanzania Long, England Seattle, Washington

29. 29 Intervention Measured outcomes before and after intervention Intervention was a 19-point checklist Checklist had three parts: – Sign in – Time out – Sign out

30. 30 Sign In Before anesthesia, team verbally confirms: – Verification of patient identity, surgical site, procedure, and consent – Surgical site is appropriately marked – Pulse oximeter is on the patient and functioning – All team members are aware of patient allergies – Patient risk of airway obstruction and aspiration has been evaluated and appropriate equipment is available – Blood and fluids are available

31. 31 Time out Before incision, team verbally confirms: – All team members are introduced by name and role – Patient identity, surgical site, and procedure – Review of anticipated critical events – Administration of prophylactic antibiotics before incision or that they are not indicated – Essential imaging results are displayed

32. 32 Sign Out Before patient leaves the OR: – Nurse reviews aloud with the team: Name of the procedure as recorded That needle, sponge, and instrument counts are complete That specimen is correctly labeled Whether there are any equipment issues – Team reviews aloud key concerns for recovery and patient care

33. 33 Complications BeforeAfterp value Surgical site infection 6.23.4<.001 Death 1.50.8<.01 Process Measures Prophylactic antibiotics given 56.182.6<.001 Orally confirm patient’s identity 54.492.3<.001 Results

34. 34 Summary Introducing surgical checklist: – Reduced surgical site infections and death by ½ – Increased compliance with safety process measures What did the checklist accomplish? – Ensured similar information for the team – Improved team communication – Supported shared vision

35. Break 35

36. Road Map The importance of shared vision Managing information asymmetry The role of standardized assessment Implementation examples 36

37. 37 Data-driven decision making

38. Service planning exercise Case vignette System-level service plan Importance of measurement and communication 38

39. 39 “Bender” Video Clip 1

40. 40 Identify 3 service plan needs

41. “Bender” Video Clip 2 41

42. Okay, now what? 42

43. 43 Measurement as Communication Purpose of all measurement is to communicate – Time (e.g., days), Temperature (e.g., F o, C o ), Value (e.g., dollars) Physical sciences – Concern phenomena accessible via instrumentation – Instrumentation improves; measurement stays the same Human and social sciences – Sometimes concern phenomena similarly measured – Many times involves phenomena not readily accessible

44. 44 Social Science Measurement Logical empiricist perspective – Measurement is application of procedure to show verifiable truth from observation – Implication is that some real phenomenon exists and that this “truth” can be revealed Classical test and item response theories are based on a similar philosophies

45. 45 Communication Theory Traditional perspective also empiricist – Knowledge transfers from one mind to another Main critique of the “transfer” model – Overly linear – A  B with no room for impact of B on A and information coming from A Constitutive / constructivist model – Communication as shared meaning making

46. 46 Shared Meaning Making

47. 47 Communication and Measurement From a communication / constructivist perspective, measurement is: – Less about “revealing truth” – More about “creating shared meaning” Classical test theory – Multiple items; internal consistency reliability

48. 48 Internal Consistency Reliability Cronbach’s alpha is the statistic used to define internal consistency reliability The equation is as follows:

49. 49 Classical Test and Communication Theories Complex systems require clear and concise communication about multiple constructs Classical test theory: reliability = redundancy Inter-rater reliability is therefore both the – Relevant measure of reliability – Concept most related to shared meaning – Directly facilitates decision support

50. Reduce bias Integrate multiple perspectives Attend to all relevant information Communicate and use information at all system levels Measurement Helps 50

51. 51 TCOM Grid of Activities Family & Youth ProgramSystem Decision Support Service Planning EligibilityResource Management Quality Improvement Case Management & Supervision Accreditation CQI Transformation System Evolution Outcome Monitoring Service Transitions & Celebrations EvaluationPerformance Contracting

52. CANS Assessment Interactive Training Environment Agency Rater Groups System Analytics Trained Rater Child and Family Using assessment data to support decision-making rater data creates training opportunities 52

53. 53 Tennessee DCS Implementation Case managers are annually trained to reliability Children are assessed at several points during a custody episode – Entry and discharge – Placement transitions and regular intervals Support is provided by network of Centers of Excellence (COEs) – Training, consultation, independent 3 rd party review – Data management and analytics Data used to: – Monitor child outcomes; Support placement decisions – Inform treatment plans; Plan array of services

54. 54 Training: January 2008 – April 2009 Training% Completed Initial100 Recertification95 PBC Agencies85 Initial and recertification trainings are ongoing Increasingly use CANS for decision support Initiated provider trainings and “Guide Program”

55. 55 Assessment Volume: July - April 2009 Month No. of completed assessments July 1185 August 1310 September 1219 October 1349 November 1127 December 1230 January 1642 February 1560 March 1607 Total12,229

56. 56 Algorithm Services Intensity Recommendation: As of April 2009 (Initial assessments = 5119) Services intensity recommendation FrequencyPercent 1239146.7 2163231.9 384016.4 42565.0

57. 57 Secondary data utilization Review of high-risk children Placement quality team review Regional utilization review support Well being team meetings Unified Assessment process Assessment of Services Quality (ASQ)

58. 58 Special projects Restructuring the continuum model project (DCS) Transition age youth (GOCCC) Ad-hoc region level reports (DCS) Surveillance of foster care youth with scores of ‘2’ or ‘3’ on sexual aggression, sexually reactive, and danger to others items (TAC) TFACTS placement matching project (DCS) Substance abuse (T-ACT, GOCCC) Sexually problematic behaviors (SPB Task Force, GOCCC) OJJDP juvenile court screening pilot

59. 59 Characteristics of DCS children, age 5 years or more, assessed between 2/18/08 – 2/31/09 Demographics#% Age in years 5 – 12266329.8 13 – 16379042.5 17 – 19247327.7 Race / ethnicity African-American273730.7 Non-Hispanic White569963.8 Other2412.7 Gender Female372341.7 Male520358.3 Adjudication Status Dependent / Neglect / Unruly516857.9 Delinquent207423.2 3 regions with the most assessments East Tennessee143316.1 Shelby101011.3 Mid Cumberland99411.1 *Totals may not add up to 100% due to missing data

60. 60 Three most prevalent Risk Behaviors by age Actionable Risk Behaviors5 – 1213 – 1617 – 19 Suicide1.34.42.5 Self Mutilation1.53.61.7 Other Self Harm2.07.95.9 Danger to Others5.114.08.5 Runaway0.813.415.7 Firesetting0.60.90.3 Social Behavior14.837.827.8 Sexually Reactive Behavior7.05.73.5 Sexual Aggression1.64.12.3 Delinquency2.228.424.2 Substance Use0.214.418.1

61. 61 Age at which each risk behavior is most prevalent Actionable Risk Behaviors5 – 1213 – 1617 – 19 Suicide1.34.42.5 Self Mutilation1.53.61.7 Other Self Harm2.07.95.9 Danger to Others5.114.08.5 Runaway0.813.415.7 Firesetting0.60.90.3 Social Behavior14.837.827.8 Sexually Reactive Behavior7.05.73.5 Sexual Aggression1.64.12.3 Delinquency2.228.424.2 Substance Use0.214.418.1

62. 62 Three most prevalent Behav. / Emo. Needs by age Actionable Behavioral and Emotional Needs 5 – 1213 – 1617 – 19 Psychosis1.12.11.7 Impulsivity21.326.518.6 Depression4.817.513.8 Anxiety6.711.17.6 Oppositional6.221.315.7 Adjustment to Trauma12.514.37.9 Attachment4.44.82.8 Anger Control12.432.622.2 Emotional Control10.523.216.0

63. 63 Age group in which each Behav. / Emo. Need is most prevalent Actionable Behavioral and Emotional Needs 5 – 1213 – 1617 – 19 Psychosis1.12.11.7 Impulsivity21.326.518.6 Depression4.817.513.8 Anxiety6.711.17.6 Oppositional6.221.315.7 Adjustment to Trauma12.514.37.9 Attachment4.44.82.8 Anger Control12.432.622.2 Emotional Control10.523.216.0

64. 64 Difficulties with adjustment to trauma by SPB category Percentage

65. 65 Hinge analysis of outcome trajectories prior to and after program initiation

66. 66 Figure 5.2 Survival analysis of time to placement disruption for children/youth whose placement matches CANS recommendations (Match=0), those whose placed is at a lower intensity than recommended (match=1) and those whose placement is more intensive than recommended (match=-1).

67. 67 Modeling crisis decisions (Leon et al., 1999) A 5 item model predicted 78% of crisis decisions Items - Suicide Risk, Danger to Others, Impulsivity, Emotional Disturbance, Behavioral Disturbance Replicated by He et al. (2004) Predicted, # (%) Observed, # (%)HospitalizeDeflectTotal Hospitalize667 (70) 285 (30) 952 (43) Deflect203 (16) 1050 (84) 1253 (57) Total870 (39) 1335 (61) 2205 (100)

68. 68 Illinois Department of Children and Family Services DCFS Wards

69. 69 Mental Health Services Illinois Department of Children and Family Services

70. 70 Mental Health Services DCFS Wards Illinois Department of Children and Family Services

71. 71 Figure 5.1 Proportion of low-risk youth hospitalized by race over the duration of an decision support initiative

72. 72 Change in Total CSPI Score by Intervention and Hospitalization Risk Level (FY06)

73. 73 Summary TCOM can help systems: – Reduce bias – Integrate multiple perspectives/shared vision – Attend to all relevant needs and strengths – Communicate and use information at all system levels – Allocate resources and measure/improve outcomes

74. 74 Selected references 1.Gawande A. (2010). The Checklist Manifesto: How to Get Things Right. New York: Metropolitan Books. 2.The Commonwealth Fund National Scorecard on U.S. Health System Performance. (2008). 3.U.S. Government Accountability Office (http://www.gao.gov).http://www.gao.gov 4.Lyons JS. (2004). Redressing the Emperor: Improving Our Children’s Public Mental Health System. Westport, CT: Praeger. 5.Lyons JS & Weiner DA. (2009). Behavioral Healthcare: Assessment, Service Planning, and Total Clinical Outcomes Management. Kingston, NJ: Civic Research Institute. 6.Lyons JS. (2010). Communimetrics: A Communication Theory of Measurement in Human Service Settings. New York: Springer. 7.Lyons JS. (2008). Total Clinical Outcomes Management. Presentation at the Annual Child and Adolescent Needs and Strengths (CANS) Conference. Nashville, TN. 8.Lyons JS. (2009). Total Clinical Outcomes Management in the service of children with behavioral and emotional needs: an update. Presentation to the Tennessee Council on Children’s Mental Health. Nashville, TN. 9.Shiv B & Fedorikhin A. (1999). Heart and mind in conflict: the interplay of affect and cognition in consumer decision making. Journal of Consumer Research, 26(3), 278-292. 10.Pronovost P, Berenholtz S, Dorman T, et al. (2003). Improving communication in the ICU using daily goals. Journal of Critical Care, 18, 71-75. 11.Rawal PH, Anderson TR, Romansky JR & Lyons JS. (2008). Using decision support to address racial disparities in mental health service utilization. Residential Treatment for Children and Youth, 25(1), 73-84. 12.Pronovost P, Needham D, Berenholtz S., et al. (2006). An intervention to decrease catheter-related bloodstream infections in the ICU. New England Journal of Medicine, 355, 2725-2732. 13.Makary MA, Mukherjee A, Sexton B, et al. (2007). Operating room briefings and wrong-site surgery. Journal of the American College of Surgeons, 204(2), 236-243. 14.Haynes AB, Weiser TG, Berry WR et al. (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine, 360(5), 491-499. 15.Keysar B, Barr DJ, Balin JA & Brauner JS. (2000). Taking perspective in conversation: the role of mutual knowledge in comprehension. Psychological Science, 11(1), 32-38. 16.Video clips are from: “12 Angry Men” and “The Breakfast Club” 17.Epstein RA. (2008). Needs and strengths of children entering state custody. Presentation at the Annual Child and Adolescent Needs and Strengths (CANS) Conference. Nashville, TN. 18.Epstein RA & Cull M. (2009). Tennessee Department of Children’s Services CANS Implementation. Presentation to the Tennessee Council on Children’s Mental Health. Nashville, TN. 19.Leon SC, Lyons JS, Uziel-Miller ND, Tracy P. (1999). Psychiatric hospital utilization of children and adolescents in state custody. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 305-310.

75. Questions Richard A. Epstein, Ph.D., M.P.H. richard.a.epstein@vanderbilt.edu Michael J. Cull, Ph.D., M.S.N. michael.cull@vanderbilt.edu 75