Two Year Outcomes Analysis for Persons Served Places for People Nathan Dell, MSW; Gary Morse, Ph.D. and Allison Murphy, MSW August 13, 2018

Backstory Agency had long history of in-depth effectiveness evaluations of new programs but lacked agency-wide assessment of outcomes for persons served A number of benefits are possible from an agency-wide outcome monitoring system, including: Informs management about service and program outcomes Can help to identify high-risk/high-need groups and individuals Can be integrated with treatment planning Demonstrates service client outcomes accountability to payers Improves marketability within health care system dynamics PfP designed a comprehensive outcome system tailored for community-based services to people with severe mental illness (and co-occurring disorders) Outcome system initiated starting January 2014

Backstory (continued) Information is collected in 11 core domains (e.g., housing, employment, health, mental health, substance abuse, hospitalizations, etc.) Outcome system is integrated within EMR Sources of data: Persons served, staff informants, clinician ratings Data is collected at “baseline” and every three months (comprehensiveness of assessments vary by time period) throughout time of services Data has grown exponentially: Now 12,684 assessments (including 1650 “baselines,” 887 12 months, etc.) Previously (12/16) provided Board report for 12 month outcomes Today’s question and report: For persons receiving 24 months of services, what are the mental health, substance abuse, and psychosocial outcomes associated with treatment?

Analytic Method 1551 outcome packets completed between 01 Jan 2014 and 31 Dec 2017 Of 1000 unique clients, approximately 400 had completed baseline, 6, 12, 18, and 24 month outcomes Who is left out of this analysis? Clients who graduated or were terminated prior to 24 months Clients who were enrolled less than 24 months Clients who teams did not complete all outcomes packets How was data analyzed? Descriptive statistics to summarize data Paired samples to look at significant changes over time Statistical significance was set at p=.05. Clinical significance is open to interpretation

Demographic Information Based on baseline data

Demographic Information Race African-American , 65.4% (n=1014) white, 30.0,% (n=465) Other, 4.6% (n=72) Age M=46.25, SD=12.43 Min = 19, Max = 84 Gender Male, 57.8% (n=895) Female, 42.2% (n=656) Transgender, not reported, although transgender clients have made up 2-3% of HHH and PBHCI evaluation projects

Outcome Assessment “Baseline” to 24 Months

Employment Employed Baseline 6 Months 12 Months 18 Months 24 Months Yes  8%  9%  13%  12%  No  92%  91%  87%  88%  Percent employed over 2 years (n=404) Significant changes in employment status over a two year period for the 404 persons with complete data (χ(4)=16.20, p=.003, r=.806). Significant increases in employment were observed from baseline to 18 months (p=.015) and from baseline to 24 months (p=.045). 

Housing Stability

Housing Stability for Clients Homeless at Baseline

Housing Stability for Clients Homeless at Baseline  For clients homeless at baseline, significant changes in housings stability were observed (F(8, 34)=7.431, p<.001, η2=.636).  Housing Type  Baseline  6 Months  12 Months  18 Months  24 Months  F  η2  Homeless  43.52 (33.13)  12.81 (25.79)  7.71 (23.26)  6.71 (19.72)  3.00 (16.62)  29.91***  .422  Institutional  7.38 (12.00)  5.33 (16.62)  4.19 (14.63)  7.19 (21.23)  7.05 (23.43)  .361  .009  Housed  39.10 (32.23)  71.86 (29.37)  78.10 (27.24)  76.10 (29.89)  79.95 (27.38)  20.69***  .335  Housing Stability for Persons with Any Nights homeless at Baseline (n=42); ***p<.001 

Housing Stability Overall, significant changes in housing stability were observed (F(8, 365)=4.051, p<.001, η2=.082).  Housing Type Baseline 6MO 12MO 18MO 24MO F η2 Homeless 4.90 (17.63) 2.21 (11.20) 2.59 (12.96) 1.74 (9.95) 1.39 (8.68) 6.590**  .017  Institutional 6.11 (16.11) 4.72 (15.67) 3.74 (13.74) 4.41 (16.25) 5.84 (19.57) 2.498 .007  Housed 78.99 (23.67) 83.07 (19.49) 83.67 (19.37) 83.85 (19.22) 82.78 (21.72) 5.940**  .016  Housing Stability over 24 Months (n=373) **p<.01

Substance Use

Changes in Substance Use Severity For the entire sample, severity of alcohol use and severity of drug use did not change significantly over time (p>.05) (n=318). For persons with substance use disorders, significant changes were observed over time.   B  6  12  18  24  F(df)  η2  SATS (n=248) 4.48 (2.54)  5.35 (2.76)  5.14 (2.82)  5.08 (2.80)  5.31 (2.69)  7.305*** (3.86, 953.97)   .029  AUS (n=86) 3.49 (.61)  2.88 (1.00)  2.91 (1.11)  2.85 (1.10)  2.66 (1.15)  16.531*** (4,340)  .163  DUS (n=85) 3.41 (.54)  2.58 (1.21)  2.33 (1.11)  2.26 (1.17)  2.39 (1.15)  25.08*** (4, 336)  .230  Changes in severity of substance use for persons with use disorders. ***p<.001

Changes in Substance Use Severity

Mental Health Measures

Depression Depression decreased significantly overall and most strongly for people clinically depressed at baseline Decreases in depression plateau after six months   B  6  12  18  24  F(df)  η2  Overall (n=286) 9.44 (6.87)  7.83 (6.16)  7.87 (6.03)  7.72 (6.79)  7.72 (6.81)  7.73***  (4, 1140)  .026  Depressed at Baseline (n=128) 15.87 (4.51)  10.77 (6.03)  10.83 (6.29)  11.04 (7.06)  10.93 (6.90)  26.65***  (4, 508) .173  Changes in severity of depression. ***p<.001

Brief Psychiatric Rating Subscales   Baseline  12 Months  24 Months  Subscale  M (SD)  F  η2  Anxiety/Depression  12.67 (5.71)  11.74 (4.97)  10.87 (4.92)  16.298***  1.91, 529.89 .056  Withdrawal  10.23 (3.40)  9.83 (3.66)  9.50 (3.92)  4.117* 1.95, 540.28 .015  Thought  8.84 (4.94)  8.14 (4.24)  7.86 (4.20)  6.780**  1.87, 518.39  .024  Hostility  5.76 (3.15)  5.43 (2.89)  5.33 (2.89)  2.513 1.89, 524.32 .009  Activity  8.37 (3.96)  8.07 (3.67)  7.70 (3.22)  3.639* 2,554  .013  BPRS Subscale change over time (n=278); *p<.05; **p<.01; ***p<.001 

General Health Measures

Self-Rated Health, Primary Care Linkage, Tobacco Use 6  12  18  24  Health Status  F 2.59 (1.05)  2.61 (.861)  2.58 (.93)  2.54 (.93)  2.51 (.91)  1.18  Linkage to Primary Care χ Yes  75%  83%  84%  89%  66.70*** No  25%  17%  16%  11%  Smoking Status  ns 76%  74%  24%  26% 

Recovery Orientation

Happiness & Caring People Caring, supportive relationships are an essential vehicle for the recovery journey Happiness ranged from “not very happy” (1) to “very happy” (3). Both happiness and number of caring people did not change significantly over time. Caring folks  Baseline  12 Months  24 Months  No  7.9% (121)  4.5% (36)  4.7% (29)  Yes  91.5% (1419)  95.5 (760)  95.3 (588)  Caring folks are not only PfP Staff  96.2% (1365)  97.2% (739)  96.9% (570)  Happiness  (n=418) 1.86 (.68)  1.94 (.69)  1.89 (.66) 

Recovery Assessment Scale  A significant increase in overall recovery orientation was observed (F(2,554)=5.697, p=.004, η2=.020), as well as in specific sub-scales.   B  12  24  F(d η2  Not Dominated by Symptoms 9.64 (2.81  10.11 (2.71) 10.34 (2.55) 7.983***  .0297 Hope and Personal Confidence 33.54 (5.88) 34.39 (5.34) 34.39 (5.01) 5.077** .017 Asking for Help 12.17 (1.98)  12.37 (1.84)  12.16 (1.97)  1.513 .005 Goal-Orientation 19.98 (3.34)  20.18 (3.22)  19.99 (3.08)  .685 .002 Reliance on others 15.46 (2.92)  15.96 (2.62)  15.72 (2.57)  164.279* .014  Changes in recovery orientation. ***p<.001, **p<.01, *p<.05. (n=294)

Client Satisfaction Conduct ongoing random sample surveys of persons served Interviews conducted by non-service provider/peer research assistant

Consumer Satisfaction Survey Clients could score between 10 and 60 points, with the higher score indicating higher satisfaction Actual scores ranged from 26 to 60, mean of 54.85 (SD=7.01) and a median of 58

Consumer Satisfaction Survey (continued) Consumer Satisfaction Survey Question Mean (SD) Median I am not satisfied with the amount of services I received 5.10 (1.57) Strongly Disagree The services I received will help me to deal more effectively with my problems 5.32 (1.29) Strongly Agree The staff treated me with respect and dignity 5.45 (1.18) I feel I have a right to approve the services I receive 5.46 (1.11) I was able to get the services I thought I needed 5.48 (0.81) I want to remain in this program for the time being 5.50 (1.27) Overall, I am satisfied with the services I received 5.53 (1.00) Staff and services were sensitive to my personal needs and preferences 5.58 (0.90) I liked the services I received 5.62 (0.75) I would not recommend this agency to a friend or family member 5.76 (0.91) Consumer Satisfaction Scale items (lowest to highest; scores range from 1 to 6; n=124)

Consumer Satisfaction Survey (continued): Perception of Care Clients could score between 14 and 70 points, with the higher score indicating higher satisfaction. Actual scores ranged from 27 to 70 points; mean is 58.75 (SD=8.57), median of 59 Perception of Care Survey Question Mean (SD) Median Staff were sensitive to my cultural background. 4.05 (0.90) Agree Staff told me what side effects to watch out for 4.07 (0.83) Staff respected my wishes about who is and who is not to be given information about my treatment 4.09 (0.99) I, not staff, decided my treatment goals 4.10 (0.78) I feel free to complain. 4.16 (0.77) I was given information about my rights. 4.17 (0.66) I was encouraged to use consumer run programs (e.g., support groups, drop-in centers) 4.18 (0.76)

Consumer Satisfaction Survey (continued): Perception of Care Perception of Care Survey Questions Mean (SD) Median If I had other choices, I would still get services from this agency. 4.20 (0.85) Agree Staff encouraged me to take responsibility for how I live my life 4.24 (0.85) I would recommend this agency to a friend or family member 4.24 (0.80) Staff helped me obtain the information I needed so that I could take charge of managing my illness 4.26 (0.92) I like the services I received here 4.30 (0.82) Staff here believe that I can grow, change, and recover 4.34 (0.77) I felt comfortable asking questions about my treatment and medication 4.34 (0.74) Perception of Care Scale items, ranked lowest to highest; Scores range from 1 to 5

Future Directions Fine-tuning enhancements of outcome measures Additional analyses Over longer time intervals and with bigger samples Subgroup analyses Outcomes by program Additional efforts to increase data to inform clinical service practices Possible further development/joint venture with EMR or other possible partners to expand and market outcome tools  

Comments and Questions? Thank you!