1. Two Year Outcomes Analysis for Persons Served Places for People Nathan Dell, MSW; Gary Morse, Ph.D. and Allison Murphy, MSW August 13, 2018

2. Backstory
Agency had long history of in-depth effectiveness evaluations of new programs but lacked agency-wide assessment of outcomes for persons served
A number of benefits are possible from an agency-wide outcome monitoring system, including:
Informs management about service and program outcomes
Can help to identify high-risk/high-need groups and individuals
Can be integrated with treatment planning
Demonstrates service client outcomes accountability to payers
Improves marketability within health care system dynamics
PfP designed a comprehensive outcome system tailored for community-based services to people with severe mental illness (and co-occurring disorders)
Outcome system initiated starting January 2014

3. Backstory (continued)
Information is collected in 11 core domains (e.g., housing, employment, health, mental health, substance abuse, hospitalizations, etc.)
Outcome system is integrated within EMR
Sources of data: Persons served, staff informants, clinician ratings
Data is collected at “baseline” and every three months (comprehensiveness of assessments vary by time period) throughout time of services
Data has grown exponentially: Now 12,684 assessments (including “baselines,” months, etc.)
Previously (12/16) provided Board report for 12 month outcomes
Today’s question and report: For persons receiving 24 months of services, what are the mental health, substance abuse, and psychosocial outcomes associated with treatment?

4. Analytic Method
1551 outcome packets completed between 01 Jan 2014 and 31 Dec 2017
Of 1000 unique clients, approximately 400 had completed baseline, 6, 12, 18, and 24 month outcomes
Who is left out of this analysis?
Clients who graduated or were terminated prior to 24 months
Clients who were enrolled less than 24 months
Clients who teams did not complete all outcomes packets
How was data analyzed?
Descriptive statistics to summarize data
Paired samples to look at significant changes over time
Statistical significance was set at p=.05. Clinical significance is open to interpretation

5. Demographic Information
Based on baseline data

6. Demographic Information
Race
African-American , 65.4% (n=1014)
white, 30.0,% (n=465)
Other, 4.6% (n=72)
Age
M=46.25, SD=12.43
Min = 19, Max = 84
Gender
Male, 57.8% (n=895)
Female, 42.2% (n=656)
Transgender, not reported, although transgender clients have made up 2-3% of HHH and PBHCI evaluation projects

7. Outcome Assessment “Baseline” to 24 Months

8. Employment Employed Baseline 6 Months 12 Months 18 Months 24 Months
Yes 
8% 
9% 
13% 
12% 
No 
92% 
91% 
87% 
88% 
Percent employed over 2 years (n=404)
Significant changes in employment status over a two year period for the 404 persons with complete data (χ(4)=16.20, p=.003, r=.806). Significant increases in employment were observed from baseline to 18 months (p=.015) and from baseline to 24 months (p=.045). 

9. Housing Stability

10. Housing Stability for Clients Homeless at Baseline

11. Housing Stability for Clients Homeless at Baseline
 For clients homeless at baseline, significant changes in housings stability were observed (F(8, 34)=7.431, p<.001, η2=.636). 
Housing Type 
Baseline 
6 Months 
12 Months 
18 Months 
24 Months  F 
η2 
Homeless 
43.52 (33.13) 
12.81 (25.79) 
7.71 (23.26) 
6.71
(19.72) 
3.00
(16.62) 
29.91*** 
.422 
Institutional 
7.38 (12.00) 
5.33 (16.62) 
4.19 (14.63) 
7.19
(21.23) 
7.05
(23.43) 
.361 
.009 
Housed 
39.10 (32.23) 
71.86 (29.37) 
78.10 (27.24) 
76.10
(29.89) 
79.95
(27.38) 
20.69*** 
.335 
Housing Stability for Persons with Any Nights homeless at Baseline (n=42); ***p<.001 

12. Housing Stability
Overall, significant changes in housing stability were observed (F(8, 365)=4.051, p<.001, η2=.082). 
Housing Type
Baseline
6MO
12MO
18MO
24MO F η2
Homeless
4.90
(17.63)
2.21
(11.20)
2.59
(12.96)
1.74
(9.95)
1.39
(8.68)
6.590** 
.017 
Institutional
6.11
(16.11)
4.72
(15.67)
3.74
(13.74)
4.41
(16.25)
5.84 (19.57)
2.498
.007 
Housed
78.99 (23.67)
83.07 (19.49)
83.67 (19.37)
83.85
(19.22)
82.78 (21.72)
5.940** 
.016 
Housing Stability over 24 Months (n=373) **p<.01

13. Substance Use

14. Changes in Substance Use Severity
For the entire sample, severity of alcohol use and severity of drug use did not change significantly over time (p>.05) (n=318).
For persons with substance use disorders, significant changes were observed over time. B  6 
12 
18 
24 
F(df) 
η2 
SATS (n=248)
4.48 (2.54) 
5.35 (2.76) 
5.14 (2.82) 
5.08 (2.80) 
5.31 (2.69) 
7.305***
(3.86, )  
.029 
AUS (n=86)
3.49 (.61) 
2.88 (1.00) 
2.91 (1.11) 
2.85 (1.10) 
2.66 (1.15) 
16.531*** (4,340) 
.163 
DUS (n=85)
3.41 (.54) 
2.58 (1.21) 
2.33 (1.11) 
2.26 (1.17) 
2.39 (1.15) 
25.08***
(4, 336) 
.230 
Changes in severity of substance use for persons with use disorders. ***p<.001

15. Changes in Substance Use Severity

16. Mental Health Measures

17. Depression
Depression decreased significantly overall and most strongly for people clinically depressed at baseline
Decreases in depression plateau after six months B  6 
12 
18 
24 
F(df) 
η2 
Overall (n=286)
9.44 (6.87) 
7.83
(6.16) 
7.87
(6.03) 
7.72
(6.79) 
7.72 (6.81) 
7.73*** 
(4, 1140) 
.026 
Depressed at Baseline (n=128)
15.87 (4.51) 
10.77 (6.03) 
10.83 (6.29) 
11.04
(7.06) 
10.93 (6.90) 
26.65*** 
(4, 508)
.173 
Changes in severity of depression. ***p<.001

18. Brief Psychiatric Rating Subscales
Baseline 
12 Months 
24 Months 
Subscale 
M (SD)  F 
η2 
Anxiety/Depression 
12.67 (5.71) 
11.74 (4.97) 
10.87 (4.92) 
16.298*** 
1.91,
.056 
Withdrawal 
10.23 (3.40) 
9.83 (3.66) 
9.50 (3.92) 
4.117*
1.95,
.015 
Thought 
8.84 (4.94) 
8.14 (4.24) 
7.86 (4.20) 
6.780** 
1.87,  
.024 
Hostility 
5.76 (3.15) 
5.43 (2.89) 
5.33 (2.89) 
2.513
1.89,
.009 
Activity 
8.37 (3.96) 
8.07 (3.67) 
7.70 (3.22) 
3.639*
2,554 
.013 
BPRS Subscale change over time (n=278); *p<.05; **p<.01; ***p<.001 

19. General Health Measures

20. Self-Rated Health, Primary Care Linkage, Tobacco Use6 
12 
18 
24 
Health Status  F
2.59 (1.05) 
2.61 (.861) 
2.58 (.93) 
2.54 (.93) 
2.51 (.91) 
1.18 
Linkage to Primary Care χ
Yes 
75% 
83% 
84% 
89% 
66.70***
No 
25% 
17% 
16% 
11% 
Smoking Status  ns
76% 
74% 
24% 
26% 

21. Recovery Orientation

22. Happiness & Caring People
Caring, supportive relationships are an essential vehicle for the recovery journey
Happiness ranged from “not very happy” (1) to “very happy” (3).
Both happiness and number of caring people did not change significantly over time.
Caring folks 
Baseline 
12 Months 
24 Months 
No 
7.9% (121) 
4.5% (36) 
4.7% (29) 
Yes 
91.5% (1419) 
95.5 (760) 
95.3 (588) 
Caring folks are not only PfP Staff 
96.2% (1365) 
97.2% (739) 
96.9% (570) 
Happiness  (n=418)
1.86 (.68) 
1.94 (.69) 
1.89 (.66) 

23. Recovery Assessment Scale
 A significant increase in overall recovery orientation was observed (F(2,554)=5.697, p=.004, η2=.020), as well as in specific sub-scales. B 
12 
24 
F(d
η2 
Not Dominated by Symptoms
9.64 (2.81 
10.11 (2.71)
10.34 (2.55)
7.983*** 
.0297
Hope and Personal Confidence
33.54 (5.88)
34.39 (5.34)
34.39 (5.01)
5.077**
.017
Asking for Help
12.17 (1.98) 
12.37 (1.84) 
12.16 (1.97) 
1.513
.005
Goal-Orientation
19.98 (3.34) 
20.18 (3.22) 
19.99 (3.08) 
.685
.002
Reliance on others
15.46 (2.92) 
15.96 (2.62) 
15.72 (2.57)  *
.014 
Changes in recovery orientation. ***p<.001, **p<.01, *p<.05. (n=294)

24. Client Satisfaction
Conduct ongoing random sample surveys of persons served
Interviews conducted by non-service provider/peer research assistant

25. Consumer Satisfaction Survey
Clients could score between 10 and 60 points, with the higher score indicating higher satisfaction
Actual scores ranged from 26 to 60, mean of (SD=7.01) and a median of 58

26. Consumer Satisfaction Survey (continued)
Consumer Satisfaction Survey Question
Mean (SD)
Median
I am not satisfied with the amount of services I received
5.10 (1.57)
Strongly Disagree
The services I received will help me to deal more effectively with my problems
5.32 (1.29)
Strongly Agree
The staff treated me with respect and dignity
5.45 (1.18)
I feel I have a right to approve the services I receive
5.46 (1.11)
I was able to get the services I thought I needed
5.48 (0.81)
I want to remain in this program for the time being
5.50 (1.27)
Overall, I am satisfied with the services I received
5.53 (1.00)
Staff and services were sensitive to my personal needs and preferences
5.58 (0.90)
I liked the services I received
5.62 (0.75)
I would not recommend this agency to a friend or family member
5.76 (0.91)
Consumer Satisfaction Scale items (lowest to highest; scores range from 1 to 6; n=124)

27. Consumer Satisfaction Survey (continued): Perception of Care
Clients could score between 14 and 70 points, with the higher score indicating higher satisfaction.
Actual scores ranged from 27 to 70 points; mean is (SD=8.57), median of 59
Perception of Care Survey Question
Mean (SD)
Median
Staff were sensitive to my cultural background.
4.05 (0.90)
Agree
Staff told me what side effects to watch out for
4.07 (0.83)
Staff respected my wishes about who is and who is not to be given information about my treatment
4.09 (0.99)
I, not staff, decided my treatment goals
4.10 (0.78)
I feel free to complain.
4.16 (0.77)
I was given information about my rights.
4.17 (0.66)
I was encouraged to use consumer run programs (e.g., support groups, drop-in centers)
4.18 (0.76)

28. Consumer Satisfaction Survey (continued): Perception of Care
Perception of Care Survey Questions
Mean (SD)
Median
If I had other choices, I would still get services from this agency.
4.20 (0.85)
Agree
Staff encouraged me to take responsibility for how I live my life
4.24 (0.85)
I would recommend this agency to a friend or family member
4.24 (0.80)
Staff helped me obtain the information I needed so that I could take charge of managing my illness
4.26 (0.92)
I like the services I received here
4.30 (0.82)
Staff here believe that I can grow, change, and recover
4.34 (0.77)
I felt comfortable asking questions about my treatment and medication
4.34 (0.74)
Perception of Care Scale items, ranked lowest to highest; Scores range from 1 to 5

29. Future Directions Fine-tuning enhancements of outcome measures
Additional analyses
Over longer time intervals and with bigger samples
Subgroup analyses
Outcomes by program
Additional efforts to increase data to inform clinical service practices
Possible further development/joint venture with EMR or other possible partners to expand and market outcome tools

30. Comments and Questions?
Thank you!