Therapeutic Alliance and Length of Stay Implementation of Patient Feedback A total of 41 clinicians and 9 supervisors participated in the Patient Feedback study. Mean age of clinicians was n=47. 46% of clinicians were female. 71% of clinicians were Caucasian Level of education: 7 Bachelor 31 Master 2 Doctorate 1 MD/DO Implementation Results 100% of supervisors conducted monthly Team Meetings, and faxed the Team Meeting form. 86% of the eligible clinicians participated in the monthly team meetings. A total of 2,814 PF Surveys were collected and faxed by the six participating clinics, for an overall collection rate of 75.5% of the clinic census. 564 Feedback Reports were posted to the PF website. 100% PF Reports were posted within one-week of faxing, and 83% were posted within 48 hours. 100% of the clinic supervisors downloaded their Clinic Feedback Report at least monthly. 100% of clinicians downloaded at least one Caseload Report, and a total of 112 Caseload Reports, or 2.7 reports per clinician, were downloaded during the feasibility trial. 100% of the Attendance Forms were faxed on schedule by the participating clinics. The PF Website operated continuously throughout the feasibility study, with a total of nearly 5,000 web views by the 50 participating staff members. The PF News was published on a monthly basis with a circulation of nearly 300 recipients. During the one year sustainability phase (post-intervention), in which clinics were allowed but not obligated to use the Patient Feedback system as often as they wished, three clinics continued to use PF at the same rate (every other week) as during the intervention phase; two clinics faxed PF surveys on about a monthly basis; one clinic chose to use PF on a semi-annual basis. Patient Feedback Results and Study Limitations Overall, patients reported clinicians were performing very well, with relatively low average levels of patient drug and alcohol use. Similarly, alliance and treatment satisfaction ratings were high across all assessments. Only one clinic showed relatively higher levels of drug use at the start of the intervention phase, although there is evidence of some improvements in drug use outcomes over time. There was also evidence of improvements in average patient therapeutic alliance scores over time during the intervention phase for those clinicians who had initially poor treatment satisfaction ratings. This study demonstrated that the implementation of a semi-automatic quality improvement system for clinicians in addiction treatment facilities was feasible from both a research and clinical perspective. The limitations of this study include a) there was no control condition; b) the study sample was small; and c) the PF intervention was employed for only 3 months. The Future of Quality Improvement Quality Improvement (QI) is a core component of healthcare delivery systems throughout the US, yet we are unaware of any published trials testing the effectiveness of QI in addiction treatment settings. QI is viewed as one potential bridge for introducing evidence-based practices into community-based treatment programs. In the original Patient Feedback feasibility study presented, the feasibility and acceptability of PF was established. A larger scale Patient Feedback effectiveness trial has been funded by the National Institute on Drug Abuse and will begin October This trial is being conducted by the University of Pennsylvania and NYU School of Medicine, with up to 32 clinics participating. It is expected that this clinical trial will provide valuable information about the PF intervention, as well as QI interventions, in substance use treatment settings. Additional knowledge will be gained about the relationship between patient, clinic, clinician, and supervisor characteristics and improvements in patient attendance and abstinence in outpatient settings. Acknowledgements Lead Investigator: Robert Forman, Ph.D. National Lead Node: University of Pennsylvania, Treatment Research Institute National Lead Node: University of Pennsylvania, Treatment Research Institute Supported and developed by the National Institute on Drug Abuse Clinical Trials Network Patient Feedback: A Quality Improvement Study In Outpatient Settings Agatha Kulaga 1 M.S.W., Robert Forman 2 Ph.D., John Rotrosen 1 M.D., Paul Crits-Christoph 2 Ph.D., Matt Worley 2 B.A. 1 NYU School of Medicine and VA NYHHS, 2 University of Pennsylvania Clinical Trials Network The Clinical Trials Network (CTN) is a national partnership between researchers and community treatment providers. The mission is to conduct effectiveness research in order to determine if interventions developed and studied in research clinics work when tested in real world settings. Research results are disseminated to providers and their patients within the community with the goal of implementing such new treatments as part of usual care. Current and past CTN protocols address substance use disorders, infections and addiction, smoking cessation, issues related to women and trauma, adolescent substance abuse, and HIV/AIDS. The New York Node The New York Node, a consortium of academic and clinical investigators and providers, has contributed to the CTN mission by collaborating in research, training, clinical service delivery, policy making, and funding for addictive disorders. Participating Community Based Treatment Programs (CTPs) include a wide range of programs and treatment models, and provide services for, and access to, a diverse clinical population. Performance Improvement Patient Feedback (PF) is a web based quality improvement (QI) system designed to monitor patient ratings of therapeutic alliance and other quality indicators in order to empower clinical staff. Improvement in therapeutic alliance has been shown to be associated with improvement in a variety of clinically important outcomes including attendance, retention, and abstinence. This feasibility study was designed and implemented at 6 CTN clinics beginning in April 2004: New York: Lower East Side Service Center & Addiction Research Treatment Corporation Great Lakes: Chelsea Arbor Addiction Treatment Center New England: Adcare Outpatient Treatment Center North Carolina: SouthLight, Inc. South Carolina: Dorchester Alcohol and Drug Commission The Patient Feedback System 1. Every other week, at the close of group, clients are invited to complete a 12-item self-administered confidential PF Survey in which they rate therapeutic alliance and their satisfaction with the group. 2. The surveys are collected and faxed by an administrative assistant to the University of Pennsylvania, where they are converted into feedback reports. 3. Confidential Clinician and Clinic PF Reports are posted to a password protected website within about an hour of being faxed. 4. Individual clinicians access Feedback Reports for their own caseloads, as well as the aggregated data from the clinic, from the PF website. Clinic supervisors access Feedback Reports summarizing data from the combined clinic caseload. 5. On a monthly basis, the clinic supervisors and clinicians meet as a team to discuss the Feedback Reports, identify quality indicators they would like to improve, and discuss the improvement strategies they plan to implement. 6. A monthly newsletter publicizes clinic successes. 7. The PF cycle is repeated every other week. Patient Feedback System Overview of PF Survey (cont’d) Item Scale – The first 7 PF Survey items use a five point Likert-like scale: Not at all - A little Bit – Moderately - Quite a bit - Very much so Patient Demographic Items (Survey Items 8-10) Item 8. Do you consider yourself (please select only one): White; African American/Black; American Indian or Alaska Native; Hispanic or Latino; Asian; Native Hawaiian or Pacific Islander Item 9. Are You: Male; Female Item 10. Concerning this admission, about how long have you been in treatment? Less than 1 week; 1 – 4 weeks; months; More than 3 months Self-reported Substance Use (Survey Items 11-12) Item 11. How many days in the PAST WEEK did you drink any alcohol (beer, wine, liquor)? Item 12. How many days in the PAST WEEK did you use any drugs (marijuana, cocaine, heroin, other)? Unlike items 1-9, responses to items are not be fed back to supervisors or clinicians; instead they are used in the outcomes evaluation. Attendance Data In addition to the data obtained from the PF Surveys, the second data source for PF is attendance data extracted from the clinic’s administrative record. Every other week, the administrative assistant extracts attendance data from the clinic’s records and enters those data onto the Attendance Form. The PF data system converts data from the Attendance Form into attendance rates using the following calculation: Attendance= Total # of pt. sessions attended by the clinician caseload Total # of sessions scheduled Clinic and Caseload Feedback Reports The Feedback Reports are provided to study participants (clinicians) as seven graphs and two data tables. Individual clinicians can access both the Clinic and Caseload Reports; supervisors can only access the Clinic Report. The seven time-series graphs reflect: a) Therapeutic Alliance and Length of Stay b) Therapeutic Alliance and Ethnicity c) Therapeutic Alliance and Gender d) Treatment Satisfaction and Length of Stay e) Treatment Satisfaction and Ethnicity f) Treatment Satisfaction and Gender g) Attendance. Beneath each graph, a data table for each graph is appended. Finally, one data table presents the combined data for all of the graphs in the current Report, and a second table presents “Frequency Data” for each of the first seven items for each Survey distribution. Sample PF Feedback Report The following is an example of one of the seven graphs included in each PF Report. The graph shows feedback from a clinic where the percent of clients giving the highest rating on therapeutic alliance (“very much so”) increased from 67% in May 1, 2004 to 84% in July 10, This graph presents ratings of therapeutic alliance based on how long clients have been in treatment. Within this specific clinic there were few differences between the ratings from new patients ( 3 months). Other graphs present ratings of therapeutic alliance or treatment satisfaction based on gender and ethnicity. Training was provided on how to quickly “read” all Patient Feedback Reports. Feedback Reports Attendance Treatment Satisfaction All Current Data Item Ratings Ethnicity Gender Length of Stay Therapeutic Alliance Abstinence Overview of PF Survey Therapeutic Alliance Items (Survey Items 1-4) Item 1. Did you feel accepted and respected by your clinician? Item 2. Did you feel that you and your clinician were working together to overcome your problems? Item 3. Did you feel that your clinician understood what you hoped to get out of your treatment? Item 4. Did you feel confident that through your own efforts and those of your clinician you will gain relief from your problems? Treatment Satisfaction Items (Survey Items 5-7) Item 5. Did you feel comfortable raising issues or concerns? Item 6. Were things explained to you in a way you could understand? Item 7. Was the session helpful?