MAT-PDOA Program Evaluation Diana Seybolt, Ph.D. Karen McNamara, Ph.D. Systems Evaluation Center (SEC)

Program Evaluation A systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency.

Purpose of MAT-PDOA Program Evaluation Track progress on MAT PDOA goals and objectives Identify potential disparities in sub-population access, utilization, and outcomes Inform ongoing implementation efforts at the state and local levels and identify areas requiring adjustments Provide information for required reporting to SAMHSA regarding program implementation efforts, challenges encountered, and strategies implemented

Baltimore City MAT-PDOA Objectives Objectives 1. Increase the number of individuals evaluated for MAT by recruiting opioid overdose survivors into SRD treatment 2. Increase the number of opioid overdose survivors receiving MAT 3. Increase length of stay in MAT by 40%; Reduce patients leaving MAT against medical advice by 20% 4. Decrease illicit drug use by those in MAT treatment 5. Increase number of residents in 3.7 treatment undergoing buprenorphine induction treatment 6. Increase referrals from 3.7 treatment to MAT programs 7. Prevent opioid-related deaths

Baltimore City MAT-PDOA Objectives ObjectivesEvaluation Activities 1. Increase the number of individuals evaluated for MAT by recruiting opioid overdose survivors into SRD treatment Survivor Program Contact and Referral Data, Stakeholder Interviews 2. Increase the number of opioid overdose survivors receiving MAT Claims/Service Utilization Data, Contact and Referral Data, Stakeholder Interviews, Survivor Program Contact and Referral Data 3. Increase length of stay in MAT by 40%; Reduce patients leaving MAT against medical advice by 20% Claims/Service Utilization Data, Stakeholder Interviews 4. Decrease illicit drug use by those in MAT treatment GPRA Questionnaire (Section B, alcohol and drug use), Stakeholder Interviews, Treatment Progress Data 5. Increase number of residents in 3.7 treatment undergoing buprenorphine induction treatment Contact and Referral Data, Claims/Service Utilization Data, Stakeholder Interviews 6. Increase referrals from 3.7 treatment to MAT programs Contact and Referral Data 7. Reduce opioid-related deaths Office of Chief Medical Examiner Data

Additional Outcome and Process Evaluation ObjectivesEvaluation Activities Progress, Barriers, Efforts to Overcome Barriers All of the above Outcome Questions: intervention effects, factors associated with outcomes, effect durability, 6-month follow up. GPRA Questionnaire (Section B, Alcohol and Drug Use), Stakeholder Interviews, Treatment Progress Data Process Questions: implementation, changes made, changes to address disparity, service provision, # reached through project Stakeholder Interviews, Contact and Referral Data, Claims/Service Utilization Data

Evaluation Guidelines Perform a thorough evaluation that provides useful information to our grantor as well as to program decision-makers Minimize intrusion on staff time Build upon your existing procedures Develop methods of data collection that provide useful information to individuals “on the ground”

Evaluation Activities Contact and Referral Data Treatment Progress Data Stakeholder Interviews Claims/Service Utilization/OCME data Government Performance and Results Act (GPRA) Questionnaire

Contact and Referral Data Who State Care Coordinators and Care Coordinators at MAT programs OSOP/Other Outreach Peers Administrative Data What Counts of the following, including demographic information, to understand client flow and assess for potential disparities clients inducted for buprenorphine clients referred from induction to MAT services clients with opioid overdoses receiving peer outreach clients referred to Care Coordinator for MAT Where Total Health Care Reflective Treatment Center Daybreak Rehabilitation Center Tuerk House Other Locations (for peers and outreach) When Collected monthly, reported quarterly How Contact Logs, Clinic Data Collection Procedures

Treatment Progress Data This evaluation piece is under development. General idea is to capture basic information about client progress, such as number of clients making progress on treatment goals per month. Will be developed in consultation with MAT- PDOA community partners. We will also be able to capture more traditional outcomes data through the GPRA questionnaires, and perhaps eventually the BHA Outcomes Measurement System (OMS).

Stakeholder Interviews Who Clients Peers (MARS, outreach) Care Coordinators MAT program staff County and State Administrators (grant leadership) What: Group interviews with each stakeholder group using a semi-structured interview (currently in development) Where Locations convenient to stakeholder groups (to be determined) When Approximately every 12 months on a rolling basis How Interviews conducted and summarized by the UMB Systems Evaluation Center

Claims/Service Utilization/OCME data Who Treatment Providers will enter claims and authorization data into the ASO system (ValueOptions®) as usual business practice What Specific variables to analyze currently being identified. Where Data files provided by ValueOptions® to BHA Office of the Chief Medical Examiner (OCME) currently provides monthly data to BHA regarding Maryland deaths. When Extracted and analyzed monthly, once available How Data to be analyzed by UMB Systems Evaluation Center

Government Performance and Results Act (GPRA) Questionnaire Who State Care Coordinators conduct interviews and enter data What: a client interview/performance measure mandated by SAMHSA Where Healthcare Access Maryland enters information into SAMSHA database When Enrollment, 6 months, and Discharge How Data made available by SAMSHA will be analyzed by UMB Systems Evaluation Center staff

What we’ll need from you Everyone Meet with evaluators so that we can understand your workflow, data availability, etc. Provide feedback regarding program evaluation workflow and evaluation tools as developed Assist in arranging annual group interviews for staff and/or clients (inviting participants, possibly arranging space, etc.) Participate in annual group interviews

What we’ll need from you MAT Programs Collect Contact and Referral Data and submit to SEC monthly Collect Treatment Progress Data and submit to SEC monthly Tuerk House Collect Contact and Referral Data and submit to SEC monthly Collect Treatment Progress Data (possibly) and submit to SEC monthly State Care Coordinators Conduct GPRA Interviews and enter data Program Care Coordinators and Other Peer Outreach Collect Contact and Referral Data and submit to SEC monthly MARS Collect evaluation information to be determined (e.g. Total meetings per month, total number of attendees)

What we will produce Quarterly Summary Reports - The SEC will create summary reports integrating these various sources of data and provide them to the Project Director and local administrators at least quarterly for grant management. Group Interview Summaries - As stakeholder interviews are completed, the participating group will also receive a brief report summarizing the results.

Questions? Karen McNamara Di Seybolt