Research and Evaluation Overview SANJEEV SOCKALINGAM & JANE ZHAO June 15, 2018 1

LEARNING OBJECTIVES 1. Describe past and current research activities for Project ECHO® 2. Differentiate research and evaluation requirements for Project ECHO® 3. Identify evaluation frameworks and measures for Project ECHO ® AT THE CONCLUSION OF THIS SESSION, YOU WILL : 2

Resources: Publications & Data Analyses 80+ publications and growing Across at least 13 partners and 12 different conditions Majority of ECHOs evaluate levels one to four Four ECHOS have included patient health outcomes UNM: NEJM study on HCV (2011) Beth Israel Deaconess Medical Center: two JAMDA studies on geriatric mental health (2014, 2016) VHA: Journal of Telemedicine and Telecare study on endocrinology (2015) Cost effectiveness analyses Conf. paper on HCV that showed cost effectiveness at $3,500 per QALY Travel data in some papers that could be translated to dollars 3

Systematic Review: Current Evidence To Systematic Review: Current Evidence To Support ECHO Model Zhou, Crawford, Serhal et al., 2016 Moore’s Evaluation Framework # of Studies Results Level 1: Participation 12 Median participants = 38 Level 2: Satisfaction 13 All studies showed high levels of participant satisfaction Level 3: Learning / Knowledge 4 Increased pre-post knowledge scores Level 4: Competence 8 Used surveys and/or semi-structured interviews; 7 out of 8 studies showed improved participant self-efficacy Level 5: Performance 1 Chronic Pain: Change in service utilization (less mental health and more physical health) and increased non-opioid medication usage Level 6: Patient Health 7 Hepatitis C: Similar to SVR rates to specialists Dementia: Less hospitalized and improved behavioural issues Diabetes: Improved HbA1C levels Level 7: Community Health None

Publications by Conditions and Outcome Levels 5

Overall ECHO Impact Total number of ECHO publications to date: 86 Total number of Citations by ECHO-addressed conditions: 1223 6

Existing Data Sources for ECHOs iECHO / ECHO Admin statistics Records (chart reviews, referrals, travel savings) Existing databases Electronic Health Records Insurance claims data (e.g., Medicaid, managed care organizations) Statistics Canada-Census data 7

Research vs. Evaluation: How would you describe the difference? 8

Research Versus Evaluation Both use similar methods and analytical procedures Serve different purposes and goals Guided by different questions Target different audiences 9

Why It Is Important To Evaluate? 1010

Importance of Evaluation To determine the effectiveness of the ECHO program and if we are meeting our objectives (i.e. benefiting under-serviced, remote, and rural areas) To identify quality improvement opportunities To measure participant’s change in knowledge and self-efficacy To meet the MOHLTC funding requirements for reporting To document & distribute CME credits To demonstrate accountability within your institution To publish and add knowledge to your area 1111

Moore’s Evaluation Framework Why use this framework? We want to see how ECHO affects individual level change, but also how it relates to patient and population level health outcomes

Common ECHO Ontario Metrics Number of Spoke sites per cycle Number of sessions attended (per participant and per Spoke site) Number of professions/disciplines participating Participants knowledge - pre/post change Participants self-efficacy (includes attitudes and confidence)- pre/post change Clinical effectiveness- are recommendations implemented Participants satisfaction- per session and/or post cycle 1313

Evaluation Framework For ECHO Moore’s Evaluation Framework Evaluation Measures Sources of Data Level 1: Participation Number of Spoke sites Number of sessions attended Number of professions/disciplines participating Registration records Attendance records Level 2: Satisfaction Satisfaction surveys (looking at IT, format, learning environment) Session satisfaction surveys Post cycle satisfaction surveys Focus groups Interviews Level 3: Learning / Knowledge Change in knowledge Use pre and post test multiple choice questions around knowledge Level 4: Competence Changes in perceived self-efficacy Pre-post survey rating scales Level 5: Performance Degree to which attendees perform what ECHO intended them to do Observe performance in clinical setting Patient chart reviews to view recommendations implemented Self-report performance surveys Level 6: Patient Health How much does health of patients change as a result of ECHO Patient chart reviews Patient interviews or self-reported questionnaires on health status Level 7: Community Health Degree to which health in the community of patients changes due to ECHO-related changes in practice Epidemiological data and reports 1414

The Evaluation Process Evaluation needs to be embedded in the project planning process Meet with relevant stakeholders: Discuss a plan for your protocol Rationale for doing this evaluation-What do you want this evaluation to achieve? Identify your target population Do you want to publish? Conduct a literature search: What studies currently exist? How will this evaluation add to knowledge in this area? 1515

The Evaluation Process Start developing your protocol: Determine a design: Qualitative, Quantitative, Mixed Methods? Determine methods/tools to be used: Focus Groups, Paper Surveys, Online Surveys, Observations, Interviews? Determine types of questions to be used and how you want to ask them: Consider Moore’s Framework Knowledge, Self-Efficacy, Attitudes, Satisfaction, Open-ended questions, etc.? Multiple Choice, Likert Scale, Pre/post, etc.? Determine database and analysis tools: RedCap-Data capture software tool or SurveyMonkey SPSS or SAS for quantitative data analysis NVivo for qualitative data analysis Submit for REB or QI approval and start evaluating 1616

Choosing between QI and REB Quality Improvement Answers research question Risk must be assessed by ethics committee Rigid protocol Findings are generalized REB Improves program, process or system Minimal participant risk Adaptive, iterative design over time Findings applicable to local institution *For further guidance, connect with your institutions REB Coordinator

REB Application Process Develop Protocol (include background, rationale, methodology, risks/benefits, confidentiality, etc.) Identify PI Evaluation Development Consent Forms Assent Forms Recruitment Materials Data Collection Forms Staff Credentials/ Certificates Gather Supplementary Documents Create your REB submission package with all required documents-follow your institutions REB checklist REB Submission Revisions may be required Additional documents may need to be submitted Be aware the REB approval process can take up to months REB Decisions *For further guidance on your institutions REB process, connect with your REB Coordinator 1818

REB Review Process *For further guidance on your institutions REB process, connect with your REB Coordinator 1919

Potential Barriers To Keep In Mind Delays with REB approval Balancing evaluation needs with program needs Development of knowledge metrics for inter-professional Spokes Completion of research forms by Spokes (pre/post, weekly, etc.) Gathering patient and community level data Data sharing between ECHOs for larger evaluations 2020

Take Home Messages Plan your evaluation early- include it in your project planning process Think about what evaluation methods are the most practical and feasible for your ECHO 2121