1. Research and Evaluation Overview
SANJEEV SOCKALINGAM & JANE ZHAO
June 15, 2018 1

2. LEARNING OBJECTIVES
1. Describe past and current research activities for Project ECHO® 2. Differentiate research and evaluation requirements for Project ECHO® 3. Identify evaluation frameworks and measures for Project ECHO ®
AT THE CONCLUSION OF THIS SESSION, YOU WILL : 2

3. Resources: Publications & Data Analyses
80+ publications and growing
Across at least 13 partners and 12 different conditions
Majority of ECHOs evaluate levels one to four
Four ECHOS have included patient health outcomes
UNM: NEJM study on HCV (2011)
Beth Israel Deaconess Medical Center: two JAMDA studies on geriatric mental health (2014, 2016)
VHA: Journal of Telemedicine and Telecare study on endocrinology (2015)
Cost effectiveness analyses
Conf. paper on HCV that showed cost effectiveness at $3,500 per QALY
Travel data in some papers that could be translated to dollars 3

4. Systematic Review: Current Evidence To
Systematic Review: Current Evidence To Support ECHO Model Zhou, Crawford, Serhal et al., 2016
Moore’s Evaluation Framework
# of Studies
Results
Level 1: Participation 12
Median participants = 38
Level 2: Satisfaction 13
All studies showed high levels of participant satisfaction
Level 3: Learning / Knowledge 4
Increased pre-post knowledge scores
Level 4: Competence 8
Used surveys and/or semi-structured interviews; 7 out of 8 studies showed improved participant self-efficacy
Level 5: Performance 1
Chronic Pain: Change in service utilization (less mental health and more physical health) and increased non-opioid medication usage
Level 6: Patient Health 7
Hepatitis C: Similar to SVR rates to specialists
Dementia: Less hospitalized and improved behavioural issues
Diabetes: Improved HbA1C levels
Level 7: Community Health
None

5. Publications by Conditions and Outcome Levels5

6. Overall ECHO Impact Total number of ECHO publications to date: 86
Total number of Citations by ECHO-addressed conditions: 1223 6

7. Existing Data Sources for ECHOs
iECHO / ECHO Admin statistics
Records (chart reviews, referrals, travel savings)
Existing databases
Electronic Health Records
Insurance claims data (e.g., Medicaid, managed care organizations)
Statistics Canada-Census data 7

8. Research vs. Evaluation: How would you describe the difference?8

9. Research Versus Evaluation
Both use similar methods and analytical procedures
Serve different purposes and goals
Guided by different questions
Target different audiences 9

10. Why It Is Important To Evaluate?
1010

11. Importance of Evaluation
To determine the effectiveness of the ECHO program and if we are meeting our objectives (i.e. benefiting under-serviced, remote, and rural areas)
To identify quality improvement opportunities
To measure participant’s change in knowledge and self-efficacy
To meet the MOHLTC funding requirements for reporting
To document & distribute CME credits
To demonstrate accountability within your institution
To publish and add knowledge to your area
1111

12. Moore’s Evaluation Framework
Why use this framework? We want to see how ECHO affects individual level change, but also how it relates to patient and population level health outcomes

13. Common ECHO Ontario Metrics
Number of Spoke sites per cycle
Number of sessions attended (per participant and per Spoke site)
Number of professions/disciplines participating
Participants knowledge - pre/post change
Participants self-efficacy (includes attitudes and confidence)- pre/post change
Clinical effectiveness- are recommendations implemented
Participants satisfaction- per session and/or post cycle
1313

14. Evaluation Framework For ECHO
Moore’s Evaluation Framework
Evaluation Measures
Sources of Data
Level 1: Participation
Number of Spoke sites
Number of sessions attended
Number of professions/disciplines participating
Registration records
Attendance records
Level 2: Satisfaction
Satisfaction surveys (looking at IT, format, learning environment)
Session satisfaction surveys
Post cycle satisfaction surveys
Focus groups
Interviews
Level 3: Learning / Knowledge
Change in knowledge
Use pre and post test multiple choice questions around knowledge
Level 4: Competence
Changes in perceived self-efficacy
Pre-post survey rating scales
Level 5: Performance
Degree to which attendees perform what ECHO intended them to do
Observe performance in clinical setting
Patient chart reviews to view recommendations implemented
Self-report performance surveys
Level 6: Patient Health
How much does health of patients change as a result of ECHO
Patient chart reviews
Patient interviews or self-reported questionnaires on health status
Level 7: Community Health
Degree to which health in the community of patients changes due to ECHO-related changes in practice
Epidemiological data and reports
1414

15. The Evaluation Process
Evaluation needs to be embedded in the project planning process
Meet with relevant stakeholders:
Discuss a plan for your protocol
Rationale for doing this evaluation-What do you want this evaluation to achieve?
Identify your target population
Do you want to publish?
Conduct a literature search:
What studies currently exist?
How will this evaluation add to knowledge in this area?
1515

16. The Evaluation Process
Start developing your protocol:
Determine a design:
Qualitative, Quantitative, Mixed Methods?
Determine methods/tools to be used:
Focus Groups, Paper Surveys, Online Surveys, Observations, Interviews?
Determine types of questions to be used and how you want to ask them:
Consider Moore’s Framework
Knowledge, Self-Efficacy, Attitudes, Satisfaction, Open-ended questions, etc.?
Multiple Choice, Likert Scale, Pre/post, etc.?
Determine database and analysis tools:
RedCap-Data capture software tool or SurveyMonkey
SPSS or SAS for quantitative data analysis
NVivo for qualitative data analysis
Submit for REB or QI approval and start evaluating
1616

17. Choosing between QI and REB
Quality Improvement
Answers research question
Risk must be assessed by ethics committee
Rigid protocol
Findings are generalized
REB
Improves program, process or system
Minimal participant risk
Adaptive, iterative design over time
Findings applicable to local institution
*For further guidance, connect with your institutions REB Coordinator

18. REB Application Process
Develop Protocol (include background, rationale, methodology, risks/benefits, confidentiality, etc.)
Identify PI
Evaluation Development
Consent Forms
Assent Forms
Recruitment Materials
Data Collection Forms
Staff Credentials/ Certificates
Gather Supplementary Documents
Create your REB submission package with all required documents-follow your institutions REB checklist
REB Submission
Revisions may be required
Additional documents may need to be submitted
Be aware the REB approval process can take up to months
REB Decisions
*For further guidance on your institutions REB process, connect with your REB Coordinator
1818

19. REB Review Process
*For further guidance on your institutions REB process, connect with your REB Coordinator
1919

20. Potential Barriers To Keep In Mind
Delays with REB approval
Balancing evaluation needs with program needs
Development of knowledge metrics for inter-professional Spokes
Completion of research forms by Spokes (pre/post, weekly, etc.)
Gathering patient and community level data
Data sharing between ECHOs for larger evaluations
2020

21. Take Home Messages
Plan your evaluation early- include it in your project planning process
Think about what evaluation methods are the most practical and feasible for your ECHO
2121