1. Evaluation of Health Canada’s Hepatitis C Program: Engaging Stakeholders in the Evaluation Process
Presented by: Gail V. Barrington, PhD, CMC
Barrington Research Group, Inc.
National Hepatitis Coordinators’ Conference
Program Evaluation for Viral Hepatitis Integration Projects Workshop
San Antonio, Texas
January 30, 2003

2. Key Principles of the Evaluation Process
Wide stakeholder engagement from the evaluation design to Program recommendations
Consultation with hepatitis C experts
Guided by a Program logic model and Data Collection Matrix
Use of traditional social science and applied research methods
Peer reviews of the final report
Purpose of this presentation is to demonstrate how these key principles were used in the mid-term evaluation of Health Canada’s Hepatitis C Program with the ultimate goal of increasing the utility of the evaluation.

3. Initial Consultations
Preliminary interviews with all regional Program staff
Informal telephone interviews conducted by the Project Director
Project Director “listened in” on Program teleconferences early in the evaluation
Purpose:
To build rapport and acclimatize regional staff to the evaluation and the Evaluators
To understand the regional perspective of the evaluation
Two hepatitis C experts were invited to educate Barrington Research staff on hepatitis C issues
Building rapport with the regional Program staff was critical since they are the gatekeepers.
Several components of the Program are delivered through a regional structure.
Recognized that their (regional) cooperation would be critical for the evaluation to proceed.

4. Instrument Design
Each survey/interview question was designed to address an evaluation question from the Data Collection Matrix (DCM)
The DCM was based on the Program Logic Model
This use of the Logic Model and DCM helped to focus the questions asked in the surveys/interviews
Survey/interview tools were also reviewed by stakeholders to ensure question relevancy, appropriateness, and comprehensiveness
Health Expert Survey
Various case study tools
Community-Based Support Implementation and Outcome Achievement Survey
Researcher Survey
For each of the 4 areas addressed by the mid-term evaluation (Scope of the Problem, Program Implementation, Progress Toward Outcome Achievement & Lessons Learned), the DCM contains a set of evaluation questions and related indicators (evidence that the evaluation question has or has not been addressed by the Program).
The Program Logic Model outlines the primary Program activities and their expected outputs, as well as the expected immediate, intermediate and long-term outcomes of the Program activities.
The Health Expert Survey: Mark Swain
The case study interviews: Lynn Schindel ensured that potential social issues were addressed by the tools. Also reviewed by the case study sites.
Community-Based Survey: Regional Program staff ( process didn’t work in terms of achieving the preferred response rate)
Researcher Survey: CIHR (positive and negative aspects)
This engagement of the stakeholders likely contributed to the high response rates exhibited in most cases.

5. Sample Selection
Because this was a formative evaluation, participants from many groups were to be surveyed to gain a broad overview of the Program
Purposive sampling best addressed the need to collect data from populations of an unknown size (e.g., “health experts”)
As resources were limited, this sampling approach was appropriate
Key informants were used to identify potential participants:
Health Canada – National and regional Program staff and other stakeholders at the national/ regional levels
Hepatitis C experts – Health Expert Survey
Canadian Institutes for Health Research (research fund manager) – Researcher Survey

6. Sample Selection (cont’d)
Local level
Case studies in 7 sites across Canada to:
Access those infected with/affected by hepatitis C, community-based support projects funded by the Program, other hepatitis C service providers in the community
Explore Program implementation at the micro level
Methodology based on the work of theorists (Yin & Chelimsky)
Sites selected in consultation with Health Canada staff and based on criteria such as regional representation, service to a priority population, project type and willingness to participate
The case studies were valuable because they:
Provided a sense of how the Program plays out at the community level;
Provided insight into social issues in the community;
Allowed for access to other community agencies (and individuals such as the Lethbridge pharmacist) that might not otherwise have been identified;
Allowed us to strengthen our conclusions (e.g., clarifying info re: “doctors don’t understand my needs”);
Allowed for the discovery of info that might not otherwise have been uncovered (e.g., Hepatitis C Society supported at the national level but falling apart at the community level); and
Provided important insight into building community capacity (e.g., cannot give money to community groups without support for infrastructure, board development, etc).

7. Data Collection Reviewed Program documents
Survey/interview data collection proceeded smoothly due to:
Support of stakeholders
Respectful treatment of participants
Understanding of community agencies
Multiple response options for each instrument (in-person or phone interviews; , mail or fax surveys)
Thank you cards, gifts to the participating case study sites and incentives (grocery coupons) to clients interviewed

8. Analysis & Write-Up Used SPSS and N-Vivo for data analysis
Utility of the findings kept in mind—”What does this Program need?”
Data Collection Matrix used to guide qualitative analysis and the “triangulation” of all data sources
All data sources that addressed an evaluation question were compiled in a Data Summary
Similarities and differences across groups of respondents were compared
Findings that are included in the final report represent themes that stood out across all groups of respondents and/or documents
To reduce the potential for bias and to obtain as balanced a picture as possible, data were collected from multiple sources to address each evaluation question.

9. Analysis & Write-Up (cont’d)
Case studies:
Research team met to draft a report template (series of questions)
Team leaders organized case study data using the report template
Completed template given to Project Director for case study write-up
Preliminary review of each case by research team
Review of each case by site coordinator
Case study signed-off by site prior to distribution to Health Canada
Health Canada regional staff reviewed and made final changes
Process increased stakeholder ownership of the findings
Final Report:
Guided by the Logic Model for report structure
Peer reviewers helped clarify & focus the report
Various national and regional stakeholders reviewed the draft report
Brought in Bill Reeves for an external perspective at a late stage.
Took us a step back but was necessary due to the complexity of the study.
Bill also had extensive expertise, for example, in training requirements for physicians. His comments helped us to interpret some of the findings, particularly around community capacity.

10. Lessons Learned
The evaluation identified lessons learned and highlighted Program strengths and weaknesses
Through these insights, the Evaluators were able to propose 19 recommendations
Health Canada staff were consulted on the wording of the recommendations to facilitate their implementation