Evaluation of Health Canada’s Hepatitis C Program: Engaging Stakeholders in the Evaluation Process Presented by: Gail V. Barrington, PhD, CMC Barrington Research Group, Inc. National Hepatitis Coordinators’ Conference Program Evaluation for Viral Hepatitis Integration Projects Workshop San Antonio, Texas January 30, 2003

Key Principles of the Evaluation Process Wide stakeholder engagement from the evaluation design to Program recommendations Consultation with hepatitis C experts Guided by a Program logic model and Data Collection Matrix Use of traditional social science and applied research methods Peer reviews of the final report Purpose of this presentation is to demonstrate how these key principles were used in the mid-term evaluation of Health Canada’s Hepatitis C Program with the ultimate goal of increasing the utility of the evaluation.

Initial Consultations Preliminary interviews with all regional Program staff Informal telephone interviews conducted by the Project Director Project Director “listened in” on Program teleconferences early in the evaluation Purpose: To build rapport and acclimatize regional staff to the evaluation and the Evaluators To understand the regional perspective of the evaluation Two hepatitis C experts were invited to educate Barrington Research staff on hepatitis C issues Building rapport with the regional Program staff was critical since they are the gatekeepers. Several components of the Program are delivered through a regional structure. Recognized that their (regional) cooperation would be critical for the evaluation to proceed.

Instrument Design Each survey/interview question was designed to address an evaluation question from the Data Collection Matrix (DCM) The DCM was based on the Program Logic Model This use of the Logic Model and DCM helped to focus the questions asked in the surveys/interviews Survey/interview tools were also reviewed by stakeholders to ensure question relevancy, appropriateness, and comprehensiveness Health Expert Survey Various case study tools Community-Based Support Implementation and Outcome Achievement Survey Researcher Survey For each of the 4 areas addressed by the mid-term evaluation (Scope of the Problem, Program Implementation, Progress Toward Outcome Achievement & Lessons Learned), the DCM contains a set of evaluation questions and related indicators (evidence that the evaluation question has or has not been addressed by the Program). The Program Logic Model outlines the primary Program activities and their expected outputs, as well as the expected immediate, intermediate and long-term outcomes of the Program activities. The Health Expert Survey: Mark Swain The case study interviews: Lynn Schindel ensured that potential social issues were addressed by the tools. Also reviewed by the case study sites. Community-Based Survey: Regional Program staff ( process didn’t work in terms of achieving the preferred response rate) Researcher Survey: CIHR (positive and negative aspects) This engagement of the stakeholders likely contributed to the high response rates exhibited in most cases.

Sample Selection Because this was a formative evaluation, participants from many groups were to be surveyed to gain a broad overview of the Program Purposive sampling best addressed the need to collect data from populations of an unknown size (e.g., “health experts”) As resources were limited, this sampling approach was appropriate Key informants were used to identify potential participants: Health Canada – National and regional Program staff and other stakeholders at the national/ regional levels Hepatitis C experts – Health Expert Survey Canadian Institutes for Health Research (research fund manager) – Researcher Survey

Sample Selection (cont’d) Local level Case studies in 7 sites across Canada to: Access those infected with/affected by hepatitis C, community-based support projects funded by the Program, other hepatitis C service providers in the community Explore Program implementation at the micro level Methodology based on the work of theorists (Yin & Chelimsky) Sites selected in consultation with Health Canada staff and based on criteria such as regional representation, service to a priority population, project type and willingness to participate The case studies were valuable because they: Provided a sense of how the Program plays out at the community level; Provided insight into social issues in the community; Allowed for access to other community agencies (and individuals such as the Lethbridge pharmacist) that might not otherwise have been identified; Allowed us to strengthen our conclusions (e.g., clarifying info re: “doctors don’t understand my needs”); Allowed for the discovery of info that might not otherwise have been uncovered (e.g., Hepatitis C Society supported at the national level but falling apart at the community level); and Provided important insight into building community capacity (e.g., cannot give money to community groups without support for infrastructure, board development, etc).

Data Collection Reviewed Program documents Survey/interview data collection proceeded smoothly due to: Support of stakeholders Respectful treatment of participants Understanding of community agencies Multiple response options for each instrument (in-person or phone interviews; e-mail, mail or fax surveys) Thank you cards, gifts to the participating case study sites and incentives (grocery coupons) to clients interviewed

Analysis & Write-Up Used SPSS and N-Vivo for data analysis Utility of the findings kept in mind—”What does this Program need?” Data Collection Matrix used to guide qualitative analysis and the “triangulation” of all data sources All data sources that addressed an evaluation question were compiled in a Data Summary Similarities and differences across groups of respondents were compared Findings that are included in the final report represent themes that stood out across all groups of respondents and/or documents To reduce the potential for bias and to obtain as balanced a picture as possible, data were collected from multiple sources to address each evaluation question.

Analysis & Write-Up (cont’d) Case studies: Research team met to draft a report template (series of questions) Team leaders organized case study data using the report template Completed template given to Project Director for case study write-up Preliminary review of each case by research team Review of each case by site coordinator Case study signed-off by site prior to distribution to Health Canada Health Canada regional staff reviewed and made final changes Process increased stakeholder ownership of the findings Final Report: Guided by the Logic Model for report structure Peer reviewers helped clarify & focus the report Various national and regional stakeholders reviewed the draft report Brought in Bill Reeves for an external perspective at a late stage. Took us a step back but was necessary due to the complexity of the study. Bill also had extensive expertise, for example, in training requirements for physicians. His comments helped us to interpret some of the findings, particularly around community capacity.

Lessons Learned The evaluation identified lessons learned and highlighted Program strengths and weaknesses Through these insights, the Evaluators were able to propose 19 recommendations Health Canada staff were consulted on the wording of the recommendations to facilitate their implementation