1 The RRFSS Evaluation Framework 2005 Background Information and Progress to Date Presenters  Amira Ali  Jane Hohenadel  Kate O’Connor  Deborah Radcliffe June 22, 2005

2 Group Members  RRFSS Evaluation Group: Amira Ali, Elaine Hector, Jane Hohenadel, Irene Koren, Kathy Moran, Karen Moynagh, Lynne Russell, Louisa Wong  PHRED RRFSS Evaluation Group: Jane Hohenadel, Adriana Newbury, Kate O’Connor, Louise Picard, Deborah Radcliffe

3 Outline of Presentation  Link to strategic planning session  Link to previous evaluation (2002)  Evaluation Framework – RRFSS Evaluation 2005  Tasks and Timelines

4 History  Project was identified in March 2004  Proposal developed in May 2004 and an Evaluation committee was formed with representation from each of the RRFSS regional working groups  The PHREDs were identified as partners in the project – proposal submitted to them in October 2004  Terms of reference for the RRFSS group and the PHRED group were finalized in December 2004

5 Link to Strategic Planning Session

6 Strategic Planning Session  Strategic Planning Session in December 2004 To develop strategic statements (vision, mission, values, goals, strategies) that will guide the collective RRFSS work over the next 3 years To develop strategic statements (vision, mission, values, goals, strategies) that will guide the collective RRFSS work over the next 3 years To determine the specific objectives and activities for the next year To determine the specific objectives and activities for the next year To create an action document summarizing the directions set To create an action document summarizing the directions set

7 Strategic Planning Session  Vision: RRFSS envisions that all decisions within the public health system to promote and protect health and wellbeing and prevent adverse health events are informed by valid, timely and relevant health intelligence  Mission: Ontario’s RRFSS is a flexible, timely and responsive surveillance system designed to meet local Public Health intelligence needs and address information, geographic and time- related data gaps

8 Strategic Planning Session  Four strategies were identified: 1.Advocacy 2.Communication 3.Governance 4.Quality Improvement and Development

9 Strategic Planning Session  Strategy: Quality Improvement and Development Priority Activity: “Conduct the RRFSS 2005 Evaluation to identify areas of the surveillance system in need of improvement” The full report “Strategic Plan for Action” can be viewed at:

10 Link to RRFSS Evaluation 2002

11 RRFSS Evaluation 2002  Covered first year of operation (2001)  Number of health units ranged from 6 to 12 during the evaluation period  Purpose:  Inform change to system procedures  Contribute to continuous quality improvement  Act as a baseline and template for future evaluations The full evaluation report can be viewed at: The full evaluation report can be viewed at:

12 RRFSS Evaluation 2002  Evaluation Framework covered 4 areas:  Process Effectiveness  Collaboration Effectiveness  Usefulness  Cost Effectiveness

13 RRFSS Evaluation 2002  Performance criteria linked to following outcomes expected from RRFSS:  Increased local awareness of surveillance information  Increased involvement in generating appropriate, accurate information  Increased practice of evidence-based program planning  Demonstrated or potential cost benefit due to RRFSS

14 RRFSS Evaluation 2002 Fourteen Key Issues Identified (grouped as below for 2005 evaluation)  Local Resources (2)  Centralization Issues Provincial Implications (3) Provincial Implications (3) Implications for ISR (4) Implications for ISR (4)  Process Considerations (5)

15 Key Issues from 2002 Evaluation Local Resources  Adequate staff/resources for data analysis and making data useable  RRFSS coordinators overwhelmed by work at the local level

16 Key Issues from 2002 Evaluation Centralization – Provincial Implications  Need for stability and support for the central work of the RRFSS partnership  Sustaining funding for existing and new participants (smaller health units in particular); includes funding for survey administration and analysis  Provincial representation of data

17 Key Issues from 2002 Evaluation Centralization – Implications for ISR  Increasing the sample size for a health unit  Access to ISR, decision-making, workload and resources  Improved timeliness of data from ISR  ISR Ability to handle increase in HU participation

18 Key Issues from 2002 Evaluation Process Consideration Issues  Increasing use of data in program planning and evaluation in the health units  Limited new question/module capacity under current questionnaire  Process for revising old modules and developing new modules  Upgrading RRFSS website  Sharing data among organizations

19 RRFSS Evaluation 2005

Evaluation Framework Objectives 1. To document progress made in the key areas identified in the 2002 Evaluation 2. To document some of the “best practices” used by Health Units 3. To envisage what it would take to get RRFSS on board for all Ontario Health Units and how it could work as a provincial system

Evaluation Framework Refinement of the Key Issues  To address the evaluation objectives, we began with what were identified as Key Issues in the 2002 Evaluation  Evaluation group members were asked to review each of those 14 Key issues, and: report on progress report on progress whether it was still an issue now whether it was still an issue now if so, how important it was for the 2005 evaluation if so, how important it was for the 2005 evaluation

Evaluation Framework Refinement of the Key Issues  It was felt that seven issues had been largely addressed, and that the 2005 evaluation needed only to document their evolution since 2002  The issues that we felt were addressed: increasing the sample size for a health unit increasing the sample size for a health unit improved timeliness of data from ISR improved timeliness of data from ISR access to ISR, decision-making, workload and resources, and access to ISR, decision-making, workload and resources, and limited new question/module capacity limited new question/module capacity

Evaluation Framework Refinement of the Key Issues  The issues that we felt were addressed (cont’d): limited new questionnaire/module capacity under current questionnaire limited new questionnaire/module capacity under current questionnaire process for revising old modules and developing new modules process for revising old modules and developing new modules upgrading RRFSS website upgrading RRFSS website

Evaluation Framework Refinement of the Key Issues  The remainder were felt to still be issues for the RRFSS partnership and should remain in the 2005 evaluation  The outstanding issues were refined, defined and expressed as evaluation questions

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Local Resources Local ResourcesQuestion Do HUs have adequate staff and resources for regional and local administrative tasks of RRFSS? Do HUs have adequate staff and resources for regional and local administrative tasks of RRFSS? Data Collection Tool Questionnaire Source of Data RRFSS reps, MOH of participating HU

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Local Resources Question Do HUs have adequate staff and resources locally for data analysis and making data useable? Data Collection Tool Questionnaire Source of Data RRFSS reps, MOH of participating HU

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Local Resources Question Are local RRFSS reps able to meet the local work demand for RRFSS module development? Data Collection Tool Questionnaire Source of Data RRFSS reps, MOH of participating HU, program and research staff, RRFSS coordinator, RRFSS Evaluation Group, ISR

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Local Resources Question Are local RRFSS reps able to meet the local work demand for RRFSS module development? Data Collection Tool Questionnaire Source of Data RRFSS reps, MOH of participating HU, program and research staff, RRFSS coordinator, RRFSS Analysis Group, ISR

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Local Resources Question Are local RRFSS reps able to meet the local work demand for RRFSS module review? Data Collection Tool Questionnaire Source of Data RRFSS reps, MOH of participating HU, program and research staff, RRFSS coordinator, RRFSS Analysis Group, ISR

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Centralization Issues: Provincial Implications Question What resources are needed currently to support and to ensure the stability of the provincial and administrative work of the RRFSS Partnership? Data Collection Tool Questionnaire Source of Data RRFSS reps, MOH of participating HU, ISR, RRFSS Coordinator

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Centralization Issues: Provincial Implications Question What funds are required to enable continued participation of HUs currently participating in RRFSS and allow non-participating HUs to join? Data Collection Tool Key Informant Interviews Source of Data RRFSS reps, MOH of all HUs, ISR, MOHLTC

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Centralization Issues: Provincial Implications Question Is provincial representation of data a desired product of RRFSS? Data Collection Tool Key Informant Interviews Source of Data RRFSS reps, MOH of all HUs, MOHLTC, PHRED, Chief MOH, external partners

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Centralization Issues: Provincial Implications Question What would it take to extend RRFSS to all Ontario HUs? Data Collection Tool Key Informant Interviews Source of Data RRFSS reps, epidemiologists in non-participating HUs, MOHLTC, ISR, Chief MOH, MOH of all HUs

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Process Considerations Question To what extent are RRFSS data being used in PP&E in the health units? Data Collection Tool Questionnaire, Case Examples Source of Data RRFSS reps, program and research staff

Evaluation Framework Evaluation Questions for Outstanding Issues Issue Process Considerations Process ConsiderationsQuestion To what extent are provincial RRFSS data being shared among agencies and organizations? Data Collection Tool Questionnaire, Key Informant Interview Source of Data RRFSS Coordinator, ISR, RRFSS reps

36 Evaluation Framework for RRFSS Evaluation Questions for Outstanding Issues Issue Process Considerations Process ConsiderationsQuestion To what extent are health unit specific RRFSS data being shared among agencies and organizations? Data Collection Tool Questionnaire Source of Data RRFSS reps

Evaluation Framework Evaluation Question for Objective 2 Issue Process Considerations Question What Success Stories can be identified for RRFSS? Data Collection Tool Workshop, Questionnaire Source of Data RRFSS reps, program and research staff

Evaluation Framework Evaluation Question Issue Process Considerations (Parked Issue) Question What is the quality of RRFSS data? Data Collection Tool Questionnaire Source of Data RRFSS reps, ISR, partners

39 Tasks and Timelines

40 RRFSS Evaluation Tasks  Finalize development of research tools. Questionnaire Questionnaire Focus groups Focus groups Key informant interviews Key informant interviews Document reviews Document reviews  Data collection: RRFSS Workshop-- “Success stories” RRFSS Workshop-- “Success stories” Primarily August and September Primarily August and September  Data Analysis  Draft and finalize the report

41 RRFSS Evaluation Timelines

42