1. The CIHR Perspective on Impact Measurement
Presentation to International Healthcare Funders
March 30, 2015
David Peckham
Chief Audit and Evaluation Executive
Canadian Institutes of Health Research

2. Introduction
The collection of attributable output and outcome data is challenging and can require the use of considerable resources
funding organizations struggle with finding feasible and sustainable methods for measuring and tracking the impact of their investments in supporting health research

3. Informing decision-making Broad Economic and Social Benefits
CAHS Impact Framework
Canadian Academy of Health Sciences Report Making an Impact (2009) identifies 5 impact categories
Advancing Knowledge
Capacity Building
Informing decision-making
Health Benefits
Broad Economic and Social Benefits
David: The second tool was the identification of five impact categories. Looking at the pathway five impacts can result from investments and support pending the purpose of the assessment and the funding mechanism: David thought you could summarize what you have later on for CIHR the key characteristics of each:
Advancing knowledge refers to
Capacity Building to
Informing Decision Making to
Health Benefits to
Broad Economic and Social Benefits

4. CIHR Performance Measurement Regime
‘Toolbox’ of around 80 measures available (down from several hundred)
Based on CAHS Framework impact categories
Every CIHR funding initiative required to have a performance measurement strategy
Flexibility – comparability but also some latitude
David:

5. CIHR Toolbox Impact Category Descriptions
Sample of Indicators
Advancing knowledge
This category includes discoveries/breakthroughs, contributions to the scientific literature (grey or peer-reviewed) and may include measures of research quality, activity, outreach and structure.
Number of scientific products produced through CIHR funded research
Number of reports/technical reports produce by CIHR-funded grants
Building capacity
This category includes the development and enhancement of research skills in individuals and teams, additional research-activity funding (e.g., funding partnerships) as well as the development/enhancement of platforms.
Number of research collaborations among CIHR funded grants and awards
Total number of CIHR grants supporting teams
Informing decision-making
This category includes the impacts of health research in the areas of health related decision making (i.e., science/research, general public, clinical and managerial decision-making, practice and policy and health products decision making).
Percentage of CIHR grants reporting citation by others
Field analysis of citations
David:

6. CIHR Toolbox - Impact Category Descriptions
Sample of Indicators
Health impacts
This category includes advances in the prevention, diagnosis, treatment and palliation as well as changing health status and determinants of health.
Potential years of life lost in Canada
Community well-being index
Health system impacts
This category includes advances and efficiencies in the way the health system functions.
Mortality rate from treatable causes in Canada (age-standardized per 100,000 population)
Hospital readmission rates in Canada
Broad economic and social impacts
This category includes commercialization of research discoveries, human capital gains, health benefits (specific costs of implementing research findings in the broad health system), well-being and social benefit indicators.
Total number of staff being paid through CIHR grants
Economic productivity levels of sick people in Canada
David:

7. Examples of CIHR Baseline Data
Measure
Baseline Data
Actual Data
Trend
Percentage of researchers reporting the creation of new knowledge
92%
Number of applications received
18,802
16,851
Percentage of fundable applications
69%
Co-author Analysis
• 13,030 international collaborations (49% collaboration rate);
• interprovincial collaborations (12% collaboration rate);
• intersectoral collaborations (32% collaboration rate)
• 13,581 international collaborations (51% collaboration rate)
• 3,437 interprovincial collaborations (13% collaboration rate)
• 8,959 intersectoral collaborations (33% collaboration rate)
Number of highly cited publications
828 (of a total of 7,341 – based on data for 2012, drawn in 2013)
860 (of a total of 7,279 – based on data for 2013, drawn in 2014)
Percentage of papers acknowledging CIHR that are highly cited
11.3%
11.8%

8. Examples of CIHR Baseline Data
Indicator
Impact Category
Baseline
Actual
Ratio of leveraged-funds for Institute-Driven Initiatives BC
$1:$0.36
$1:$0.42
Ratio of leverage-funds for Horizontal Initiatives
$1 : $0.56
$1:$0.76
Canada's international ranking for the number of health researchers per thousand workforce
3rd among G7
External Data source available every 5 years
Percentage of Institute-Driven grants reporting stakeholder involvement in the research process
IDM
89%
94%
Percentage of operating grants reporting translation of knowledge
57%
61%
Percentage of priority-driven grants reporting translation of knowledge
N/A
68%

9. Lessons Learned after 12 months
Measures in the Broad Social and Economic Benefits and Health and Health System Impact Categories have less easily available data and there are concerns over attribution
Data availability can be an issue if relying on external sources – e.g. national statistical offices
Further streamlining may be possible
Example – Stakeholder engagement – as part of a grant this could be a building capacity item, however, when you look at the type of stakeholder involved such as Policy experts, you are looking to inform decision making