The CIHR Perspective on Impact Measurement Presentation to International Healthcare Funders March 30, 2015 David Peckham Chief Audit and Evaluation Executive Canadian Institutes of Health Research

Introduction The collection of attributable output and outcome data is challenging and can require the use of considerable resources funding organizations struggle with finding feasible and sustainable methods for measuring and tracking the impact of their investments in supporting health research

Informing decision-making Broad Economic and Social Benefits CAHS Impact Framework Canadian Academy of Health Sciences Report Making an Impact (2009) identifies 5 impact categories Advancing Knowledge Capacity Building Informing decision-making Health Benefits Broad Economic and Social Benefits David: The second tool was the identification of five impact categories. Looking at the pathway five impacts can result from investments and support pending the purpose of the assessment and the funding mechanism: David thought you could summarize what you have later on for CIHR the key characteristics of each: Advancing knowledge refers to Capacity Building to Informing Decision Making to Health Benefits to Broad Economic and Social Benefits

CIHR Performance Measurement Regime ‘Toolbox’ of around 80 measures available (down from several hundred) Based on CAHS Framework impact categories Every CIHR funding initiative required to have a performance measurement strategy Flexibility – comparability but also some latitude David:

CIHR Toolbox Impact Category Descriptions Sample of Indicators Advancing knowledge This category includes discoveries/breakthroughs, contributions to the scientific literature (grey or peer-reviewed) and may include measures of research quality, activity, outreach and structure. Number of scientific products produced through CIHR funded research Number of reports/technical reports produce by CIHR-funded grants Building capacity This category includes the development and enhancement of research skills in individuals and teams, additional research-activity funding (e.g., funding partnerships) as well as the development/enhancement of platforms. Number of research collaborations among CIHR funded grants and awards Total number of CIHR grants supporting teams Informing decision-making This category includes the impacts of health research in the areas of health related decision making (i.e., science/research, general public, clinical and managerial decision-making, practice and policy and health products decision making). Percentage of CIHR grants reporting citation by others Field analysis of citations David:

CIHR Toolbox - Impact Category Descriptions Sample of Indicators Health impacts This category includes advances in the prevention, diagnosis, treatment and palliation as well as changing health status and determinants of health. Potential years of life lost in Canada Community well-being index Health system impacts This category includes advances and efficiencies in the way the health system functions. Mortality rate from treatable causes in Canada (age-standardized per 100,000 population) Hospital readmission rates in Canada Broad economic and social impacts This category includes commercialization of research discoveries, human capital gains, health benefits (specific costs of implementing research findings in the broad health system), well-being and social benefit indicators. Total number of staff being paid through CIHR grants Economic productivity levels of sick people in Canada David:

Examples of CIHR Baseline Data Measure Baseline Data 2012-13 Actual Data 2013-14 Trend Percentage of researchers reporting the creation of new knowledge 92% Number of applications received 18,802 16,851 Percentage of fundable applications 69% Co-author Analysis   • 13,030 international collaborations (49% collaboration rate); • 3370 interprovincial collaborations (12% collaboration rate); • 8535 intersectoral collaborations (32% collaboration rate) • 13,581 international collaborations (51% collaboration rate) • 3,437 interprovincial collaborations (13% collaboration rate) • 8,959 intersectoral collaborations (33% collaboration rate) Number of highly cited publications 828 (of a total of 7,341 – based on data for 2012, drawn in 2013) 860 (of a total of 7,279 – based on data for 2013, drawn in 2014) Percentage of papers acknowledging CIHR that are highly cited 11.3% 11.8%

Examples of CIHR Baseline Data Indicator Impact Category Baseline 2012-13 Actual 2013-14 Ratio of leveraged-funds for Institute-Driven Initiatives BC $1:$0.36 $1:$0.42 Ratio of leverage-funds for Horizontal Initiatives $1 : $0.56 $1:$0.76 Canada's international ranking for the number of health researchers per thousand workforce 3rd among G7 External Data source available every 5 years Percentage of Institute-Driven grants reporting stakeholder involvement in the research process IDM 89% 94% Percentage of operating grants reporting translation of knowledge 57% 61% Percentage of priority-driven grants reporting translation of knowledge N/A 68%

Lessons Learned after 12 months Measures in the Broad Social and Economic Benefits and Health and Health System Impact Categories have less easily available data and there are concerns over attribution Data availability can be an issue if relying on external sources – e.g. national statistical offices Further streamlining may be possible Example – Stakeholder engagement – as part of a grant this could be a building capacity item, however, when you look at the type of stakeholder involved such as Policy experts, you are looking to inform decision making