An Integrated Approach to Performance Measurement, Evaluation and Risk Monitoring of S&T Programs Presented to: AEA Evaluation Conference November 2, 2006 Presented by: George Teather Performance Management Network Inc. Tel: (613) 236-2320 ext. 231 Fax: (613) 236-8644 Email: george.teather@pmn.net

Presentation Outline Discuss Canadian integrated approach to performance measurement, evaluation and risk monitoring for funded programs Present a specific example of the process, as it applies to the Stem Cell Network (SCN)

Overview Many S&T programs are funded on a 3-5 year cycle, and have to go back to central agencies (Treasury Board) for continued funding. Renewal involves examination of performance over the funding period, often includes an evaluation Most new programs are required to develop a “Results-based Management and Accountability Framework” (RMAF) as part of the initial funding process. An RMAF describes the program’s objective, the design and delivery strategy, identifies how the performance will be monitored and reported on annually, and identifies the evaluation questions that will be addressed and the approach that will be used to conduct the evaluation. A Risk Based Audit Framework (RBAF) is also developed to identify major risks that may impact on project success, describe how to monitor those risks and describe strategies for managing and mitigating those risks.

RMAF Table of Contents Profile – summary description of objectives, design and delivery, intended outcomes Governance – description of organization, responsibilities for accountability and reporting Logic Model Performance Measurement and Reporting Strategy – performance measures, indicators, sources of information and reporting schedule Evaluation Strategy – identification of issues (relevance, objectives achievement and design and delivery alternatives / improvements), evaluation methodology, and schedule

Program Management, Measurement and Evaluation Lifecycle Design an Intervention/ Develop Strategic Plan Reconsider, redesign, expand, reduce or end Initial Situation Program Design and Delivery Program delivery Identify a need Program or policy start-up Outcome or Summative Evaluation Direct Outcomes Intermediate Outcomes Final Outcomes Mid-term or Formative Baseline Performance Measurement and Evaluation RMAF RBAF Risk Assessment Ongoing Performance Measurement, Program Management Ongoing Risk Monitoring

The Stem Cell Network SCN is one of a number of programs under the National Centres of Excellence initiative funded by Industry Canada, to develop scientific networks in specific targeted areas to support scientific development and innovation leading to economic growth and benefits to society Funding of $5.3 million / yr for an initial seven year period SCN managed by a Board of Directors, provide annual reports on performance to NCE program office RMAF developed to support the information needs of Board of Directors, NCE and federal government: Identify information to support ongoing operational management by the Board of Directors, and provide the basis for annual performance review and reporting (accountability) Support periodic in-depth examination of the extent to which the program is achieving its intended outcomes and objectives

SCN Program Description - Activities Seek proposals, select and fund research linked to clinical, commercial or policy objectives Fund initiatives to attract, develop and retain stem cell researchers within the Canadian system Disseminate and exploit knowledge and technology developed under SCN Develop and support networks and partnerships within Canadian and international stem cell community and with those applying SCN knowledge and technology Provide strategic and operational program oversight and management

Generic Program Logic Model Program Objective: high level strategic purpose Resources Reach Results HOW? WHO / WHERE? WHAT do we want? WHY? users / clients / co-deliverers / beneficiaries activities outputs direct outcomes intermediate outcomes ultimate impacts Program / Service Delivery Client Management Policy & Issue Management Financial Management Human Resources Management Asset Management Program deliverables Policy guidelines, regulations Communications plans - internal communications - promotion - info transfer - consultations - meetings/events Funding Service Outputs Primary Targets (clients, ultimate beneficiaries) Co-delivery Agents Other Stakeholders Client Service - addresses needs - meets / exceeds expectations - service quality Behavioral Influence awareness understanding attitude / perception support New knowledge Improved capability Improved decision making Target group changes in behaviour / other outcomes Sector / Industry / Regional Impact Economic/ Environmental/ Societal Impact Contribution to organizational objective

Spheres of Influence* State Behavioural Change Operational Time Your environment of indirect influence e.g., Industrial sectors, government decision makers, other communities of interest where you do not make direct contact Behavioural Change Your environment of direct influence e.g. people and groups in direct contact with your programs, staff (i.e. clients, target audience, co-delivery partners Time Operational Your operational environment You have direct control over the behaviours within this sphere *reference S. Montague, www.pmn.net

Stem Cell Network Results Logic SCN Operations (control) Stem Cell Network Management Activities and Outputs Networking and Partnership Development Research HQP / Training Knowledge and Technology Exchange and Exploitation Awareness, cooperation, participation, engagement of Network partners and stakeholders Immediate Outcomes Technology Exploitation Improved PI entrepreneurial capability Protection and transfer of IP Research / HQP Well managed program High quality research results relevant to partners Increased research capacity Training of HQP Public Policy / Regulatory / Framework SC research findings contribute to discussions and decisions Public awareness of SC research findings SCN Key Partners (direct influence) Intermediate Outcomes

Stem Cell Network Results Logic SCN Operations (control) Stem Cell Network Management Activities and Outputs Networking and Partnership Development Research HQP / Training Knowledge and Technology Exchange and Exploitation Awareness, cooperation, participation, engagement of Network partners and stakeholders Immediate Outcomes Technology Exploitation Improved PI entrepreneurial capability Protection and transfer of IP Research / HQP Well managed program High quality research results relevant to partners Increased research capacity Training of HQP Public Policy / Regulatory / Framework SC research findings contribute to discussions and decisions Public awareness of SC research findings SCN Key Partners (direct influence) Intermediate Outcomes Development of key enabling technologies Translation of research into clinical applications and commercial products Attract, train and retain stem cell researchers and entrepreneurs in Canada Development of ethical and legal stem cell research framework that has public support SCN Stakeholders (contributing influence) Realizing the Full Impact of Stem Cell Research for Canadians Societal Impacts Improved treatment of illness and disease using stem cell therapies Improved health and quality of life for Canadians Economic Impacts Growth in Canadian firms commercializing stem cell research New jobs created in Canada outside the network Final Outcomes Marketplace / Society (contributing influence)

Performance Measures Resources Outputs – knowledge creation Annual expenditures, % from other sources (leveraging) Allocation among program activity components Outputs – knowledge creation Number and type of publications and reports generated from SCN funded projects Immediate Outcomes Awareness, cooperation, participation and engagement (networking and collaborations, interactions with other groups) Research (achievement of milestones, quality and relevance, training of new researchers) Technology Exploitation (improved entrepreneurial capability, protection and transfer of IP) Public Policy / Regulatory Support (input to public policy, public awareness of reports)

Performance Measures (continued) Intermediate Outcomes Development of key technologies (patent applications, patents) Transformation of research into clinical applications and commercial products (licenses, nondisclosure agreements) Attract, train and retain stem cell researchers (SCN funded students graduating and staying in Canada) Public supported ethical and legal stem cell research framework (evidence of contribution) Final Outcomes- measured in evaluation only, not annually Economic growth and productivity (profile of firms commercializing SCN research, effect on existing industry) Societal impacts / health and well being (examples of contributions) SCN Management, interactions with other groups) Board and stakeholder support for program delivery and performance monitoring strategy Management and delivery of SCN program (use of RMAF, level of participation, % of total costs for admin, overall success) Internal communication (SCN publication frequency, level of participation at AGM)

Evaluation Issues Relevance Objectives Achievement contribution to meeting needs of SCN relevance to achievement of NCE and federal objectives for innovation and economic and social benefits Objectives Achievement Engagement of target communities (researchers, partners and stakeholders) Project level outcomes (technologies, products, processes, networking) Program level outcomes (technologies, clinical and commercial applications, retention of HQP) Contribution to final outcomes (improved treatment of disease, health and quality of life and economic growth of stem cell related firms) Examination of unintended outcomes Alternatives / Design and Delivery Factors facilitating or impeding objectives achievement Changes to improve efficiency, effectiveness, level of success

Evaluation Methodology (multiple lines of evidence) Review and analysis of internal and external documents and reports Surveys of researchers, partners and stakeholders (following prepared questionnaire) In-depth interviews of selected researchers, partners and stakeholders (in-person or telephone, following open ended interview guide) Case studies of selected projects (to examine relationships, outcomes, barriers, sector and societal impacts) Cost / benefit analysis Benchmark studies comparing program to other similar programs Integration and analysis of evidence collected on each issue

Summary RMAF is an integrated program management tool to support operational and strategic program management and meet the funding agencies requirements for accountability and outcomes / impact information RMAF identifies evaluation issues at the beginning of the program well ahead of implementation and communicates evaluation purpose and approach to both performers and funders Critical to have stakeholders participate in development rather than have it developed by consultants separately and imposed In the case of SCN, RMAF is being used by Board of Directors as an important source to identify program performance information. Reporting templates have been developed based on the RMAF

Bibliography Treasury Board Website – RMAF / RBAF www.tbs-sct.gc.ca/eval/eval_e.asp Stem Cell Network Website www.scn.ca Performance Management Network www.pmn.net