1. An Integrated Approach to Performance Measurement, Evaluation and Risk Monitoring of S&T Programs
Presented to:
AEA Evaluation Conference
November 2, 2006
Presented by:
George Teather
Performance Management Network Inc.
Tel:
(613) ext. 231
Fax:
(613)

2. Presentation Outline
Discuss Canadian integrated approach to performance measurement, evaluation and risk monitoring for funded programs
Present a specific example of the process, as it applies to the Stem Cell Network (SCN)

3. Overview
Many S&T programs are funded on a 3-5 year cycle, and have to go back to central agencies (Treasury Board) for continued funding. Renewal involves examination of performance over the funding period, often includes an evaluation
Most new programs are required to develop a “Results-based Management and Accountability Framework” (RMAF) as part of the initial funding process.
An RMAF describes the program’s objective, the design and delivery strategy, identifies how the performance will be monitored and reported on annually, and identifies the evaluation questions that will be addressed and the approach that will be used to conduct the evaluation.
A Risk Based Audit Framework (RBAF) is also developed to identify major risks that may impact on project success, describe how to monitor those risks and describe strategies for managing and mitigating those risks.

4. RMAF Table of Contents
Profile – summary description of objectives, design and delivery, intended outcomes
Governance – description of organization, responsibilities for accountability and reporting
Logic Model
Performance Measurement and Reporting Strategy – performance measures, indicators, sources of information and reporting schedule
Evaluation Strategy – identification of issues (relevance, objectives achievement and design and delivery alternatives / improvements), evaluation methodology, and schedule

5. Program Management, Measurement and Evaluation Lifecycle
Design an
Intervention/ Develop Strategic Plan
Reconsider,
redesign, expand,
reduce or end
Initial Situation
Program Design and Delivery
Program delivery
Identify
a need
Program or policy start-up
Outcome or
Summative
Evaluation
Direct Outcomes
Intermediate Outcomes
Final Outcomes
Mid-term or
Formative
Baseline
Performance Measurement
and Evaluation
RMAF
RBAF
Risk Assessment
Ongoing Performance Measurement, Program Management
Ongoing Risk Monitoring

6. The Stem Cell Network
SCN is one of a number of programs under the National Centres of Excellence initiative funded by Industry Canada, to develop scientific networks in specific targeted areas to support scientific development and innovation leading to economic growth and benefits to society
Funding of $5.3 million / yr for an initial seven year period
SCN managed by a Board of Directors, provide annual reports on performance to NCE program office
RMAF developed to support the information needs of Board of Directors, NCE and federal government:
Identify information to support ongoing operational management by the Board of Directors, and provide the basis for annual performance review and reporting (accountability)
Support periodic in-depth examination of the extent to which the program is achieving its intended outcomes and objectives

7. SCN Program Description - Activities
Seek proposals, select and fund research linked to clinical, commercial or policy objectives
Fund initiatives to attract, develop and retain stem cell researchers within the Canadian system
Disseminate and exploit knowledge and technology developed under SCN
Develop and support networks and partnerships within Canadian and international stem cell community and with those applying SCN knowledge and technology
Provide strategic and operational program oversight and management

8. Generic Program Logic Model
Program Objective: high level strategic purpose
Resources
Reach
Results
HOW?
WHO / WHERE?
WHAT do we want? WHY?
users / clients / co-deliverers / beneficiaries
activities
outputs
direct outcomes
intermediate outcomes
ultimate impacts
Program / Service Delivery
Client Management
Policy & Issue Management
Financial Management
Human Resources Management
Asset Management
Program deliverables
Policy guidelines, regulations
Communications
plans
- internal communications
- promotion
- info transfer
- consultations
- meetings/events
Funding
Service Outputs
Primary Targets (clients, ultimate beneficiaries)
Co-delivery Agents
Other Stakeholders
Client Service
- addresses needs
- meets / exceeds expectations
- service quality
Behavioral Influence
awareness
understanding
attitude / perception
support
New knowledge
Improved capability
Improved decision making
Target group changes in behaviour / other outcomes
Sector / Industry / Regional Impact
Economic/ Environmental/ Societal Impact
Contribution to organizational objective

9. Spheres of Influence* State Behavioural Change Operational Time
Your environment of indirect influence
e.g., Industrial sectors, government decision makers, other
communities of interest where you do not
make direct contact
Behavioural Change
Your environment of direct influence
e.g. people and groups in direct contact with your programs, staff (i.e. clients, target audience, co-delivery partners
Time
Operational
Your operational environment
You have direct control over the behaviours within this sphere
*reference S. Montague,

10. Stem Cell Network Results Logic
SCN Operations
(control)
Stem Cell Network Management
Activities and Outputs
Networking and Partnership Development
Research
HQP / Training
Knowledge and Technology Exchange and Exploitation
Awareness, cooperation, participation, engagement of Network partners and stakeholders
Immediate Outcomes
Technology Exploitation
Improved PI entrepreneurial capability
Protection and transfer of IP
Research / HQP
Well managed program
High quality research results relevant to partners
Increased research capacity
Training of HQP
Public Policy / Regulatory / Framework
SC research findings contribute to discussions and decisions
Public awareness of SC research findings
SCN
Key Partners
(direct influence)
Intermediate Outcomes

11. Stem Cell Network Results Logic
SCN Operations
(control)
Stem Cell Network Management
Activities and Outputs
Networking and Partnership Development
Research
HQP / Training
Knowledge and Technology Exchange and Exploitation
Awareness, cooperation, participation, engagement of Network partners and stakeholders
Immediate Outcomes
Technology Exploitation
Improved PI entrepreneurial capability
Protection and transfer of IP
Research / HQP
Well managed program
High quality research results relevant to partners
Increased research capacity
Training of HQP
Public Policy / Regulatory / Framework
SC research findings contribute to discussions and decisions
Public awareness of SC research findings
SCN
Key Partners
(direct influence)
Intermediate Outcomes
Development of key enabling technologies
Translation of research into clinical applications and commercial products
Attract, train and retain stem cell researchers and entrepreneurs in Canada
Development of ethical and legal stem cell research framework that has public support
SCN Stakeholders
(contributing influence)
Realizing the Full Impact of Stem Cell Research for Canadians
Societal Impacts
Improved treatment of illness and disease using stem cell therapies
Improved health and quality of life for Canadians
Economic Impacts
Growth in Canadian firms commercializing stem cell research
New jobs created in Canada outside the network
Final Outcomes
Marketplace / Society
(contributing influence)

12. Performance Measures Resources Outputs – knowledge creation
Annual expenditures, % from other sources (leveraging)
Allocation among program activity components
Outputs – knowledge creation
Number and type of publications and reports generated from SCN funded projects
Immediate Outcomes
Awareness, cooperation, participation and engagement (networking and collaborations, interactions with other groups)
Research (achievement of milestones, quality and relevance, training of new researchers)
Technology Exploitation (improved entrepreneurial capability, protection and transfer of IP)
Public Policy / Regulatory Support (input to public policy, public awareness of reports)

13. Performance Measures (continued)
Intermediate Outcomes
Development of key technologies (patent applications, patents)
Transformation of research into clinical applications and commercial products (licenses, nondisclosure agreements)
Attract, train and retain stem cell researchers (SCN funded students graduating and staying in Canada)
Public supported ethical and legal stem cell research framework (evidence of contribution)
Final Outcomes- measured in evaluation only, not annually
Economic growth and productivity (profile of firms commercializing SCN research, effect on existing industry)
Societal impacts / health and well being (examples of contributions)
SCN Management, interactions with other groups)
Board and stakeholder support for program delivery and performance monitoring strategy
Management and delivery of SCN program (use of RMAF, level of participation, % of total costs for admin, overall success)
Internal communication (SCN publication frequency, level of participation at AGM)

14. Evaluation Issues Relevance Objectives Achievement
contribution to meeting needs of SCN
relevance to achievement of NCE and federal objectives for innovation and economic and social benefits
Objectives Achievement
Engagement of target communities (researchers, partners and stakeholders)
Project level outcomes (technologies, products, processes, networking)
Program level outcomes (technologies, clinical and commercial applications, retention of HQP)
Contribution to final outcomes (improved treatment of disease, health and quality of life and economic growth of stem cell related firms)
Examination of unintended outcomes
Alternatives / Design and Delivery
Factors facilitating or impeding objectives achievement
Changes to improve efficiency, effectiveness, level of success

15. Evaluation Methodology (multiple lines of evidence)
Review and analysis of internal and external documents and reports
Surveys of researchers, partners and stakeholders (following prepared questionnaire)
In-depth interviews of selected researchers, partners and stakeholders (in-person or telephone, following open ended interview guide)
Case studies of selected projects (to examine relationships, outcomes, barriers, sector and societal impacts)
Cost / benefit analysis
Benchmark studies comparing program to other similar programs
Integration and analysis of evidence collected on each issue

16. Summary
RMAF is an integrated program management tool to support operational and strategic program management and meet the funding agencies requirements for accountability and outcomes / impact information
RMAF identifies evaluation issues at the beginning of the program well ahead of implementation and communicates evaluation purpose and approach to both performers and funders
Critical to have stakeholders participate in development rather than have it developed by consultants separately and imposed
In the case of SCN, RMAF is being used by Board of Directors as an important source to identify program performance information. Reporting templates have been developed based on the RMAF

17. Bibliography Treasury Board Website – RMAF / RBAF
Stem Cell Network Website
Performance Management Network