1. Evaluation at NRCan: Information for Program Managers Strategic Evaluation Division Science & Policy Integration July 2012

2. Purpose
The purpose of this document is to provide program managers with an overview of the evaluation function at NRCan.
The TBS Policy on Evaluation (April 2009) requires that all direct spending, including all G&C Programs, be evaluated every five years
most program managers will find themselves participating in an evaluation at some point.

3. What is Evaluation? Evaluations Audits
Evaluations are the systematic collection and analysis of evidence on the outcomes of programs to make judgments about
their relevance;
performance; and
alternative ways to deliver them or to achieve the same results.”
Evaluations must be neutral, and evidence-based.
An evaluation is not the same as an audit.
Evaluations
Audits
Focus on whether we are doing the right things, and the extent to which a program is achieving its expected outcomes, in a cost-effective manner.
Internal audit is looking at financial management, processes, controls and risk.
Make assessments on the relevance & performance of programs
Identify strengths and weaknesses in the management control framework

4. Why do Evaluations?
The objective of evaluation is to create a comprehensive and reliable base of evidence to support:
policy and program improvement;
expenditure management;
Cabinet decision-making; and
public accountability.
Evaluations are often required to support TB submissions and Memoranda to Cabinet.
They are also a critical source of evidence for Strategic Reviews, to support resource reallocation (next review will occur in 2014).

5. Background: Evaluation Stakeholders
Evaluation Reports
Policy & Program Improvement
Expenditure Management/
Cabinet Decision-making
Accountability/
Public Reporting
Cabinet
Parliament
NRCan DM
Strategic Review
Canadian Public
Sector ADMs
Program Managers
TBS
The challenge is to meet the diverse information needs of many stakeholders for decision-making AND public accountability in a timely fashion.

6. Some phases may overlap.
The Evaluation Cycle
Evaluation
Planning
Approvals/
Posting
(2-3 months)
Assessment
(1-3 months)
Report &
Recommendations
(1-4 months)
Implementing
Change
Contracting
(1-2 months)
Field Work/Analysis
(6-8 months)
Management Responses
(1 month)
Including planning and approvals/posting, large evaluations typically take months to complete.
Some phases may overlap.

7. Evaluation Planning
NRCan must evaluate all direct program spending, including all ongoing grant and contribution programs, every five years.
NRCan has developed a five-year Evaluation Plan based on PAA units that is updated annually and approved by the Evaluation Committee.
In most cases, the evaluation of an individual program will be conducted within the scope of a broader evaluation of a PAA unit.
The current plan summary appears on the Strategic Evaluation Internet site at:

8. Phases of an Evaluation
Evaluation Assessment (1-3 months)
research and planning to understand the programs
develop the Terms of Reference
obtain approval from the Evaluation Committee
Contracting (1-2 months)
Consultants are often used to supplement in-house staff. Their roles will vary by project.
Fieldwork or Data Collection/Analysis (6-8 months)
develop a detailed methodology report
methodologies: key informant interviews; focus groups; file/document/literature reviews; surveys; case studies; and data and economic analysis
analyse information collected from these multiple lines of evidence to develop conclusions

9. Phases of An Evaluation, cont’d
Reporting & Development of Recommendations (2-4 months)
prepare preliminary findings and discuss with programs
draft report
address comments and revisions
develop recommendations
Management Responses (1 month)
obtain ADM-approved management responses and action
plans to the recommendations
Approvals/Posting of report (2-3 months)
recommendation by the Evaluation Committee
approval by the DM
translation, ATIP review, media lines, release on Internet

10. Evaluation Questions and Issues
Evaluations address relevance and performance.
Relevance issues focus on:
continued need for program;
alignment with government priorities; and
alignment with federal roles and responsibilities.
Performance issues focus on effectiveness:
achievement of expected outcomes; and
demonstration of efficiency and economy.
Evaluators work with program managers to develop more detailed evaluation questions relevant to their program.

11. Roles and Responsibilities
Under the TBS Evaluation Policy, Deputy Ministers are responsible for the evaluation function.
NRCan’s Departmental Evaluation Committee – an ADM-level Committee – is chaired by the DM
NRCan’s Head of Evaluation – who is also the DG of Planning and Performance Management Reporting – reports to the Evaluation Committee.

12. The Role of the Evaluation Division
The Strategic Evaluation Division (SED) is responsible for:
Proposing a five year departmental evaluation plan to the Evaluation Committee, and updating it annually;
Managing and conducting evaluation studies, including managing contracts and deliverables when consultants are used and issuing reports in a timely manner.
Additionally, SED will help program managers develop their performance measurement strategies, with the goal of ensuring that good data is collected to support future evaluations.
Evaluation will work with your team to develop objectives, a logic model, a performance measurement framework and evaluation requirements.
NRCan's Strategic Evaluation Division is also responsible for reviewing and providing advice on the accountability and performance provisions in Cabinet documents (Memoranda to Cabinet (MCs) and TB Submissions).

13. The Role of Program Managers
Program managers are key to conducting evaluations.
They are responsible for developing, implementing and monitoring ongoing performance measurement – the foundation of evaluation.
Additionally, during an evaluation, they must be actively involved in:
explaining how their programs work;
contributing to evaluation planning, including identifying more detailed evaluation questions;
providing performance measurement information on resources used, activities undertaken and results achieved;
providing detailed documentation (see next slide) and suggestions on potential interviewees, case studies etc.;
participating in working groups to review questionnaires, preliminary findings, draft evaluation reports, etc.;
developing management responses and action plans for their ADMs and implementing them after the evaluation.

14. Key Documents for An Evaluation
In preparation for an evaluation, program managers will be asked to provide key documents as early as possible, including:
Legislation, Regulations, MCs, TB Submissions
RMAFs, RBAFs or Performance Measurement Frameworks
references in budgets, SFTs, DPRs, RPPs
briefing notes, reports --including annual and project reports, studies, databases
websites and communications products
five-years of financial expenditures for the PAA unit including G&C expenditures, O&M, and salaries

15. Questions and Assistance
If you have any questions on evaluation, or wish evaluation assistance in developing performance measurement information or Cabinet documents, please contact:
The Director of Strategic Evaluation
(613)
Electronic copies of this document, completed evaluation reports and the Terms of Reference for the Evaluation Committee are available at: