1. Evaluation in the Government of Canada
Robert Lahey
Senior Director, Centre of Excellence for Evaluation
Treasury Board Secretariat of Canada
16 October 2001

2. Centre of Excellence for Evaluation
Outline of Presentation
Canada at a Glance
Key Players in Evaluation
Perspectives on the Past
TBS Study
New Agenda: Results for Canadians
New Evaluation Policy
TBS Centre of Excellence for Evaluation
Community Renewal: Competency Profile
Resources
Bob’s notes: Add – standards, HR strategy, and competency profile
Centre of Excellence for Evaluation

3. Centre of Excellence for Evaluation
Canada at a Glance
Sovereign parliamentary democracy
Population 31 million
Federation of 10 provinces and 3 territories
Areas of federal responsibility include: defence, criminal law, postal service, census, copyrights, trade regulation, external relations, money and banking, transportation, citizenship, and Indian affairs.
Centre of Excellence for Evaluation

4. Centre of Excellence for Evaluation
Government Accountability
Parliament of Canada
House of Commons
Parliamentary Committees
PM and Cabinet
TB Minister
All other ministers
Treasury Board Secretariat
(secretary appointed)
government policy
oversees spending
Federal depts/agencies
(DMs appointed)
government operations
approved budgets
Public servants
Public servants
Auditor General
(appointed)
Independent audits of govt operations
Centre of Excellence for Evaluation

5. Centre of Excellence for Evaluation
Key Players in Evaluation
Treasury Board Secretariat (TBS)
Government’s management board (financial, management and human resources)
Comptrollership function
Centre of Excellence for Evaluation
Policies and standards
Capacity building
Links evaluation and performance measurement
TBS: The items listed are roles. Responsibilities: policy, standards, evaluation of policy and capacity building initiatives
Departments: Responsibilities: Evaluation reports on programs and departmental policies; performance reports on departments
Centre of Excellence for Evaluation

6. Centre of Excellence for Evaluation
Key Players in Evaluation
Auditor General (AG)
Conducts independent audits of government operations
Produces periodic oversight reports on the conduct of evaluation
Promotes accountability and best practices
Reports directly to Parliament
Note: Wording derived from the OAG deck.
Conducts independent audits and examinations of government operations, including the conduct of evaluation, that provide information, advice and assurance to Parliament
Centre of Excellence for Evaluation

7. Centre of Excellence for Evaluation
Key Players in Evaluation
Departments/Agencies
Deputy Ministers (DMs) accountable for the application of Evaluation Policy within their departments
Heads of Evaluation implement policy as per TBS standards and guidelines
Internal accountability and reporting to DMs
External accountability and reporting to TBS and Parliament
TBS: The items listed are roles. Responsibilities: policy, standards, evaluation of policy and capacity building initiatives
Departments: Responsibilities: Evaluation reports on programs and departmental policies; performance reports on departments
Centre of Excellence for Evaluation

8. Centre of Excellence for Evaluation
Perspectives on the Past: Activity Graph
– Intensity of activity
1960s
1970s
1980s
1990s
2000s
Graphical Interpretation of the Extent of Formal Evaluative Activity in the Federal Government
Centre of Excellence for Evaluation

9. Centre of Excellence for Evaluation
1960s
Growing demand for information to support increasingly complex and costly programs
New management systems created for financial administration and planning, programming and budgeting
Evaluation as a practice not yet formalized
Lessons Learned
Need for formal evaluation increases as resources become scarcer and the identification of priorities becomes more important
Increasingly complex and costly programs
Growing demand for information to support program changes
Established a new planning, programming and budgeting system
Introduced new financial administration and auditing systems
Politicians, senior managers and Canadians demanded information to support program improvement, modification and management
Efforts made to report on accountability and to justify expenditures through a new planning, programming and budgeting system
Centre of Excellence for Evaluation

10. Centre of Excellence for Evaluation
1970s
Building of key infrastructure elements in departments
Treasury Board Evaluation Policy (1977) was the first formalized evaluation policy in Canada
Evaluation Policy centre created within the new Office of the Comptroller General (1978)
Lessons Learned
Necessary, but not sufficient, conditions for an effective evaluation system are: government investment and support; formalized policies and standards; and, leadership for capacity building
Period of increased funding and decentralization
Building of evaluation infrastructure (policies, standards and central agencies) and setting the stage for more formalized evaluation
Treasury Board Evaluation Policy (1977) was the first formalized evaluation policy in Canada
Program Evaluation Branch of the new Office of the Comptroller General created (1978)
Branch issued Policy, guides and standards to assist departments in establishing and maintaining an evaluation function
Centre of Excellence for Evaluation

11. Centre of Excellence for Evaluation
1980s
High expectations for evaluations not always met
Government-wide review led by Parliament
Lessons Learned
Evaluation quality depends on an approach that balances: timeliness, usefulness, methodological purity, client requirements and cost
High expectations for evaluations in spite of known theoretical and methodological challenges
Government-wide review led by Parliament changed organizations and reduced spending
Review process and recommendations changed government, reduced spending and highlighted the lack of adequate and timely evaluations and performance reports
Highlighted the lack of adequate and timely evaluations and performance reports for departments, central agencies and parliamentary committees
Centre of Excellence for Evaluation

12. Centre of Excellence for Evaluation
1990s
Evaluation capacity affected by general government downsizing
AG report (1993) indicated renewal of evaluation capacity needed
New Review Policy (1994) linked evaluation closer to internal audit
Lessons Learned
Critical mass in capacity is required to ensure evaluation remains credible, relevant and strategic
Evaluation capacity and critical mass needed for evaluation lacking due to downsizing
Auditor General reported (1993) on need to improve the evaluation system and renew evaluation capacity
New approach linked evaluation closer to internal audit with the new Review Policy (1994)
Evaluation resources often diverted to other areas related to program design and management
Centre of Excellence for Evaluation

13. Centre of Excellence for Evaluation
TBS Study (2000)
TBS study pointed to uneven delivery of evaluation across departments
Strong support to reinvest in evaluation
Commitment to linking evaluation to broader accountability and reporting requirements
Objectivity, not independence, needed
The Auditor General and other champions effectively argued that good performance reporting is an essential tool for modern democratic governments
Results for Canadians agenda articulated the role of evaluation in management for results
Centre of Excellence for Evaluation

14. Centre of Excellence for Evaluation
New Agenda: Results for Canadians (2000)
Agenda to:
Improve government services and respond to demands for better value and transparency
Move from reporting on results to managing for results
Applied through a wide-reaching series of initiatives, including the new Policy on Evaluation (2001)
Perspective on modern management shared by the political level and the Public Service
Impetus for implementing a new Policy on Evaluation (separate from Internal Audit Policy)
Transparency – definition/description
Centre of Excellence for Evaluation

15. Centre of Excellence for Evaluation
New Evaluation Policy: Highlights
Re-emphasis on the need for evaluation to be strategic, comprehensive and systematic
Scope broadened to include programs, policies and broad initiatives
Provides guidance on standards
Evaluators are encouraged to work directly with managers to build evaluation into the “life cycle” of programs
Highlights links between evaluation and results-based performance measurement
Establishment of a stand-alone evaluation function
Scope broadened to include a full range of activities, including policies, as well as interdepartmental and intergovernmental programs, programs, policies, regulations, R&D, services, initiatives and broad areas of spending, such as human resources
Provides guidance in the interpretation and application of standards for evaluation
Resourcing available to build evaluation capacity and create the Centre of Excellence for Evaluation
Centre of Excellence for Evaluation

16. Centre of Excellence for Evaluation
Highlights (continued)
Managers are responsible for the active monitoring of their programs
Government is committed to the public reporting of evaluations
Centre of Excellence for Evaluation created in TBS
New funding available for departments to build and meet future evaluation capacity
Establishment of a stand-alone evaluation function
Scope broadened to include a full range of activities, including policies, as well as interdepartmental and intergovernmental programs, programs, policies, regulations, R&D, services, initiatives and broad areas of spending, such as human resources
Provides guidance in the interpretation and application of standards for evaluation
Resourcing available to build evaluation capacity and create the Centre of Excellence for Evaluation
Centre of Excellence for Evaluation

17. Centre of Excellence for Evaluation
New Evaluation Policy: Standards
Six standards for evaluation:
Evaluation planning and issues
Competency
Objectivity and Integrity
Consultation and Advice
Measurement and Analysis
Reporting
Guidance based on ‘good practices’
Evaluation Planning and Issues
Standard
The department must apply the discipline of evaluation to assess the performance of its policies, programs and initiatives, both departmental and inter-organisational, taking into account its priority concerns as well as those of its partners and the government as a whole.
Guidance
Evaluators should develop a strategically focussed plan that is based on assessments of risk, departmental priorities and reporting requirements, and priorities of the government as a whole.
The full range of evaluation issues should be considered at the planning stage of an evaluation:
−        does the policy, program or initiative continue to be consistent with departmental and government-wide priorities and does it realistically address an actual need? (relevance);
−        is the policy, program or initiative effective in meeting its objectives, within budget and without unwanted outcomes? (success); and
−        are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches? (cost-effectiveness).
Evaluators should address issues that are needed for accountability reporting, including those involving key performance expectations (a) identified and conveyed to the Treasury Board or (b) resulting from Cabinet decisions requesting evaluation information.
Centre of Excellence for Evaluation

18. Centre of Excellence for Evaluation
TBS Centre of Excellence for Evaluation (CEE)
Pro-active leadership role in the renewal of evaluation in Canada through:
Building evaluation capacity
Communicating and networking
Policy development, implementation and evaluation
Repositions and renews the evaluation function across government
Emphasizes partnerships and knowledge management
Centre of Excellence for Evaluation

19. Centre of Excellence for Evaluation
Community Renewal: Competency Profile
Draft report (2001) identifies two key attributes of successful evaluators:
Cognitive capacity
Ability to communicate well
The right people:
Respect diversity
Work collaboratively and openly
Recognize and diffuse conflict
Demonstrate sensitivity, tact and empathy
Care about professional practices and standards
The competency profile can be used by central agencies and departments to: establish criteria for the identification of entry-level evaluators; analyze organizational gaps; develop personal assessment tools; build strong teams; plan careers and successions
Senior evaluators add an ability to:
Manage group dynamics and interpersonal relationships across project teams
Centre of Excellence for Evaluation

20. Centre of Excellence for Evaluation
Resources: Evaluation Infrastructure Checklist
Concise Version
Cultural
institutions are prepared to divulge information
managers trust that assessments will be objective
agencies are willing to be reviewed
managers have the courage to make changes and implement recommendations
evaluation function is prepared to evaluate itself
relevant accountabilities have been clarified
Operational
technical, professional and financial resources are available
time is sufficient
evaluation policies and standards are in place
business/strategic plans are developed
need for objectivity can be met
authority exists to oversee evaluations and act on findings
Government of Canada Evaluation System
Strengths/Opportunities
Leadership and support for evaluation in the Government of Canada comes from the policy centre
Evaluation is linked to a broader management agenda
Heightened awareness about the uses and usefulness of evaluation
New Evaluation Policy is in place
New investments have been made in the evaluation function
Challenges
Further investment is needed to build evaluation capacity and cultivate a positive management culture for evaluation
Continuing need to demonstrate the uses and usefulness of evaluation
Vigilance required to monitor evaluation the evaluation system and ensure that the evaluation infrastructure remains healthy
Achieving the right balance between methodological purity and timeliness to enable managers to base decisions on evaluation results
Centre of Excellence for Evaluation

21. Centre of Excellence for Evaluation
Resources: Selected GoC Web Sites
Treasury Board Secretariat of Canada (TBS)
TBS Centre of Excellence for Evaluation
TBS, Guide for the Development of Results-based Management and Accountability Frameworks
TBS, Evaluation Policy
Centre of Excellence for Evaluation

22. Centre of Excellence for Evaluation
Contact Information
Robert Lahey
Senior Director
Centre of Excellence for Evaluation
Treasury Board of Canada Secretariat
300 Laurier Street
Ottawa, Ontario K1A 0R5
CANADA
Telephone: (country code 1) –
Fax: (country code 1) –
Centre of Excellence for Evaluation