The Evolution of Evaluation Practices in the Canadian Federal Government Presented at CES / AEA Conference Toronto, Ontario, Canada October 29, 2005 Presented by:George Teather Tel:(613) Fax:(613)

PRESENTATION OUTLINE u General overview of evolution of Canadian federal government evaluation practices u Description of use of evaluation in program management u Review of challenges u International comparisons 2

Review of Evaluation Practices u Long standing practice, experience with evaluation, as the formal requirement for evaluation began by 1980 u Evaluation groups are in Departments, with central agency overview, guidelines and some workshops u Private sector consultants, with evaluation and performance measurement expertise, conduct many of the studies under the oversight of Departmental staff u Evaluation has incorporated key characteristics ä Use of logic model to describe the program ä Generic issue groups Ô Relevance Ô Objectives achievement Ô Alternatives (design and delivery) ä Multiple lines of evidence to increase validity and credibility u In mid 1990s, periodic, indepth evaluation and ongoing performance measurement became more closely linked 3

Integration of Evaluation and Performance Measurement with Management Practices u Results-based Management and Accountability Framework (RMAF), introduced in 2001, provides an integrated approach to annual performance measurement and periodic evaluation ä Logic model, description of program ä Performance measurement strategy, indicators, sources of info ä Evaluation issues, methodological approach ä Accountability, reporting requirements and schedule u Departmental management practices incorporate RMAF concepts and information ä Key results commitments describe objectives ä Annual Departmental Report on Plans and Priorities identifies annual plans, intended results ä Annual Departmental Performance Report provides information on annual outcomes and achievements ä More recently, Program Activity Architecture has been introduced, which identifies key results linked to Departmental objectives and indicators to demonstrate level of achievement and performance 4

How is Evaluation Used? What is Its Impact on Policy Development and Implementation? u Evaluation is embedded in management practices ä RMAF is required for all new and renewing contribution programs and recommended for others ä Annual performance measurement and reporting is required ä Evaluation studies are usually required before renewal of funding u However, evaluation is only one of many inputs to decision making ä Political imperatives ä Policy groups, stakeholder influence u Location of evaluation in Departments leads to: ä Use to describe program benefits, and defend programs from central agencies ä Use to adjust program design and delivery in some cases, almost never call for cancellation of program u Because existing programs are being evaluated, evaluation has a greater role in policy implementation than policy development u Logic model has been useful in program design and development as well as evaluation and performance measurement 5

What are the Most Compelling Evaluation Challenges? u Main challenges are closely linked: ä engaging senior management’s attention and interest in understanding and utilizing evaluation results ä educating policy analysts and central agencies as to what can be expected from evaluation and performance measurement ä producing relevant, credible evaluation studies u There is a long standing issue of linking policy development and implementation with scientific evidence ä Council of Science and Technology Advisors produced a report on Scientific Advice for Government Effectiveness (SAGE) that identified ways to improve linkages and improve consideration of evidence ä Difficult to undertake horizontal, multidepartment evaluations due to location of evaluation expertise ä Most S&T policy emphasis is linked to economic policy, with S&T as a driver for economic growth, less consideration of the role of S&T in social policy and public good u Government has cut back on training and building evaluation capability in departments specific to the needs of government, with a negative effect on expertise, quality of work ä Some evaluation studies identify results and benefits that have limited attribution to the program and don’t identify other contributors ä There is no regular independent / central agency review of the quality and credibility of evaluation reports 6

Tracking National Performance – What are the Correct Indicators for International Comparisons? u Comparisons require international agreement on indicators, access to credible, consistent data u Standard comparisons include: ä Resources Ô GERD, (funding by sector, contribution of government, industry)) Ô Highly qualified personnel (engineers, scientists, PhDs) ä Outputs Ô Scientific publications (English journals) Ô Patents u Example for Biotechnology ä Statistics Canada has developed an international standard for the measurement of biotechnology activities that is being adopted by OECD countries, including U.S. u More work needs to be done on identifying indicators, developing data sources ä Need the support of Statistics Canada, other agencies ( RSA) ä Comparison of national innovation systems, public / private sector interactions 7

Most Promising Evaluation Methods u There has been a general realization of the need to extend the logic model concept to explicitly identify: ä the group or individuals (participants, clients, beneficiaries) that the program intends to influence to change their behaviour in order to achieve objectives ä The partners, collaborators and stakeholders whose participation and support are required for the program to fully succeed u In a given program, there may be different groups at different stages of the program, or participating for different purposes, objectives ä Research – university, gov’t scientists ä Development – engineers, innovative firms ä Commercialization – venture capital, banks, companies u Examination of actual compared to intended clients of a program is an important issue ä Profile of clients and participants u Identification of key partners and their level of involvement is also important in examining the “pathway to success” ä Network analysis ä Analysis of roles and relationships 8

Generic Program Logic Model* Program Objective: high level strategic purpose HOW? WHO / WHERE? WHAT do we want? WHY? ResourcesReachResults Program / Service Delivery Client Management Policy & Issue Management Financial Management Human Resources Management Asset Management Program deliverables Policy guidelines, regulations Communications -plans - internal communications - promotion - info transfer - consultations - meetings/events Funding Service Outputs Primary Targets (clients, ultimate beneficiaries) Co-delivery Agents Other Stakeholders activitiesoutputs users / clients / co-deliverers / beneficiaries direct outcomes intermediate outcomes ultimate impacts Client Service - addresses needs - meets / exceeds expectations - service quality Behavioral Influence -awareness -understanding -attitude / perception -support New knowledge Improved capability Improved decision making Target group changes in behaviour / other outcomes Sector / Industry / Regional Impact Economic/ Environmental/ Societal Impact Contribution to organizational objective *reference 9

Operational Your operational environment You have direct control over the behaviours within this sphere Behavioural Change Your environment of direct influence e.g.people and groups in direct contact with your programs, staff (i.e. clients, target audience, co-delivery partners State Your environment of indirect influence e.g., Industrial sectors, government decision makers, other communities of interest where you do not make direct contact Spheres of Influence* Time * reference S. Montague, 10

References u Treasury Board RMAF and other evaluation policies and guidelines u Council of Science and Technology Advisors SAGE and other reports u Evaluation and performance measurement courses and consulting services 11