Evaluation capacity building: the international experience Scott Bayley, AusAID

Overview What is evaluation capacity building (ECB)? Why do it? Strategies for ECB International examples What works? – guiding principles

What is ECB? The UNDP (2002) defines capacity as the ability of people, institutions and societies to perform functions, solve problems, and achieve their goals. ECB refers to activities and initiatives taken to mainstream a regime that supports the ongoing production and use of evaluations.

Why do it? Governments and organizations build evaluation systems because they believe such systems will help them to improve their means and methods of governance. In particular … ECB is a means to an end, not an end in of itself

… ECB can be used to support: Why do it? ... … ECB can be used to support: Planning & policy making Program management Resource allocations, budgeting Government control, coordination External reporting & accountability Participation by civil society.

What is successful ECB? The production of appropriate quality evaluations A high level of utilization of evaluation findings Country/agency ownership Sustainability over time as governments & officials change.

Strategies for ECB 1. Whose capacity and for what purpose? Legislatures Federal/state governments Sectors, eg. health, education, water, agriculture, justice Agencies that commission and fund evaluations

Individual evaluation practitioners and networks strategies for ECB … Individual evaluation practitioners and networks Those who use the results of evaluations to help guide their policy making and management activities, govt agencies, donors, NGOs

Whose capacity and for what purpose? strategies for ECB … Groups affected by the programs being evaluated General public, the academic community, civil society. Whose capacity and for what purpose?

2. Diagnostics current practices politics of reform strategies for ECB … 2. Diagnostics current practices politics of reform desired functioning

strategies for ECB … Current practices -current policies and systems, existing M&E activities, institutional capacities, linkages to planning and budgeting

strategies for ECB … Politics of reform -drivers for change, supporters and opponents of reform, resources, incentives

strategies for ECB … Desired functioning -intended users and uses, supply and demand factors, supporting institutional infrastructure

2. Diagnostics X

Overseas fact finding tours Training workshops for practitioners strategies for ECB … 3. Tactics Overseas fact finding tours Training workshops for practitioners Production of policy and operational manuals, evaluation plans Legislation/regulation Supply side This approach has been tried and failed in many many countries, it just doesn’t work! e.g. Viet Nam, Uganda

 Demand for and ability to use evaluative feedback, then sequencing ECB …  Demand for and ability to use evaluative feedback, then Supporting institutional infrastructure & Supply side considerations - Guiding principles (ECDG group): ownership; relevance; integration; usefulness. Need to be able to first plan, budget, audit and then evaluate

Identify and work with powerful champions of reform strategies for ECB … Identify and work with powerful champions of reform Link ECB to other significant reforms Actively manage the change process - ECB develops over time and in stages Other reforms: evidence based policy making, improved strategic planning, results agendas, customer service charters, better programming, knowledge management

Incentives: strategies for ECB … Incentives to increase demand and use the example set by leaders, rewards, education and information, practical support Incentives to increase demand and use Incentives to reduce perceived costs and increase benefits within agencies (eg. guarantee of no job losses or budget cuts)

L/T partnerships and joint evaluations for skill transfer/learning strategies for ECB … Training to raise awareness of those who might demand evaluations to enhance service quality i.e. internal senior managers; external bodies such as NGOs, consumer advocates, media L/T partnerships and joint evaluations for skill transfer/learning Train your critics!

Train the trainer (to provide on the job training to practitioners) strategies for ECB … Train the trainer (to provide on the job training to practitioners) Hold managers accountable for learning/making improvements Capitalize on existing demand and windows of opportunity Link evaluation into systems for policy making, planning, budgeting, reporting, etc.

International examples First wave, 60s & 70s Second wave, 80s & 90s Third wave 90s+ First wave about policy and program improvements Second wave about cost control Third wave is eclectic Discuss selected international examples

What works? – guiding principles 1. ECB is a political activity with technical implications. and not vice versa! ECB is about organizational change, link it to other significant reforms ECB creates winners and losers - which means supporters and opponents. ECB is not an end in itself, need to link evaluation to other high profile reform efforts/initiatives, e.g. results agenda, evidence based policy making, budgetary reforms, customer service standards, etc.

2. Start with good diagnostics. principles… 2. Start with good diagnostics. local context and history matter! aim to build upon local strengths and interests; target functional needs rather than ‘correcting’ deficiencies.

3. Building and capitalizing on demand is the key. principles… 3. Building and capitalizing on demand is the key. seek to match the supply of evaluations to current demand, build demand (and capacity to use eval info) over time attempting to force the creation of an evaluation culture through legislation or a supply side push simply doesn’t work a limited availability of internal evaluation skills is not a fundamental constraint, use contractors if need be. Hire contractor A to help you clarify your information need and to employ/supervise contractor B who actually undertakes the evaluation

4. Institutionalize evaluation in a way that aligns supply to demand. principles… 4. Institutionalize evaluation in a way that aligns supply to demand. the evaluation function needs to be located (or anchored) where the demand and users are we have a matrix arrangement to support accountability in government. Evaluation would benefit from a similar approach.

5. Learn from others but avoid best practice models. principles… 5. Learn from others but avoid best practice models. different countries/agencies have different starting points, aspirations, constraints & opportunities search for adaptable ideas, not blueprints for reform Cautious evolving experimentation is the way to go (along with M&E of ECB activities) - ECB is not a linear process.

6. ECB is a long term process principles… 6. ECB is a long term process 5 to 10 years for agencies in more developed countries 10 to 20 years in developing countries political and organizational interest in evaluation is cyclical - ECB is ongoing, not a “one time” event.

Summary What is evaluation capacity building (ECB)? Why do it? Strategies for ECB International examples What works? – guiding principles

Questions & Discussion

For further information: scottbayley56@yahoo.com.au * Abonyi, George. 2002, Toward a Political Economy Approach to Policy-based Lending. ADB (on the web). * Boyle, Richard, and Donald Lemaire, eds. 1999, Building Evaluation Capacity. New Jersey: Transaction Publishers. * ECDG, 2011, International workshop agreement on evaluation capacity development (on the web). * Lahey, Robert. A Framework for Developing an Effective Monitoring and Evaluation System in the Public Sector (on the web). * Mackay, Keith. 2007. How to Build M&E Systems to Support Better Government. World Bank (on the web).