Evaluation Methods Training and Capacity Building Programs Nidhi Khattri Independent Evaluation Group November 17, 2008

IEG ’ s Mandate ► The World Bank’s independent evaluation function established about 35 years ago ► The goals: learning from experience accountability for the achievement of objectives

IEG ’ s Independence: Direct Report to the Board ► Direct reporting to the Board of Executive Directors ► Headed by a Director-General, Evaluation (DGE) Appointed by the Board No World Bank Group position after current position ► Evaluations to the Committee on Development Effectiveness (CODE) ► Evaluation content not negotiated with CODE/Board

IEG ’ s Links to Bank Management ► Bank management has opportunity to comment ► Draft Bank Management Response accompanies evaluation ► IEG responds to Bank management comments at Board meetings ► IEG’ Management Action Record (MAR) reports on management’s progress on actions noted in management response

IEG ’ s Evaluation Products ► Project Evaluations Project assessments (ICR Reviews, PPARs) Impact studies (e.g., Bangladesh Health, Ghana Education) ► Sector and Thematic Evaluations Often linked to policy revision (e.g., forestry – altered sector policy) ► Country Evaluations Country Assistance Evaluations Country Impact Reviews (IEG-IFC) Reviews of CAS Completion Reports ► Global and Regional Program Evaluations ► Corporate Evaluations Annual Review of Development Effectiveness (which now includes the Annual Report on Operations Evaluation)

Evaluation Approaches ► Based on evaluation products ► Primarily Objectives Based for Projects and Programs Outcome Risk to Development Outcome Bank Performance Borrower Performance Monitoring and Evaluation (M&E) Quality

Outcome ► The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently ► Outcome = Relevance + Efficacy + Efficiency

Bank Performance ► The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision ► Bank Performance = Quality at Entry + Quality of Supervision

Monitoring and Evaluation (M&E) Quality ► M&E design—the extent to which the project was designed to collect appropriate (input, output, outcome, and impact) data given project objectives and given already available data ► M&E implementation—the extent to which appropriate data was actually collected using appropriate collection methods (to ensure data quality) ► M&E utilization—the extent to which appropriate data was used to inform decision-making and resource allocation

Corporate Evaluation: Annual Review of Development Effectiveness (ARDE) ► Annual meta-evaluations that provide a comprehensive assessment of the Bank’s development effectiveness ► Draw on IEG’s recent project, sector, thematic, country, and global evaluations ► Synthesize lessons that can be used to increase the development effectiveness of World Bank assistance ► Highlight the findings of recent IEG evaluations around a common theme

Some ARDE Themes ► Bank’s effectiveness in achieving Millennium Development Goals ► Effectiveness of Bank support for policy reform ► Bank’s contribution to poverty reduction ► Effectiveness of Bank’s results agenda

Recent IEG Evaluations around Training and Capacity Building ► Using Training to Build Capacity (2007) Avg amount of client training estimated at $720 mio per year (90% through projects, rest through WBI) Key component in 60% of investment projects, particularly in social, rural, public sectors 37 (incl. 8 WBI) training programs (Bangladesh, Burkina Faso, Mexico, Tunisia), 6 country surveys of 550 trainees, comparison with other DTIs, etc. ► Capacity Building in Africa (2005) 25% of Bank lending to Africa; $9 bill between ► Public Sector Reform (2008) 1/6 of Bank lending and advisory support – and increasing

Methodology in Evaluation of Training ► Key Questions: To what extent did Bank-financed training have an impact on the capacity of target organizations? What factors contribute to successful training? To what extent are such factors present in Bank-financed training? ► Main Methods – using objectives-based methodology Survey of training participants Survey of training institutions In-depth field reviews

Training Results in Capacity Building Only When Certain Conditions are Met Input Training Output Learning Outcome Workplace behavior change Impact Enhanced institutional or organizational capacity

Effectiveness of Training: What is the Evidence? Outputs Performance Outcomes Impact on Capacity Other Findings Most training resulted in demonstrable learning But: Individual learning gains poor predictor of impact Project-based trained lacked basic results measures About half of trainees surveyed reported substantial positive changes in work performance 10 project-based programs had significant impact (e.g., Procurement reform, Community Groups, Exporters, SME) Best combined project funds + outside expertise WBI programs not rated due to lack of data Type of training provider not correlated with success Good training outcomes in both higher- and low- capacity environments

Training Design: What Works? Targeting of Training Content Pedagogy Workplace Transfer Adequate diagnosis of capacity gaps associated with strong client commitment / involvement Training needs assessment the norm in highly-rated programs, but often subject to funding constraints (WBI) Good participant selection requires engagement and supervision (e.g., IMF Institute, MASHAV) Poor targeting most important cause for lack of impact Generally high marks for design & teaching standards But course length and topic coverage needs to be better matched with capacity building goals (ITCILO,JICA) Allow time for practical learning techniques / action plans (InWent, JICA, MASHAV) Provide systematic follow-on support (Motorola, InWent)

When Training Works: What Matters Most? Workplace Context Incentives Client Commitment Support by managers and peers is key driver for successful workplace implementation (~90% feedback) But: 1/3 of trainees didn’t have adequate resources, incentives or org. support to apply what they learned Focus needs assessment on organize. bottlenecks and whether training is indeed the right tool. Lack of institutional incentives recurring problem in civil service and highly decentralized training programs; Stronger incentives at work in training programs for community and farmer groups, private sector firms Hi-level support for training key for workplace transfer Stand-alone training with limited ability for diagnosis, dialogue, influence and follow-up

Key Internal Inhibitors for Effective Training? Results Orientation Standards Access to Expertise Alignment At design, most training programs fail to specify training objectives and expected performance outcomes, reflecting lack of broader capacity needs assessments At completion, performance evaluation rarely done No feedback / accountability loop >> no improvements Lack of established standards for training design and implementation undercuts quality assurance; Team leaders for project-based training lack adequate in-house support and voiced demand for more WBI earns high marks for Country Team consultations, but collaboration at task level remains rare. Risk of diffusion of program.

Persistent Capacity Gaps in Africa Despite Substantial Inputs (IEG, 2005) Obstacles Recommended Actions Recent Changes 40% of sampled lending operations achieved CB goals, with better outcomes in roads than in health / education. CB lacked clear results framework (only 1/3 of projects clear about relationships among individual, organizational. and institutional aspects of capacity) Weak diagnostics of pol. econ and available country capacity High fragmentation of efforts; supply driven TA Training not embedded in broader HR strategies; Strengthen K-base, operational framework, M&E Develop sector-specific guidance Promote country-led approach Re-assess role / modalities of training Capacity Building has moved to center stage in AFR CASs Focus shifting beyond individual skills to institutional support leadership, donor harmonization and better coherence Key Finding

Mixed Outcomes on Public Sector Reform (IEG, 2008) Successful Themes Reform Challenges Recommended Actions Public Financial Management: good diagnostics, indicators, joint undertaking with govts (PEFA) Tax Administration: strong MoF motivation, good TA Transparency: widened access to information Civil Service and Administrative Reforms: lacks good models, indicators, but too important to ignore Government-wide anti-corruption: key are political commitment, strong judiciary Recognize complex political + sequencing issues; focus on basics first Prioritize anti-corruption effort on most harmful aspects Underpin civil service reforms through better diagnosis

Opportunities Going Forward? Guidelines Learning & Quality Awareness Frontier Review & adopt good practice standards for training programs based on a realistic results framework Pool evaluation findings to expand practices/lessons Encourage systematic use in decision-making Pool expertise around core training and capacity building management practices Identify case studies of influential training programs Pilot new forms of collaboration Bridge the gap between training and other modes of capacity building New modalities / increased demand for client-led training How training can contribute to development objectives Scope & Scale

THANK YOU! IEG Website :