1 Measuring Results and Evaluating Impact: What You Need to Know Paul Gertler and Barbara Bruns Human Development Network November 2006

2 How do we turn this teacher…

3 …into this teacher?

4 What do we need to know?? How many teachers How many are performing like teacher A, like teacher B, or absent altogether How many kids: –enrolled in school –attending school –repeating grades –dropping out –completing primary school Trends over time: are things moving in right direction?

5 What do we need to know?? Student learning – which kids are learning and how much? – who is benefiting? Costs (per student) What inputs, policies and incentives have greatest impact on learning? –remedial teachers, multi-grade teacher training, deworming pills –higher teacher standards, curriculum focused on early grade reading –contract teachers, school-based management, vouchers Which are most cost-effective?

6 Message 1: Monitoring and Evaluation is our core business All TTLs need tools and techniques for –Better monitoring of Bank/IDA operations: defining project development objectives clearly credible results chains selecting indicators –Building client capacity for M&E –Impact evaluation Both are important, and some complementarities –Standardized student learning data and good administrative data (health facility surveys, school census data) lower costs and improve quality of impact evaluations

7 Message 2: We’re not doing a very good job at present All Networks struggling with need for better monitoring and more impact evaluation –INF study: over 200 different indicators in water supply projects, measuring similar things, most w/only 1 obs. –OPCS study: 78 education projects exiting in FY04-05, Only 3 reported MDG (primary completion rate) Only 5 had any data on student learning – and none comparable Less than 50% tracked enrollments (mix of GER, NER, raw numbers, gender) –In S Asia HNP projects, avg. operation had 26 indicators, only 33% with clear definition, baseline and collection schedule

8 Little Progress of M&E Over Time (from Ben Loevinsohn and Aaka Pande)

9 Message 3: HD under special pressure with “results agenda” Greater accountability for major global goals –2/3 of MDGs are HD outcomes –HD outcomes important in IDA 14 results build country capacity to measure these outcomes link PDOs to “higher level” sectoral or MDG outcomes Greater role in “selling” aid –OECD electorates focused on HD issues –selling IDA 15 on “results”, esp. HD results

10 Message 4: there is help…. Tools –Checklist for M&E in HNP operations, also education –Guidance on indicators (AFR Generic Results Framework for HIV projects) –Common instruments for classroom observation and student learning assessment Technical advice –“on demand” impact evaluation clinics –FPSI “on demand” PDO/monitoring clinics Financing –BNPP funding to launch prospective impact evaluations Training

11 Plan for next 3 days TODAY Understanding the value of impact evaluation Understanding the results agenda from an institutional perspective TTL experience in improving M and E 4 hour module on Improving “M” -- results focus in projects TUESDAY TTLs will meet in MC Researchers will meet in IFC (F-5-P100) All day on impact evaluation – methods, operational issues WEDNESDAY Everyone will meet in MC Teams organized along sector lines will prepare evaluation designs for projects of their choice & write concept notes Teams present in evaluation clinics