1. IMPACT EVALUATION WORKSHOP 24 February 2011

2. Workshop objectives Our goals for this workshop: 1. You know when and how an impact assessment can help you to evaluate and improve your work. 2. You can be part of a team conducting an impact assessment with the support of experts. 3. You can apply knowledge gained from other impact assessment in Africa and the world.

3. AGENDA I. Introduction to IPA II. Impact evaluation III. Lessons from evaluations: Africa & beyond IV. Discussion and exercise: designing an evaluation for your sector

4. AGENDA I. Introduction to IPA II. Impact evaluation III. Lessons from evaluations: Africa & beyond IV. Discussion and exercise: designing an evaluation for your sector

5. I.INTRODUCTION TO IPA 24 February 2011

6. Who is IPA? Innovations for Poverty Action (IPA) is a nonprofit organization that creates, evaluates, and replicates innovative approaches to solving real-world problems faced by the poor in developing countries. IPA disseminates proven findings about what works, and what does not work, to policymakers, practitioners, and donors around the world. IPA works with a network of researchers from major universities around the world who are interested in issues related to the fight against poverty.

7. What do we do? Uses rigorous evaluations to determine what works and what does not work  Surprisingly little is known about what works in the development field  Sometimes conventional wisdom needs to be rethought  Estimate magnitude of effects  Most cost-effective solutions; can be surprising  Know how to improve programs  Where to spend limited resources IPA was created in 2002 to answer questions about what poverty alleviation programs work best

8. IPA’s approach InnovateEvaluateReplicate Communi- cate Scale - Understand market failures -Develop innovative solutions to poverty - Use frontier knowledge from economics, and psychology Randomized Controlled Trials -Impact evaluations - Comparing variations of an intervention - Experiment with product designs Replicate evaluations in various settings to : - Generalize research findings - Tell practitioners what works (and not), when Effectively communicate to practitioners: -Conferences -Workshops with policy makers and practitioners -Policy memos and focus notes Facilitate scale- up of effective solutions : - Active policy outreach -Practitioners’ toolkits - Hands-on technical assistance We generate insights on what works and what does not through randomized evaluations, and ensure that those findings will be useful to, and used by practitioners and policy makers

9. Innovations for Poverty Action IPA works on 300 projects in 42 countries

10. IPA in Ghana Here since 2005 16 active projects, 40 full time staff, up to 250 part time, 2 permanent offices Partnerships with: Ghana Education Service, Sinapi Aba Trust, Freedom from Hunger, Mumuadu Rural Bank, Zenith Bank, Presbyterian Agricultural Service Implementation and Evaluation Teams Dissemination of results to local policy makers and practitioners Providing education and support on RCTs Scaling ups and Replications

11. Impact evaluation warm-up Suppose there is a doctor in Ghana who wants to test the effectiveness of a new drug that she has developed to prevent diabetes. Diabetes is a disease marked by high levels of sugar in the blood; obesity, poor eating and exercising habits, high cholesterol, and genetics can be risk factors for developing diabetes. The doctor wants to know how the drug will affect the ordinary Ghanaian. A few NGOs wanted to test the drug and proposed the evaluation ideas listed below. Consider these questions as you read through:  What is the evaluation measuring?  Who is it measuring the impacts of the drug on?  What could be problems with this evaluation?

12. Scenario 1: Clinic A medical clinic agrees to test the drug. The clinic is an old a prestigious organization and their offices are located on the top floor of a very tall storey-building. The doctor is excited to partner with such a well-renowned clinic (her only complaint is that there is no elevator and she has to climb the stairs every time she meets with their staff!) The staff of the medical clinic want to go to the market and give a coupon for a free treatment to the first 500 people that they meet. Their staff says that they will take the first 250 people who come to their office to collect the drug and will provide the drug for free until the study ends. Several years later, they will find all the 500 people and assess their health. The clinic staff says that this way they can show a better comparison: how the group receiving the drug responds to treatment by showing how the people who do not receive the drug compare

13. Scenario 2: NGO A health-focused NGO is also interested in how nutrition and exercise affect whether one develops diabetes. They propose to include the drug in a study they are running. In their study, 500 participants are given advice and help in preparing healthy meals as well as a free membership to a gym. The director of the NGO says that they will give half of these people the pill and see how it affects whether they develop diabetes.

14. AGENDA I. Introduction to IPA II. Impact evaluation III. Lessons from evaluations in Africa and around the world IV. Discussion and exercise: designing an evaluation for your sector

15. II. IMPACT EVALUATION 24 February 2011 What is an impact assessment? What makes an impact assessment? The keys to a good impact assessment

16. II. IMPACT EVALUATION February 16, 2011 What is an impact assessment? Why randomize? The keys to a good impact assessment

17. Programs /Policies Knowledge Evidence Experience Personal collective Ideology Own External Support Budget Political Capacity What goes into policymaking?

18. Surprisingly little hard evidence on what works Can do more with given budget with better evidence If people knew money was going to programs that worked, could help increase pot for anti-poverty programs Instead of asking “do aid/development programs work?” should be asking:  Which work best, why and when?  How can we scale up what works? Why focus on impact evaluation? 18

19. Why evaluate? What makes a good evaluation?  What do we hope to learn? What do different type of evaluation do?  Understanding the program  Impact evaluation  Defining impact (preview) When to do a randomized evaluation? Lecture Overview

20. Lesson learning  Beneficiaries  Program  Organization  World Accountability Reduced poverty through more effective programs Different types of evaluation contribute to these different objectives of evaluation Objective of evaluation

21. Ask the right questions  For accountability  For lesson learning Answers those questions in unbiased and definitive way To do that you need a model: logical framework/theory of change  Who is the target?  What are their needs?  What is the program seeking to change?  What is the precise program or part of program being evaluated? 21 What makes a good evaluation?

22. Example of Evaluation Types Problem: Children are not learning to read at young ages, and they get left behind as they get older Needs AssessmentProcess EvaluationImpact Evaluation Program Evaluation What is the problem? What are ways we can address the problem? Are they being implemented? Solution: An NGO provides after school classes What are the program results? Literacy improves among those students How can we improve going forward?

23. The different types of evaluation Evaluation (M&E) Program Evaluation Impact Evaluation Randomize d Evaluation

24. Needs Assessment Program Theory Assessment Process evaluation Impact evaluation Cost-benefit/Cost-effectiveness analysis 24 Different levels of program evaluation

25. Program Evaluation Evaluation (M&E) Program Evaluation

26. Program Evaluation: community mobilization in education

27. Clear sense of target population Clear sense of need program will fill 27 Needs Assessment Clear articulation of program benefits Clear sense of alternatives

28. How will the program address the needs put forth in your needs assessment?  What are the prerequisites to meet the needs?  How and why are those requirements currently lacking or failing?  How does the program intend to target or circumvent shortcomings?  What services will be offered? 28 Program Theory Assessment

29. Are basic tasks being completed? Are the services being delivered? Is the intervention reaching the target population? Is the intervention being completed well or efficiently and to the beneficiaries’ satisfaction? Process Evaluation

30. Impact Evaluation Evaluation (M&E) Program Evaluation Impact Evaluation

31. 31 The program happened, how did it change lives? What does Theory of Change say we might expect to change? Primary outcome (impact): did community monitoring cause children to learn more (e.g. measured by test scores)? Also distributional questions such as, did the program impact some children more than others? Impact Evaluation

32. What would have happened in the absence of the program? Take the difference between what happened (with the program) …and -what would have happened (without the program) =IMPACT of the program How to measure impact?

33. Impact: What is it? Time Primary Outcome Impact Counterfactual Intervention

34. The counterfactual Ideally, we would want to observe how the exact same subject would react having not received and received the program. This is the counterfactual – what otherwise would have occurred to the same group of people if the program had not been given to them. Unfortunately, this is impossible to observe Thus, we must try to invent an approximation of the counterfactual – comparison groups that are as similar as possible; where the only difference between them is whether they receive the program or not Discussion Point: Why is this important?  If we don’t have two groups that are similar, then if we see a difference between them at the end of the trial, that difference could be the result of a million other factors, rather than the program, and we cannot say whether the program or something else caused the difference.

35. Counterfactual is often constructed by selecting a group not affected by the program Randomized:  Use random assignment of the program to create a control group which mimics the counterfactual. Non-randomized:  Argue that a certain excluded group mimics the counterfactual. 35 Constructing the counterfactual

36. When we answer a process question, we need to describe what happened. When we answer an impact question, we need to compare what happened to what would have happened without the program 36 How impact differs from process?

37. Experimental:  Randomized Evaluations Quasi-experimental  Instrumental Variables  Regression Discontinuity Design Non-experimental  Pre-post  Difference in differences  Regression  Matching 37 Methodologies in impact evaluation

38. Questions ?

39. II. IMPACT EVALUATION February 24, 2011 What is an impact assessment? Why randomize? The keys to a good impact assessment

40. Preview: why randomize? Evaluation (M&E) Program Evaluation Impact Evaluation Randomize d Evaluation

41. The Four W’s and How To of Randomization  What  Why  When  Where  How To

42. What is a randomized evaluation? Think clinical test!  EX: Say we want to test a new diet pill  We draw lots among the 200 volunteers: 100 receive the pill, 100 do not. This is key between we get two groups of 100 similar people in all respects (same initial weight, same motivation, etc.).  The clinical test allows us to say something about the effect of the intervention by comparing the average outcome for each group after treatment.  Avg Treatment Weight < Avg Control Weight  The pill!

43. Now, let’s apply an RE to a program Suppose you have the ability to reach 250 new beneficiaries in your program. Before starting the program : 1. Make a list of 500 eligible individuals. 2. Perform a lottery and the 250 who are selected form the treatment group. * We get two groups that are similar in all respects* 3. Rollout the program to the 250 selected individuals only. 4. After you feel that the program has been in operation long enough to observe an effect, interview both groups – the 250 (treatment) and 250 (comparison)

44. Accountability Lesson learning Reduced poverty through more effective programs Solid reliable impact evaluations are the building blocks for more general lesson learning 44 Why? -- Let’s Review! Objectives for evaluation

45. Example: Pratham’s Read India program Why randomize ? Non-randomized methods are often biased

46. Needs assessment gives you the metric for measuring impact Process evaluation gives you the costs of all the inputs Impact evaluation gives you the quantified benefits Identifying alternatives allows for comparative cost benefit 46 Clear cost-benefit analysis

47. 47 Eg: Comparative cost effectiveness

48. Learning from a single study Learning from an accumulation of studies We learn more if we place each evaluation inside a general learning strategy, helps us be explicit about  questions we want to answer  hypotheses we want to test 48 When ? Learning from impact evaluations

49. When there is an important question you want/need to know the answer to Timing--not too early and not too late Program is representative not gold plated  Or tests an basic concept you need tested Time, expertise, and money to do it right Develop an evaluation plan to prioritize 49 When to do a randomized evaluation?

50. When the program is premature and still requires considerable “tinkering” to work well When the project is on too small a scale to randomize into two “representative groups” If a positive impact has been proven using rigorous methodology and resources are sufficient to cover everyone After the program has already begun and you are not expanding elsewhere When NOT to do an RE

51. Where?  YOUR PROGRAM

52. Determine key questions for your organization How many can you already answer? Select top priority questions for impact evaluation Establish plan for answering them 52 How? Developing an evaluation strategy

53. Things to Keep in Mind :  With key questions answered from impact evaluations, process evaluation can give your overall impact  A few high quality impact studies are worth more than many poor quality ones  If you ask the right question, more likely to use it 53 How? Developing an evaluation strategy

54. The steps for a randomized evaluation Compile a list of 200 eligible villages Complete a survey based on a sample of the households in these villages Randomize who receives and not 100 treatment villages100 control villages Intervention: fertilizer distribution Complete another survey for all the villages Impact = T-C Hypothesis: The NGO has the capacity to reach 100 new villages with fertilizer distribution

55. Randomly sample from area of interest Random sampling and random assignment

56. Randomly sample from area of interest Randomly assign to treatment and control Random sampling and random assignment Randomly sample from both treatment and control

57. Different levels of randomization are possible depending on the context The randomization can be both at the individual level of the level of villages (or neighborhoods), classes, schools, or community health center, etc. Examples: Micro-credit in a city  individual level Awareness campaigns  village level Encourage to vaccinate  community health center

58. Questions ?

59. II. IMPACT EVALUATION 24 February 2011 What is a good impact assessment? Why randomize? The keys to a good impact assessment

60. Why measure the impact?  The intuition about a program’s effect is often misleading  Measuring the impact of several alternative programs allows for the optimal allocation of resources  Impact assessments increase the confidence of donors.  If you design the evaluation in advance, stick to the plan and are open about the results, it shows you’re willing to be accountable, to learn and improve you projects  The impact assessment can also be carried out by an independent organization  Much thought goes into an impact assessment, beyond the simple question of effect. It can tell us:  The best way to implement the program  Who to target  Several questions at the same time

61. Sample size The number of units measured in a study is also important The more people or households or schools we can collect data from, the better we can estimate the broader effects of a program.

62. Sample Size Exercise Take a coin and flip it twice  How many heads did each person have? Flip it 10 times Flip it 20 times As the number of flips increases, the data we collect gets closer to the real principle operating behind the question we are trying to answer: what is the probability that the coin lands heads?

63. Sample size, continued How large we make our sample will depend on the statistics behind choosing enough units to allow us to see if there is a change because of the program I.e., if we chose two people, gave one the program and the other not, could we say that any change we see in the “treated” person is reliable? Sample size may also tradeoff with funding and monitoring constraints

64. Selection bias Attrition Spillover Hawthorne effect Bias that may influence a study

65. Selection bias is present when the sample does not represent the larger population from which it was drawn Sometimes even random samples can be influenced by the type of people who are chosen to participate For example:  Suppose a pork seller wants to see how if a discount would increase his sales  So you decide to test whether a coupon for a 50% discount off a first purchase will influence if people are willing to buy pork or not  So you go to Nima and find 100 people on the street who agree to be interviewed and you randomly give 50 a coupon for 50% off their first purchase of pork.  What could go wrong? Selection bias will influence the outcome of the study positively or negatively in a way that does not accurately reflect how the study would have faired within the population you are hoping design a program for Selection bias

66. Attrition is when people dropout of your sample in a way that influences the outcome you hope to estimate.  People can attrit by refusing to continue to participate, moving without follow-up contact information, dying For example:  The University of Cape Coast wants to see how an internship will affect the future careers of its business students  So 100 students agree to participate and 50 are randomly selected to be an intern over the summer with an investment firm in Accra; 5 years later, researchers return to find 60 to interview. What could be a source of bias? Attrition affects the comparison one makes at the end of analysis. It is important to think about  who “dropped out” during the evaluation,  what those people were like, and  if the “drop out” rate was different between the treatment and the control groups Attrition

67. Spillover influences the “treatment effect” estimated at the end of an evaluation – some of the control group receiving the treatment influences the comparison we make at the end For example:  Imagine that a bank wants to increase use of its savings program among market women:  100 in one market, 50 get the offer. What could go wrong? Spillover affects the comparison you can make at the end of the evaluation; there are two kinds Spillover

68.  Intention to treat – the effect of the program on those you intended to be affected will be a simple difference between treatment and control  Treatment on the treated – the effect of the program on those who received the treatment (being given, directly, a pamphlet) by taking the intention to treat estimate and dividing by the differences in treatment receipt rates between treatment and control  E.g., if 95% of the women who received the pamphlet started an account and 85% of women who did not receive the pamphlet started an account, assuming that the intention to treat estimate effect was 0.1 percentage point difference between treatment and control, then the treatment on the treated estimate would be 0.1/(0.95 – 0.85) or 1%, significantly higher!  Discussion Point – Which do you feel is more important to informing policy – Treatment on the treated or intention to treat? Spillover, continued

69. The fact of being part of a study may influence the outcomes that are observed This effect was observed in the United States in a factory  A researcher changed certain things about the factory – the lighting, length of breaks, providing food  Each time a change was made, productivity increased because the workers knew they were being monitored E.g., think of a weight-loss medicine study in which participants know that their weight will be monitored – people in the control group may exercise and eat healthier because they are more aware of their weight Measuring this effect still happens today and is important to consider when conducting an evaluation Hawthorne effect

70. Questions ?

71. AGENDA I. Introduction to IPA II. Impact evaluation III. Lessons from evaluations: Africa & beyond IV. Discussion and exercise: designing an evaluation for your sector

72. Lessons from our impact assessments Education Health Microfinance Agriculture/Food security Governance

73. Education: Deworming in schools Hundreds of millions of children worldwide are infected with intestinal parasites. These parasites affect their health, cognitive development, schooling, etc. An assessment conducted by IPA in Kenya and in India revealed the immense impact of deworming campaigns in schools IPA now encourages deworming initiatives worldwide: in 2009, 20 million children worldwide have benefited from the program For only $0.50 per child per year: you can reduce student absence by 25%!

74. Health (1/2): Distribute free malaria nets Nets can reduce child mortality by 20%. Reducing malaria also has an impact on learning and economic conditions. However, few pregnant women and children use them How can we increase their use?  Our study aimed to inform debate about the best price http://www.povertyactionlab.org/policy-lessons/health/pricing-health- products Free distribution does not increase use - but Charging does not imply that those who need it will buy more Demand reduced by 60%, from free to 3 GHC Free distribution can save more lives, and has a lower cost per life saved (through indirect effects)

75. Health (2/2): Place chlorine dispensers at community water sources Diarrhea accounts for 20% of infant mortality in the world. They are related in large part to contamination of drinking water In western Kenya, three quarters of household are aware that treatment with chlorine reduces risk of contamination, but only 5% use it. http://www.povertyactionlab.org/policy- lessons/health/child-diarrhea Establish chlorine dispenser at water sources in communities is the best strategy to improve water quality.

76. Microfinance: Taking account of psychology in the design of savings products Nearly everywhere, men and women say that they do not save as much as they want to IPA has tested many elements that take into account psychological aspects to save, thereby building a safer future http://poverty-action.org/project/0071 Some simple interventions can increase deposits Ghana: the opportunity to give the account a name and a savings target has increased deposits by as much as 50% Peru, Bolivia, Philippines: SMS reminders to save Philippines: to enable beneficiaries to embark on savings (deposit accounts, savings schemes, etc.) increased deposits may even help stop smoking!

77. Agriculture and Food security: allow farmers to commit to use fertilizers Fertilizers can increased yields by 70% in western Kenya, but only 40% of farmers use them http://www.povertyactionlab. org/evaluation/nudging- farmers-use-fertilizer- experimental-evidence-kenya Coming to sell them at the time of harvest for fertilizer to be delivered (using vouchers) in the next planting season increased fertilizer use, with a subsidy, by 50%.

78. Decentralization / Governance: still no clear answer to the question of community control There is a great debate on the issue of community involvement in monitoring corruption. Our results differ Indonesia: audits by an external official authority were more effective than community controls to limit corruption in the construction of local roads Kenya: temporary teachers employed and supervised by a local committee were more efficient than teaching staff. India: no impact for local committees to control schools. Uganda: involvement of citizens in the functioning of health facilities improved attendance, visits, and has an impact on health indicators. http://www.povertyactionlab.org/policy-lessons/governance/community-participation We need more studies in this area!

79. Thank you for your attention!

80. AGENDA I. Introduction to IPA II. Impact evaluation III. Lessons from evaluations: Africa & beyond IV. Discussion and exercise: designing an evaluation for your sector

81. Contact information www.poverty-action.org Email : gtourek@poverty-action.org Tel : (233) 026 806 8213 IPA Ghana Ako Adjei, Osu PMB 57 Accra 0302 745 570