1. Measurement Marc Shotland J-PAL Global

2. Course Overview 1.What is evaluation? 2.Measuring impacts (outcomes, indicators) 3.Why randomize? 4.How to randomize 5.Threats and Analysis 6.Sampling and sample size 7.Cost Effectiveness Analysis and Scaling Up 8.RCT: Start to Finish

3. Course Overview 1.What is evaluation? 2.Measuring impacts (outcomes, indicators) 3.Why randomize? 4.How to randomize 5.Threats and Analysis 6.Sampling and sample size 7.Cost Effectiveness Analysis and Scaling Up 8.RCT: Start to Finish

4. Theory of Change Less Diarrhea Contaminated water is primary source of illness Drink Clean water Have access to clean water at come Access to clean water at source Choose to collect only clean water No recontamination Hygiene practices Sufficient water Clean method of extracting water Choose to drink only clean water Understand benefits of clean water Know which water is clean

5. Log Frame Objectives Hierarchy IndicatorsSources of Verification Assumptions / Threats Impact (Goal/ Overall objective) Lower rates of diarrhea Rates of diarrhea Household survey Waterborne disease is primary cause of diarrhea Outcome (Project Objective) Households drink cleaner water (Δ in) drinking water source; E. coli CFU/100ml Household survey, water quality test at home storage Shift away from dirty sources. No recontamination Outputs Source water is cleaner; Families collect cleaner water E. coli CFU/100ml; Water quality test at source continued maintenance, knowledge of maintenance practices Inputs (Activities) Source protection is built Protection is present, functional Source visits/ surveys Sufficient materials, funding, manpower Source: Roduner, Schlappi (2008) Logical Framework Approach and Outcome Mapping, A constructive Attempt of Synthesis,

6. Lecture Overview 1.Case Study Review and follow-up 2.Theory of Change, Outcomes, Indicators 3.Indicators: Use in analysis 4.Sources of Data 5.Collecting Data

7. CASE STUDY Women as Policymakers

8. What was the main purpose of the 73 rd Amendment of India’s constitution A.To reserve leadership positions for women (and caste minorities) B.To formalize local institutions of leadership C.To give women the right to vote in local elections

9. Theory of Change Public goods reflect Women’s preferences Women have different preferences Investments reflect women’s preferences Pradhan’s preferences matter Imperfect democracy Some democracy More female pradhans Reservations for Women Women are empowered

10. Log Frame Objectives Hierarchy IndicatorsSources of Verification Assumptions / Threats Impact (Goal/ Overall objective) Lower rates of diarrhea Rates of diarrhea Household survey Waterborne disease is primary cause of diarrhea Outcome (Project Objective) Households drink cleaner water (Δ in) drinking water source; E. coli CFU/100ml Household survey, water quality test at home storage Shift away from dirty sources. No recontamination Outputs Source water is cleaner; Families collect cleaner water E. coli CFU/100ml; Water quality test at source continued maintenance, knowledge of maintenance practices Inputs (Activities) Source protection is built Protection is present, functional Source visits/ surveys Sufficient materials, funding, manpower Source: Roduner, Schlappi (2008) Logical Framework Approach and Outcome Mapping, A constructive Attempt of Synthesis,

11. Data used Sources of MeasurementIndicators Household (HH) SurveyDeclared HH preferences HH perceptions of quality of public goods and services Village Leader InterviewPolitical experience Investments undertaken Village PRAVillage infrastructure + investments Perception of public good quality Participation of men and women Issues Administrative DataBudgets Balance sheets Transcript from village meetingWho speaks and when (gender) Issues raised

12. Results West BengalRajasthan IssueInvestment Issue Reserved Investment Issue Reserved Investment WMWM Drinking Water# facilities31%17%9.0954%49%2.62 Road Improvement Road Condition (0-1) 31%25%0.1813%23%-0.08 Irrigation# facilities4%20%-0.382%4%-0.02 EducationInformal education center 6%12%-0.065%13%

13. INDICATORS Use in analysis

14. The main challenge in measurement Accuracy Precision

15. The main challenge in measurement Validity Reliability

16. Validity How well does the indicator map to the outcome? –Standardized Tests, Income, empowerment Are you getting unbiased answers? –Social desirability bias (response bias) –Framing effect –Recall bias –Anchoring bias

17. Reliability The measure is very noisy, but on average, correct

18. Which is worse? A.Poor Validity B.Poor reliability C.Equally bad D.Depends E.Don’t know/can’t say

19. Validity When there’s a baseline When there’s a comparison group When bias is correlated with treatment

20. “Consistently Biased” BaselineEndlineDifference Treatment Control truth estimates

21. Bias is correlated with treatment BaselineEndlineDifference Treatment Control truth estimates

22. Reliability AND Validity Question wording Surveyor training/quality Data entry Translation Length, fatigue

23. Reliability and error Question wording –Definitions –Relationships –Recall period Answer choice –Open/Closed –Single v. Multiple option –Likert –Ranked response –Units

24. How frequently do you drive your car? A.Never B.1-2 C.3-5 D.6-10 E.More than 10 F.Don’t know/can’t say

25. The problem With the following questions…

26. Outcome: annual consumption Indicator: food expenditure in last week A.Validity B.Reliability C.Both D.Neither

27. Outcome: annual consumption Indicator: food expenditure in last three months A.Validity B.Reliability C.Both D.Neither

28. Question: have you had sex in the past week? [if yes] Did you use protection? A.Validity B.Reliability C.Both D.Neither

29. Other important considerations Ethics Might affect compliance

30. Why do you need data? Outcomes Sub-groups Covariates Predictors of compliance

31. SOURCES OF DATA

32. Where can we get data? Administrative Data Other Secondary Data Primary Data

33. Primary Data Collection Self-reported Surveys Exams, tests, etc Games Vignettes Direct Observation Diaries/Logs

34. Modules Income, consumption, expenditure Perceptions, expectations, aspirations Bargaining power Patience, risk Behavior (time use) Anthropometric Cognitive, Learning Yields

35. Public good investments Public goods reflect Women’s preferences Women have different preferences Investments reflect women’s preferences Pradhan’s preferences matter Imperfect democracy Some democracy More female pradhans Reservations for Women Women are empowered

36. Women empowerment? Women empowerment Women have different preferences Investments reflect women’s preferences Pradhan’s preferences matter Imperfect democracy Some democracy More female pradhans Reservations for Women Women are empowered

37. Outcome: Gender Bias Question: How effective is your leader? (ineffective, somewhat effective, effective, very…) A.Validity B.Reliability C.Both D.Neither

38. Perceptions and Attitudes “How effective is your leader?” (ineffective, somewhat effective, effective, very…) Listen to a Vignette (Male v. Female) Revealed preference – voting behavior Implicit Association tests

39. Implicit Association Test

40. Enter question text... Academic or Good Practitioner or Bad A.Left B.Right Happiness

41. Enter question text... Academic or Bad Practitioner or Good A.Left B.Right Pleasure

42. Enter question text... Practitioner or Bad Academic or Good A.Left B.Right Professor

43. Enter question text... Practitioner or Good Academic or Bad A.Left B.Right Happiness

44. Enter question text... Practitioner or Good Academic or Bad A.Left B.Right Research

45. DATA COLLECTION Considerations

46. Data Collection Considerations Quality Control Surveyor training Surveyor (gender) composition Human subjects Data Security Electronic v paper Costs

47. When to collect Data Baseline During the intervention Endline Scale-up, intervention

48. Results from Women Empowerment Significant electoral gains for women in subsequent unreserved elections Changed perceptions of women’s ability to lead effectively Heightened career aspirations of adolescent girls and increased level of educational attainment