Measurement Marc Shotland J-PAL Global

Course Overview 1.What is evaluation? 2.Measuring impacts (outcomes, indicators) 3.Why randomize? 4.How to randomize 5.Threats and Analysis 6.Sampling and sample size 7.Cost Effectiveness Analysis and Scaling Up 8.RCT: Start to Finish

Course Overview 1.What is evaluation? 2.Measuring impacts (outcomes, indicators) 3.Why randomize? 4.How to randomize 5.Threats and Analysis 6.Sampling and sample size 7.Cost Effectiveness Analysis and Scaling Up 8.RCT: Start to Finish

Theory of Change Less Diarrhea Contaminated water is primary source of illness Drink Clean water Have access to clean water at come Access to clean water at source Choose to collect only clean water No recontamination Hygiene practices Sufficient water Clean method of extracting water Choose to drink only clean water Understand benefits of clean water Know which water is clean

Log Frame Objectives Hierarchy IndicatorsSources of Verification Assumptions / Threats Impact (Goal/ Overall objective) Lower rates of diarrhea Rates of diarrhea Household survey Waterborne disease is primary cause of diarrhea Outcome (Project Objective) Households drink cleaner water (Δ in) drinking water source; E. coli CFU/100ml Household survey, water quality test at home storage Shift away from dirty sources. No recontamination Outputs Source water is cleaner; Families collect cleaner water E. coli CFU/100ml; Water quality test at source continued maintenance, knowledge of maintenance practices Inputs (Activities) Source protection is built Protection is present, functional Source visits/ surveys Sufficient materials, funding, manpower Source: Roduner, Schlappi (2008) Logical Framework Approach and Outcome Mapping, A constructive Attempt of Synthesis,

Lecture Overview 1.Case Study Review and follow-up 2.Theory of Change, Outcomes, Indicators 3.Indicators: Use in analysis 4.Sources of Data 5.Collecting Data

CASE STUDY Women as Policymakers

What was the main purpose of the 73 rd Amendment of India’s constitution A.To reserve leadership positions for women (and caste minorities) B.To formalize local institutions of leadership C.To give women the right to vote in local elections

Theory of Change Public goods reflect Women’s preferences Women have different preferences Investments reflect women’s preferences Pradhan’s preferences matter Imperfect democracy Some democracy More female pradhans Reservations for Women Women are empowered

Log Frame Objectives Hierarchy IndicatorsSources of Verification Assumptions / Threats Impact (Goal/ Overall objective) Lower rates of diarrhea Rates of diarrhea Household survey Waterborne disease is primary cause of diarrhea Outcome (Project Objective) Households drink cleaner water (Δ in) drinking water source; E. coli CFU/100ml Household survey, water quality test at home storage Shift away from dirty sources. No recontamination Outputs Source water is cleaner; Families collect cleaner water E. coli CFU/100ml; Water quality test at source continued maintenance, knowledge of maintenance practices Inputs (Activities) Source protection is built Protection is present, functional Source visits/ surveys Sufficient materials, funding, manpower Source: Roduner, Schlappi (2008) Logical Framework Approach and Outcome Mapping, A constructive Attempt of Synthesis,

Data used Sources of MeasurementIndicators Household (HH) SurveyDeclared HH preferences HH perceptions of quality of public goods and services Village Leader InterviewPolitical experience Investments undertaken Village PRAVillage infrastructure + investments Perception of public good quality Participation of men and women Issues Administrative DataBudgets Balance sheets Transcript from village meetingWho speaks and when (gender) Issues raised

Results West BengalRajasthan IssueInvestment Issue Reserved Investment Issue Reserved Investment WMWM Drinking Water# facilities31%17%9.0954%49%2.62 Road Improvement Road Condition (0-1) 31%25%0.1813%23%-0.08 Irrigation# facilities4%20%-0.382%4%-0.02 EducationInformal education center 6%12%-0.065%13%

INDICATORS Use in analysis

The main challenge in measurement Accuracy Precision

The main challenge in measurement Validity Reliability

Validity How well does the indicator map to the outcome? –Standardized Tests, Income, empowerment Are you getting unbiased answers? –Social desirability bias (response bias) –Framing effect –Recall bias –Anchoring bias

Reliability The measure is very noisy, but on average, correct

Which is worse? A.Poor Validity B.Poor reliability C.Equally bad D.Depends E.Don’t know/can’t say

Validity When there’s a baseline When there’s a comparison group When bias is correlated with treatment

“Consistently Biased” BaselineEndlineDifference Treatment Control truth estimates

Bias is correlated with treatment BaselineEndlineDifference Treatment Control truth estimates

Reliability AND Validity Question wording Surveyor training/quality Data entry Translation Length, fatigue

Reliability and error Question wording –Definitions –Relationships –Recall period Answer choice –Open/Closed –Single v. Multiple option –Likert –Ranked response –Units

How frequently do you drive your car? A.Never B.1-2 C.3-5 D.6-10 E.More than 10 F.Don’t know/can’t say

The problem With the following questions…

Outcome: annual consumption Indicator: food expenditure in last week A.Validity B.Reliability C.Both D.Neither

Outcome: annual consumption Indicator: food expenditure in last three months A.Validity B.Reliability C.Both D.Neither

Question: have you had sex in the past week? [if yes] Did you use protection? A.Validity B.Reliability C.Both D.Neither

Other important considerations Ethics Might affect compliance

Why do you need data? Outcomes Sub-groups Covariates Predictors of compliance

SOURCES OF DATA

Where can we get data? Administrative Data Other Secondary Data Primary Data

Primary Data Collection Self-reported Surveys Exams, tests, etc Games Vignettes Direct Observation Diaries/Logs

Modules Income, consumption, expenditure Perceptions, expectations, aspirations Bargaining power Patience, risk Behavior (time use) Anthropometric Cognitive, Learning Yields

Public good investments Public goods reflect Women’s preferences Women have different preferences Investments reflect women’s preferences Pradhan’s preferences matter Imperfect democracy Some democracy More female pradhans Reservations for Women Women are empowered

Women empowerment? Women empowerment Women have different preferences Investments reflect women’s preferences Pradhan’s preferences matter Imperfect democracy Some democracy More female pradhans Reservations for Women Women are empowered

Outcome: Gender Bias Question: How effective is your leader? (ineffective, somewhat effective, effective, very…) A.Validity B.Reliability C.Both D.Neither

Perceptions and Attitudes “How effective is your leader?” (ineffective, somewhat effective, effective, very…) Listen to a Vignette (Male v. Female) Revealed preference – voting behavior Implicit Association tests

Implicit Association Test

Enter question text... Academic or Good Practitioner or Bad A.Left B.Right Happiness

Enter question text... Academic or Bad Practitioner or Good A.Left B.Right Pleasure

Enter question text... Practitioner or Bad Academic or Good A.Left B.Right Professor

Enter question text... Practitioner or Good Academic or Bad A.Left B.Right Happiness

Enter question text... Practitioner or Good Academic or Bad A.Left B.Right Research

DATA COLLECTION Considerations

Data Collection Considerations Quality Control Surveyor training Surveyor (gender) composition Human subjects Data Security Electronic v paper Costs

When to collect Data Baseline During the intervention Endline Scale-up, intervention

Results from Women Empowerment Significant electoral gains for women in subsequent unreserved elections Changed perceptions of women’s ability to lead effectively Heightened career aspirations of adolescent girls and increased level of educational attainment