Sampling issues in PIES and PETS

But if you care about impact evaluation, register for Module 6 right now ! Impact Evaluation Studies Need to compare two populations, differentiated by –Time before and after the crisis, the boom, the earthquake, the program, … –Space North/South, rural/urban, mountain/forest, … –Policy program vs. non-program treatment vs. control

The basic question Let y 1 and y 2 be the values of an indicator in populations 1 and 2 –Prevalence of malnutrition –Average income –Literacy –etc… We are interested in the difference y 2 - y 1 y 1 and y 2 are both unknown We estimate them from samples The estimations ŷ 1 and ŷ 2 are affected by sampling error What is the error of the difference?

Let e 1 and e 2 be the standard errors of ŷ 1 and ŷ 2 If the samples are independent*, the standard error of ŷ 2 - ŷ 1 is The basic answer * this is the case when the populations are different

Panel Surveys can measure change better ŷ1ŷ1 ŷ2ŷ It seems that y 2 < y1 but… …both measures are affected by sampling errors ( e1 et e2 ) The error of the difference ŷ 2 - ŷ 1 is… …√ ( e² 1 + e² 2 ) if the two samples are independent …only √( e² 1 + e² 2 – 2ρ[y 1,y 2 ] ) if the sample is the same

Advantages and disadvantages of panels Analytical advantages –Can measure changes better –Permit understanding better why things changed –Permits correlating past and present behavior Analytical disadvantages –Become progressively less representative of the population Practical disadvantages –Sample attrition –Much harder to manage –Vulnerable to manipulation –Design them prospectively rather than in afterthought Practical advantages –No sampling design needed for the second and subsequent surveys

For independent samples State of the world H 0 accepted if ŷ 2 – ŷ 1 ≤ D H 0 rejected if ŷ 2 – ŷ 1 > D OK Type I errorOK Type II error P (Type I error) = α P (correctly accepting H 0 ) = 1 - α P (correctly rejecting H 0 ) = 1 – β P (Type II error) = β Decision rule Null hypothesis H 0 true H 0 : y 2 – y 1 = 0 Alternative hypothesis H A true H 0 : y 2 – y 1 = Δ Research findings ŷ 1 ŷ 2 Significance level of the testPower of the test With Simple Random Samples, e ² 1 = σ ² 1 /n 1 and e ² 2 = σ ² 2 / n 2 But real-life samples are almost never SRS, design effects cannot be ignored ! The decision rule based on ŷ 2 – ŷ 1 intends to detect changes in one specific direction (a 1-sided test) To detect changes in any of the two directions, it should be based on | ŷ 2 – ŷ 1 | instead (a 2-sided test) But impact evaluation is almost always one-sided ! One sided: Two sided:

Public Expenditure Tracking Studies PETS need to observe different kinds of units –Large administrative areas (e.g., regions) –Medium administrative areas (e.g. districts) –Service delivery facilities (schools, hospital, clinics…) –Service providers (teachers, doctors, …) –Clients (students, patients, …) –Households Analyses need to account for their hierarchical relationships How should the samples be selected?

?

Two approaches Top-Down (A multi-stage sample) –Select regions first –Then districts in the selected regions –Then Hospitals in the selected districts Bottom-Up –Take a sample of facilities first, –This implicitly defines the sample of districts –Which in turn defines the sample of regions Disadvantages –High design effects –Poor understanding of the upper levels –Vulnerable to unscientific choices Disadvantage –Higher (slightly higher) costs

In each facility To select providers (teachers, doctors, nurses, etc.) –The samples need to be random (perhaps stratified) –The selection has to be entrusted to fieldworkers… –…but it should be repeatable (for supervision) –Can be done with Kish tables, or with random stickers To select clients in health facilities –The selection also needs to be random and has to be entrusted to fieldworkers. –It cannot be repeatable, but the procedures should try to avoid biases due to the Day of the week Time of the day