1. Sampling and Power
Slides by Jishnu Das

2. Sample Selection in Evaluation
Population based representative surveys:
Sample representative of whole population
Good for learning about the population
Not always most efficient for impact evaluation
Sampling for Impact evaluation
Balance between treatment and control groups
Power  statistical inference for groups of interest
Concentrate sample strategically
Survey budget as major consideration
In practice, sample size is often set by budget
Concentrate sample on key populations to increase power

3. Purposive Sampling:
Risk: We will systematically bias our sample, so results don’t generalize to the rest of the population or other sub-groups
Trade off between power within population of interest and population representation
Results are internally valid, but not generalizable.

4. Type I and type II errors
Type I error: Reject the null hypothesis when it is true
Significance level  probability of rejecting the null when it is true (Type I error)
Type II error: Accept (fail to reject) the null hypothesis when it is false
Power  probability of rejecting the null when an alternative null is true (1-probability of Type II)
We want to minimize both types of errors
Increase sample size

5. Survey - Sampling Population: all cases of interest
Sampling frame: list of all potential cases
Sample: cases selected for analysis
Sampling method: technique for selecting cases from sampling frame
Sampling fraction: proportion of cases from population selected for sample (n/N)

6. Sampling Frame
Simple Sampling – almost never practical unless universe of interest is geographically concentrated
Cluster Sampling – randomly choose clusters and then randomly choose units within the cluster. Effective sample size is less than actual number of observations. This is the design, or cluster, effect
The design effect implies that, for a given sized sample, the variance increases [1 + (E-1)] where E is the number of elements in each cluster and  is the intra-class correlation, a measure of how much the observations with in a cluster resemble each other.

7. Using Power Calculations to Estimate Sample Sizes
What is the size sample needed to be able to find a difference in means at a given statistical significance.
Need idea of what difference is a plausible expectation for the intervention.
Fixing the confidence level, we observe two things when increasing sample size:
the rejection region gets larger and
the power increases

8. In Practice - I
Many sample patterns possible especially when one can vary cluster numbers and cluster sizes
May use simulations in Stata or similar package. They easily account for complicated designs
Panel and dif-in-dif calculations need to be based on ability to find significance of changes, not difference in levels. Requires an estimate of correlation over rounds
Sample needed to find difference between alternative treatments is different than that needed to compare to control

9. In Practice - II
Number of clusters improves precision and is important especially in randomized designs.
Not strictly necessary that treatment and control are equal in size or number of clusters but analysis is complicated if probability of selection differs.
Importance of transparency in randomization process
Many medical journal require registering trials prior to analysis (to avoid reporting only ‘favorable’ results).

10. An Example
Does Information improve child performance in schools? (Pakistan)
Randomized Design
Interested in villages where there are private schooling options
What Villages should we work in?
Stratification: North, Central, South
Random Sample: Villages chosen randomly from list of all villages with a private school

11. In Practice: An Example
How many villages should we choose?
Depends on:
How many children in every village
How big do we think the treatment effect will be
What the overall variability in the outcome variable will be

12. In Practice: An Example
Simulation Tables
Table 1 assumes very high variability in test-scores.
X,Y: X is for intervention with small effect size; Y for larger effect size
N: Significant < 1% of simulations
S: Significant < 10% of simulations
A: Significant > 99% of simulations

13. In Practice: An Example
Simulation Tables
Table 1 assumes lower variability in test-scores.
X,Y: X is for intervention with small effect size; Y for larger effect size
N: Significant < 1% of simulations
S: Significant < 10% of simulations
A: Significant > 99% of simulations

14. When do we really worry about this?IF
Very small samples at unit of treatment!
Suppose treatment in 20 schools and control in 20 schools
But there are 400 children in every school
This is still a small sample
Interested in sub-groups (blocks)
Sample size requirements increase exponentially
Using Regression Discontinuity Designs