1. TRANSLATING RESEARCH INTO ACTION Sampling and Sample Size Marc Shotland J-PAL HQ

2. Outline Sampling distributions Detecting impact

3. Outline Sampling distributions – population distribution – sampling distribution – law of large numbers/central limit theorem – standard deviation and standard error Detecting impact

4. Balsakhi baseline test scores

5. Mean = 26

6. Standard Deviation = 20

7. Good, the average of the sample is about 26… Population v. sampling distribution

8. But… … I remember that as my sample goes, up, isn’t the sampling distribution supposed to turn into a bell curve? (Central Limit Theorem) Is it that my sample isn’t large enough?

9. This is the distribution of my sample of 10,000 students! Population v. sampling distribution

10. Possible results & probability: 1 die This is our Population Distribution (a uniform distribution)

11. What’s the average result? If you were to roll a die once, what is the “expected result”? (i.e. the average)

12. Rolling 1 die: possible results & average

13. What’s the average result? If you were to roll two dice once, what is the expected sum of the two dice?

14. Rolling 2 dice: possible totals 12 possible totals, 36 permutations Die 1 Die 2 234567 345678 456789 5678910 6789 11 789101112

15. Rolling 2 dice: Possible totals & likelihood

16. What’s the average result? If you were to roll two dice once, what is the expected sum of the two dice?

17. What’s the average result? If you were to roll one die twice, what is the expected average of the two dice?

18. Rolling 2 dice: Average of dice & likelihood This is called a SAMPLING DISTRIBUTION of the mean

19. Sampling distribution Gives you: 1.All possible outcomes 2.The likelihood of each of those outcomes – Each column represents one possible outcome (average result) – Each block within a column represents one possible permutation (to obtain that average) 2.5

20. Rolling 3 dice: 16 results 3  18, 216 permutations

21. Rolling 4 dice: 21 results, 1296 permutations

22. Rolling 5 dice: 26 results, 7776 permutations

23. Rolling 10 dice: 50 results, >60 million permutations Looks like a bell curve, or a normal distribution

24. Rolling 30 dice: 150 results, 2 x 10 23 permutations* about as many as there are stars in the universe

25. Rolling 100 dice: 500 results, 6 x 10 77 permutations >99% of all rolls will yield an average between 3 and 4

26. Sampling in the real world When you take a sample from the real world, you usually get one shot only The population distribution isn’t always as nice and tidy as the uniform distribution of the dice

27. Population v. sampling distribution: N=1

28. Sampling Distribution: N=4

29. Law of Large Numbers: N=9

30. Law of Large Numbers: N=100

31. Central Limit Theorem: N=1

32. Central Limit Theorem : N=4

33. Central Limit Theorem : N=9

34. Central Limit Theorem : N =100

35. Standard deviation/error What’s the difference between the standard deviation and the standard error?

36. Standard Deviation/ Standard Error

37. Sample size ↑ x4, SE ↓ ½

38. Sample size ↑ x9, SE ↓ ?

39. Sample size ↑ x100, SE ↓?

40. Outline Sampling distributions Detecting impact – significance – effect size – power – baseline and covariates – clustering – stratification

41. Baseline test scores

42. Was there an impact? Endline test scores

43. Stop! That was the control group. The treatment group is green. Post-test: control & treatment

44. This is the true difference between the 2 groups Average difference: 6 points

45. Population – It is the truth – But is it due to the program? – Uncertain: we must rely on a combination of: the randomized design, and Statistical properties Population versus Sample

46. Sampling When you take a single random sample (one roll of the dice) you get only one permutation (and therefore one outcome) If you didn’t know the true average, that would be your best estimate of the average If it was a true random roll, it would be unbiased

47. Sample – How many children would we need to randomly sample to detect that the difference between the two groups is statistically significantly different from zero? OR – How many children would we need to randomly sample to approximate the true difference with relative precision? Population versus Sample

48. If our measurements show a difference between the treatment and control group, our first assumption is: – In truth, there is no impact (our H 0 is still true) – There is some margin of error due to sampling – “This difference is solely the result of chance (random sampling error)” We (still assuming H 0 is true) then use statistics to calculate how likely this difference is in fact due to random chance Hypothesis testing

49. If it is very unlikely (less than a 5% probability) that the difference is solely due to chance: – We “reject our null hypothesis” We may now say: – “our program has a statistically significant impact” Hypothesis testing: conclusions

50. Are we now 100 percent certain there is an impact? – No, we are only 95% confident – And we accept that if we use that 5% threshold, this conclusion may be wrong 5% of the time – That is the price we’re willing to pay since we can never be 100% certain – Because we can never see the counterfactual, We must use random sampling and random assignment, and rely on statistical probabilities Hypothesis testing: conclusions

51. Testing statistical significance

52. Significance: control sampling dist

53. “Significance level” (5%) Critical region

54. “Significance level” (5%) equals 5% of t h i s t ota l a r e a

55. Significance: Sample size = 4

56. Significance: Sample size = 9

57. Significance: Sample size = 100

58. Significance: Sample size = 6,000

59. POWER! What if the probability is greater than 5%? – We can’t reject our null hypothesis – Are we 100 percent certain there is no impact? No, it just didn’t meet the statistical threshold to conclude otherwise – Perhaps there is indeed no impact – Or perhaps there is impact, But not enough sample to detect it most of the time Or we got a very unlucky sample this time How do we reduce this error? Hypothesis testing: conclusions

60. When we use a “95% confidence interval” How frequently will we “detect” effective programs? That is Statistical Power Hypothesis testing: conclusions

61. YOU CONCLUDE EffectiveNo Effect THE TRUTH Effective Type II Error (low power)  No Effect Type I Error (5% of the time)  61 Hypothesis testing: 95% confidence

62. Power: main ingredients 1.Effect Size 2.Variance 3.Sample Size 4.Proportion of sample in T vs C 5.Clustering 6.Take-up

63. 1.Effect Size to be detected – The more fine (or more precise) the effect size we want to detect, the larger sample we need – Smallest effect size with practical / policy significance? 2.Variance – The more “noisy” it is to start with, the harder it is to measure effects 3.Sample Size – The more children we sample, the more likely we are to obtain the true difference 4.Proportion of sample in T vs C 5.Clustering and rho 6.Take-up (back to effect size) 63 Power: main ingredients

64. To calculate statistical significance we start with the “null hypothesis”: To think about statistical power, we need to propose a secondary hypothesis Effect Size

65. Effect Size: standard normal 1 Standard Deviation

66. Effect Size: 1 standard deviation 1 Standard Deviation

67. Effect Size: 3 standard deviations 3 Standard Deviations

68. Power: control group = H 0

69. Power: reject H 0 in critical region

70. Power: true effect is 1 SD

71. Power: when is H 0 rejected?

72. Power: 26%

73. Power: assume impact = 1 SD

74. Power: assume impact = 3 SDs

75. Power: 91%

76. DO NOT USE: “Expected” effect size What is the smallest effect that should justify the program being adopted? If the effect is smaller than that, it might as well be zero: we are not interested in proving that a very small effect is different from zero In contrast, if any effect larger than that would justify adopting this program: we want to be able to distinguish it from zero 76 Picking an effect size

77. How large an effect you can detect with a given sample depends on how variable the outcome is. The Standardized effect size is the effect size divided by the standard deviation of the outcome Common effect sizes 77 Standardized effect sizes

78. An effect size of… Is considered……and it means that… 0.2ModestThe average member of the treatment group had a better outcome than the 58 th percentile of the control group 0.5LargeThe average member of the treatment group had a better outcome than the 69 th percentile of the control group 0.8Whoa…that’s a big effect size! The average member of the treatment group had a better outcome than the 79 th percentile of the control group Standardized effect size

79. Try: Increasing the sample size You should not alter the effect size to achieve power The effect size is more of a policy question One variable that can affect effect size is take-up! – If your job training program increases income by 20% – But only ½ of the people in your treatment group participate – You need to adjust your impact estimate accordingly From 20% to 10% So how do you increase power? Effect Size: Bottom Line

80. Power: main ingredients 1.Effect Size 2.Variance 3.Sample Size 4.Proportion of sample in T vs C 5.Clustering 6.Take-up

81. Power: assume sample size = N

82. Power: assume sample size = 4*N

83. Power: 64%

84. Power: assume sample size = 9*N

85. Power: 91%

86. Power: main ingredients 1.Effect Size 2.Variance 3.Sample Size 4.Proportion of sample in T vs C 5.Clustering 6.Take-up

87. Sample split: 50% C, 50% T

88. Power: 91%

89. Sample split: 50% C, 50% T

90. Sample split: 25% C, 75% T

91. Power: 83%

92. Sample split: 75% C, 25% T

93. Power: 83%

94. Allocation to T v C

95. Power: main ingredients 1.Effect Size 2.Variance 3.Sample Size 4.Proportion of sample in T vs C 5.Clustering 6.Take-up

96. There is sometimes very little we can do to reduce the noise The underlying variance is what it is We can try to “absorb” variance: – using a baseline – controlling for other variables Variance

97. Baseline and Covariates

113. Power: main ingredients 1.Effect Size 2.Variance 3.Sample Size 4.Proportion of sample in T vs C 5.Clustering 6.Take-up

114. You want to know how close the upcoming national elections will be Method 1: Randomly select 50 people from entire Indian population Method 2: Randomly select 5 families, and ask ten members of each family their opinion 114 Clustered design: intuition

115. If the response is correlated within a group, you learn less information from measuring multiple people in the group It is more informative to measure unrelated people Measuring similar people yields less information 115 Clustered design: intuition

116. Cluster randomized trials are experiments in which social units or clusters rather than individuals are randomly allocated to intervention groups The unit of randomization (e.g. the school) is broader than the unit of analysis (e.g. students) That is: randomize at the school level, but use child-level tests as our unit of analysis 116 Clustered design

117. The outcomes for all the individuals within a unit may be correlated We call r (rho) the correlation between the units within the same cluster 117 Consequences of clustering

118. Like percentages, r must be between 0 and 1 When working with clustered designs, a lower r is more desirable It is sometimes low, 0,.05,.08, but can be high:0.62 118 Values of r (rho) Madagascar Math + Language0.5 Busia, Kenya Math + Language0.22 Udaipur, India Math + Language0.23 Mumbai, India Math + Language0.29 Vadodara, India Math + Language0.28 Busia, Kenya Math0.62

119. Analysis: The standard errors will need to be adjusted to take into account the fact that the observations within a cluster are correlated. Adjustment factor (design effect) for given total sample size, clusters of size m, intra-cluster correlation of r, the size of smallest effect we can detect increases by compared to a non-clustered design Design: We need to take clustering into account when planning sample size 119 Implications for design and analysis

120. Study# of interventions (+ Control) Total Number of Clusters Total Sample Size Women’s Empowerment2Rajasthan: 100 West Bengal: 161 1996 respondents 2813 respondents Pratham Read India4280 villages17,500 children Pratham Balsakhi2Mumbai: 77 schools Vadodara: 122 schools 10,300 children 12,300 children Kenya Extra Teacher Program 8210 schools10,000 children Deworming375 schools30,000 children Few examples of sample size

121. Testing multiple treatments ←0.15 SD→ ↖ 0.25 SD ↘ ↑ 0.15 SD ↓ ↑ 0.10 SD ↓ ←0.10 SD→ ↗ 0.05 SD ↙ 50 100 200

122. Stratification Why? – Power: increasing precision for the same sample size What? – With large samples, stratify on everything – With small samples, stratify on variables that explain most variation in outcome How? – Using baseline and/or administrative data