1. Precision Gains from Publically Available School Proficiency Measures Compared to Study-Collected Test Scores in Education Cluster-Randomized Trials June 2010 Presentation to the IES Research Conference John Deke ● Lisa Dragoset ● Ravaris Moore

2.  Data from 8 previous studies  25 different outcome / baseline test score combinations (math and reading)  Median sample size = 48.5 schools Data analysis

3.  Advantages –Available for multiple grades, subgroups, and years –Reading and Math –Inexpensive  Disadvantages –States define proficiency differently –Data is often for a different cohort –State test may not be aligned with post test School-level proficiency data

4.  School-level R 2 using proficiency data is about 1.5 times larger than a model with school district indicators alone (about 0.60 instead of 0.40)  MDE using proficiency data is 0.8 times the MDE from a model with school district indicators alone (about 0.25 instead of 0.30) Proficiency data helps… 4

5.  The school-level R 2 using proficiency data is 4/5 the size of the R 2 from study- collected pre-tests (for example, 0.60 instead of 0.75)  The MDES is 1.31 times larger (for example, 0.25 instead of 0.19)  To compensate for loss of precision, the average study would need to nearly double the number of schools …but not as much as pre-tests

6.  Covariates for missing data – how bad would missing data bias be?  Covariates for subgroup analysis – can we use quantile treatment effects? (separate paper) Other uses for baseline tests

7.  No statistically significant differences in pre-tests between treatment and control groups due to attrition bias  18 out of 25 statistically significant differences in pre-tests between those with and without post-tests (non- respondents have lower pre-tests) Missing outcome data

8.  We recommend collecting student-level pre- test scores when possible  In cases where this is not possible: –school-level proficiency data can provide precision gains, but there are no guarantees –impacts likely retain causal validity, but there may be issues with generalizability –methods exist to estimate variation in impacts by ability level Overall conclusions