1. A Comparison of Variance Estimates for Schools and Students Using Taylor Series and Replicate Weighting Ellen Scheib, Peter H. Siegel, and James R. Chromy RTI International Presented at Third International Conference on Establishment Surveys (ICES-III) June 21, 2007 RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box 12194 ■ Research Triangle Park, NC 27709 Phone 919-541-6000e-mail rti@rti.org

2. 2 Acknowledgements The data used in this presentation were produced for the U.S. Department of Education, National Center for Education Statistics (NCES), under Project no. 0207818 The views expressed in this presentation do not necessarily reflect the official policies of NCES or RTI International; nor does mention of trade names, commercial practices, or organizations imply endorsement by the U.S. Government

3. 3 Introduction – Background and Purpose Choices with replication ●One or two sampling stages ●Some or all weight adjustments ●Overall or replicate-level control totals ●Finite population correction (fpc) Replication examples in NCES studies ●National Postsecondary Student Aid Study (NPSAS) ●School and Staffing Survey (SASS) ●Education Longitudinal Study of 2002 (ELS:2002)

4. 4 Introduction – Variance Estimation Methods Taylor series linearization Replication ●Jackknife ●Bootstrap ●BRR

5. 5 Introduction – Overview of ELS:2002 Sample design ●Base-year ●First follow-up ●Transcript ●Second follow-up ■Weighting ●Nonresponse adjustment ●Poststratification/calibration

6. 6 Replication of School Sampling Stage Formed strata and PSUs for all sample schools Collapsed strata 200 replicates FPC not necessary

7. 7 Replication of Student Sampling Stage Same strata and PSUs as for schools Used school BRR weight to help compute initial student BRR weight Used prior round BRR weight as starting point for current round BRR weight

8. 8 Replication of Nonresponse Adjustment 1 adjustment for the school weight 2 adjustments for each student weight Deleted variables from the model, where necessary, to achieve convergence

9. 9 Replication of Poststratification/Calibration Base year schools poststratified to population totals Base year students not poststratified Students in follow-up rounds calibrated to previous round weight sums Replicate-level control totals - Computed weight sums for each replicate Deleted variables from the model, where necessary, to achieve convergence

10. 10 Comparison of Variance Estimates Variance estimates influenced by: ●Unequal weighting ●Stratification ●Clustering ●Nonresponse adjustment ●Poststratification

11. 11 Comparison of Variance Estimates (cont.) Poststratification to “known” population totals causes the sampling variance for estimates of the totals to go to zero Repeating the poststratification step on each half sample replicate ensures that the variance estimates for the control total estimates are zero Calibration to previous round half sample data causes the variance estimates for the control total estimates to not be zero

12. 12 Comparison of Variance Estimates (cont.) Compared standard errors computed using both the Taylor series and BRR variance estimation methods BRR standard errors more conservative BRR and Taylor series standard errors larger than simple random sample standard errors

13. 13 Base Year School Standard Errors

14. 14 Base Year School Design Effects

15. 15 Base Year Student Standard Errors

16. 16 Base Year Student Design Effects

17. 17 First Follow-Up Standard Errors

18. 18 First Follow-Up Design Effects

19. 19 Second Follow-Up Standard Errors

20. 20 Second Follow-Up Design Effects

21. 21 Conclusions BRR takes into account the variance due to weight adjustments, so these results are expected Controlling to replicate-level totals recognizes variance in base year totals due to sampling variability, so the results are more conservative Worthwhile to replicate all stages and all adjustments if time permits

22. 22 Questions? escheib@rti.org