1. RTI International is a trade name of Research Triangle Institute Sampling U.S. Schools – Frame, Sampling & Field Issues International Conference on Establishment Surveys June 21, 2007 Cynthia Augustine, Karol Krotki, Deborah Herget

2. 2 Overview  Frame Issues  Coverage, completeness  Accuracy  Sampling Topics  Stratification  Multistage Sampling  Field Processes  Response Rates  Strategies for Increasing Response Rates  Conclusions

3. 3 2006 PISA and PIRLS  National Center for Education Statistics (NCES)  PISA (Program for International Student Assessment)  15 Year Old Students  Multiple Subject Areas  PIRLS (Progress in International Reading Literacy Study)  4 th Grade Students  Literacy

4. 4 Frame Issues  US Department of Education’s of Common Core of Data (CCD) and Private School Survey (PSS) for 2003-2004  Supplementary Information  School sizes  Grade sizes  Federal school lunch program (proxy for Poverty). Private Schools assumed low poverty

5. 5 Frame Issues  Accuracy  In 2 field tests, all sampled schools were located  Good Coverage 2000-2005 Census: PISA Frame 4,158,3624,156,069 Number of 15 Year Olds Combined Frame 2006 QED Students57,879,42650,910,308 Schools140,998115,512

6. 6 Frame Issues  Out of date frames are a challenge in establishment surveys  Also a problem with PISA / PIRLS: 2003-2004 datasets for Winter 2006 data collection  Methods for updating frames  Request updated school lists from Districts  Updating on the fly – Half Open Interval (HOI) methods  Check Actual v. Frame to identify new schools and potentially re-sample  Sample Districts rather than schools

7. 7 Sample Design  PISA used implicit stratification by sorting the file by Public/Private, Region, Locale, High Minority Percent, and Grade Span  N=236 Original Schools  PIRLS used 10 Self-Representing Metropolitan Statistical Areas (MSAs) and 44 Primary Sampling Units (PSUs).  N=252 Original Schools

8. 8 Sampling Issues  PSU Creation  Manually created within states following Metropolitan Statistical Areas (MSAs) and counties  Total Number of students in PSU  Multistage Cluster Sampling (PIRLS Only)  44 PSUs selected with PPS (Students in PSU)  4 Schools selected PPS (Students in School)  Classrooms selected within schools using KeyQuest, an international sampling software

9. 9 Sampling Issues  Measures of Size  PISA: Estimated number of 15 year olds  PIRLS: Number of 4 th grade students  Oversampling  High Poverty (50% Eligible for Federal Lunch Program)  Private Schools

10. 10 Sampling Issues  Probability Proportional to Size sampling will make school-level weights inaccurate without multiplying by the Measure of Size.  Cluster sampling reduces travel costs and contacts with multiple school districts

11. 11 Field Processes  Low & Falling Response/Cooperation Rates among Schools  Preliminary Response Rate: PISA: 77% PIRLS: 84%  Strategies for Increasing Cooperation  Financial Incentives for students  Out-of-School Administration  PISA: “The Global Context of Scientific Literacy” Conference  PIRLS: Winter Olympics

12. 12 Conclusions  CCD and PSS provide good coverage, accuracy and supplementary information and are free.  Cluster sampling reduces travel costs and number of school districts contacted.  Incentives are important for increasing cooperation rates.

13. 13 Questions or Comments Cynthia Augustine RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709 919.541.6154 caugustine@rti.org

14. 14 References U.S. Census Bureau Hamann, Thomas A. “Evaluating the coverage of the U.S. National Center for Eduation Statistics’ Public / Elementary School Frame.” American Statistical Association 1999. Quality Education Data, Inc. http://www.qeddata.com/Catalogcounts/DG_FINAL_2006.pdf http://www.qeddata.com/Catalogcounts/DG_FINAL_2006.pdf