RTI International is a trade name of Research Triangle Institute Sampling U.S. Schools – Frame, Sampling & Field Issues International Conference on Establishment Surveys June 21, 2007 Cynthia Augustine, Karol Krotki, Deborah Herget

2 Overview  Frame Issues  Coverage, completeness  Accuracy  Sampling Topics  Stratification  Multistage Sampling  Field Processes  Response Rates  Strategies for Increasing Response Rates  Conclusions

PISA and PIRLS  National Center for Education Statistics (NCES)  PISA (Program for International Student Assessment)  15 Year Old Students  Multiple Subject Areas  PIRLS (Progress in International Reading Literacy Study)  4 th Grade Students  Literacy

4 Frame Issues  US Department of Education’s of Common Core of Data (CCD) and Private School Survey (PSS) for  Supplementary Information  School sizes  Grade sizes  Federal school lunch program (proxy for Poverty). Private Schools assumed low poverty

5 Frame Issues  Accuracy  In 2 field tests, all sampled schools were located  Good Coverage Census: PISA Frame 4,158,3624,156,069 Number of 15 Year Olds Combined Frame 2006 QED Students57,879,42650,910,308 Schools140,998115,512

6 Frame Issues  Out of date frames are a challenge in establishment surveys  Also a problem with PISA / PIRLS: datasets for Winter 2006 data collection  Methods for updating frames  Request updated school lists from Districts  Updating on the fly – Half Open Interval (HOI) methods  Check Actual v. Frame to identify new schools and potentially re-sample  Sample Districts rather than schools

7 Sample Design  PISA used implicit stratification by sorting the file by Public/Private, Region, Locale, High Minority Percent, and Grade Span  N=236 Original Schools  PIRLS used 10 Self-Representing Metropolitan Statistical Areas (MSAs) and 44 Primary Sampling Units (PSUs).  N=252 Original Schools

8 Sampling Issues  PSU Creation  Manually created within states following Metropolitan Statistical Areas (MSAs) and counties  Total Number of students in PSU  Multistage Cluster Sampling (PIRLS Only)  44 PSUs selected with PPS (Students in PSU)  4 Schools selected PPS (Students in School)  Classrooms selected within schools using KeyQuest, an international sampling software

9 Sampling Issues  Measures of Size  PISA: Estimated number of 15 year olds  PIRLS: Number of 4 th grade students  Oversampling  High Poverty (50% Eligible for Federal Lunch Program)  Private Schools

10 Sampling Issues  Probability Proportional to Size sampling will make school-level weights inaccurate without multiplying by the Measure of Size.  Cluster sampling reduces travel costs and contacts with multiple school districts

11 Field Processes  Low & Falling Response/Cooperation Rates among Schools  Preliminary Response Rate: PISA: 77% PIRLS: 84%  Strategies for Increasing Cooperation  Financial Incentives for students  Out-of-School Administration  PISA: “The Global Context of Scientific Literacy” Conference  PIRLS: Winter Olympics

12 Conclusions  CCD and PSS provide good coverage, accuracy and supplementary information and are free.  Cluster sampling reduces travel costs and number of school districts contacted.  Incentives are important for increasing cooperation rates.

13 Questions or Comments Cynthia Augustine RTI International 3040 Cornwallis Road Research Triangle Park, NC

14 References U.S. Census Bureau Hamann, Thomas A. “Evaluating the coverage of the U.S. National Center for Eduation Statistics’ Public / Elementary School Frame.” American Statistical Association Quality Education Data, Inc.