1. 9. Weighting and Weighted Standard Errors

2. 1 Prerequisites Recommended modules to complete before viewing this module  1. Introduction to the NLTS2 Training Modules  2. NLTS2 Study Overview  3. NLTS2 Study Design and Sampling  NLTS2 Data Sources, either 4. Parent and Youth Surveys or 5. School Surveys, Student Assessments, and Transcripts  6. Implications for Analysis and either 7. Parent and Youth Surveys or 8. School Surveys, Student Assessments, and Transcripts

3. 9. Weighting and Weighted Standard Errors 2 Overview  Purpose  Sampling weights overview  Creating NLTS2 sampling weights  Sampling weight example  Which sampling weight to use  Obtaining correct standard errors by correcting for design effects  Type 1 error  Analysis recommendations  Closing  Important information

4. 9. Weighting and Weighted Standard Errors 3 Purpose Learn how to obtain valid point estimates by using sampling weights. Learn how to obtain valid standard errors by adjusting for “design effects.”

5. 9. Weighting and Weighted Standard Errors Sampling weights overview Purpose of sampling weights  To produce point estimates that are representative the universe (i.e., national population of students with disabilities in age group; each disability category). 4

6. 9. Weighting and Weighted Standard Errors Sampling weights overview Why do we need to weight?  When a population includes some low-incidence groups of interest, those groups typically are oversampled. Oversampling ensures that the sample includes some members of those groups. But unless weighted, that sample does not represent the total population results.  Demographic groups may have differential response rates. 5

7. 9. Weighting and Weighted Standard Errors Sampling weights overview If all students in a disability group had been equally likely to be selected, then to make results represent the full population, weight each observation by the number of individuals in the population that it represents. Example using synthetic data for illustrative purposes: 6 2008 PopulationSampleWeight Students with learning disability 2,522,7355002,522,735/500 = 5,045.47 Students with visual impairment 25,79050025,790/500 = 51.58

8. 9. Weighting and Weighted Standard Errors Creating NLTS2 sampling weights Example  Local education agency (LEA) had 100 students with learning disability. Our sample had 10 students with learning disability, drawn with equal probabilities. We obtained responses from 5 students with learning disability. Students’ within-LEA weight would be 100/5 = 20.  The universe of LEAs in the cell served 400,000 students. Sampled LEAs in a cell, drawn with equal probabilities, served 2,000 students. Each student in the sampled LEAs represents 400,000/2,000 = 200 students.  Students’ total sampling weight would be 20 x 200 = 4,000. 7

9. 9. Weighting and Weighted Standard Errors 8 Creating NLTS2 sampling weights State schools were treated as a sampling cell. All were sampled; not all responded.  For each disability category, a weight was calculated by multiplying the number of students with that disability on the rosters of the responding schools by the inverse of the proportion of state schools that submitted rosters.

10. 9. Weighting and Weighted Standard Errors 9 Creating NLTS2 sampling weights Adjustments  So that weighted sample matches the number of students in each disability category, as reported to OSEP by the states for the 1999–2000 school year.  So that the weighted sample matches known characteristics of the population, such as age group and race/ethnicity.

11. 9. Weighting and Weighted Standard Errors 10 Things to note about NLTS2 sampling weights The youth is the unit of analysis.  Research questions must pertain to youth only.  With the weights provided by NLTS2, data cannot be used to represent the universe of teachers, classrooms, schools, districts, or states. There is clustering as a result of the sample design.  Clustering has implications for standard errors. This is covered a little later. Results using NLTS2 data must always be weighted for reporting or publishing. Weights differ for each wave and each instrument.

12. 9. Weighting and Weighted Standard Errors 11 Sampling weight example Disability Category Number in Sample Participated in Group Activities Weight for Category Weighted Value for Category Learning disability114.3 Speech/language impairment113.0 Mental retardation111.0 Emotional disturbance10.8.0 Hearing impairment11.1 Visual impairment11.1 Orthopedic impairment10.1.0 Other health impairment11.4 Autism10.1.0 Multiple disabilities10.1.0 TOTAL10610.08.9 Unweighted sample percentage = 60% (Column B total divided by Column A total) Weighted population estimate = 89% (Column D total divided by Column C total) Synthetic data for illustrative purposes.

13. 9. Weighting and Weighted Standard Errors Sampling weights example Differences in weights across waves and data collection instruments 12 Synthetic data for illustrative purposes.

14. 9. Weighting and Weighted Standard Errors 13 Which sampling weight to use? “Help! I have more than one weight to choose from; which one do I use?”  When combining data from multiple sources (instruments or waves) in analyses that require data from all of those sources, a general rule is to use the weights from the source for which the sum of the weights of the individuals is largest.  When combining data from sources with a lot of nonoverlapping data, proceed with caution and consult a statistician.

15. 9. Weighting and Weighted Standard Errors 14 Correcting standard errors for design effects  Reason: In a multistage sample, multiple observations are drawn a given location. Observations may be correlated.  Sample weights will produce correct point estimates but may produce incorrect standard errors.  Statistical adjustments for clustering “design effects” produce correct standard errors.

16. 9. Weighting and Weighted Standard Errors 15 Correcting standard errors for design effects Use  Taylor linearization  Replicate weights  For NLTS2, SRI’s approximation algorithm.

17. 9. Weighting and Weighted Standard Errors 16 Correcting standard errors for design effects: Taylor linearization approach With Taylor linearization, you specify the strata and the cluster variables.  Stratum variable has 64 values (geographic region, size, and wealth)  Cluster (first-level PSU) variable: LEA A problem arises: when there is only one observation per cluster.  Solution: Collapse these clusters within their stratum. Generally, we recommend using replicate weights.

18. 9. Weighting and Weighted Standard Errors 17 Correcting standard errors for design effects: Replicate weights For any data collection activity, NLTS2 provides 32 replicate weights. Each replicate is calculated as follows:  Half the LEA sample is used.  Respondents in selected LEAs are weighted up to the universe.  Differences across replicates reflect true variability in the full sample.

19. 9. Weighting and Weighted Standard Errors 18 Correcting standard errors for design effects: SRI’s simple approximation SE = Standard error M = Mean of the sampling weights V = Variance of the sampling weights Adjusted SE = SE *

20. 9. Weighting and Weighted Standard Errors 19 Correcting standard errors for design effects: What does an adjusted standard error buy you?  Accurate confidence intervals around estimates  Greater confidence that differences observed are truly differences in the population

21. 9. Weighting and Weighted Standard Errors 20 Weighted standard errors: Example means and standard errors TypeMeanStandard Error No weight82.20.32 Sampling weight (erroneously using a frequency weight) 81.90.02 Sampling weight and adj. for design effects 81.91.20 Randomly selected subset of the NLTS2 data used in all examples

22. 9. Weighting and Weighted Standard Errors 21 Weighted standard errors: Example estimates and p values for gender difference TypeEstimatep Value No weight0.43.56 Sampling weight (erroneously using frequency weight) 1.2<.001 Sampling weight and adj. for design effects 1.2.40 Randomly selected subset of the NLTS2 data used in all examples

23. 9. Weighting and Weighted Standard Errors Examples of statistical packages that calculate weighted standard errors SAS STATA WESVAR Taylor linearization or replicate weights SUDAAN SPSS – Taylor linearization only 22

24. 9. Weighting and Weighted Standard Errors 23 Weighted standard errors Examples of statistical packages that calculate weighted standard errors  NLTS2 training modules use SAS and SPSS for analysis examples.  SPSS – General statistical package Requires SPSS Complex Samples Module for calculating accurate standard errors GUI interface; Taylor linearization  SAS – General statistical package Requires the SAS Statistics Module for calculating accurate standard errors in PROC SURVEY procedures Taylor linearization or replicate weights

25. 9. Weighting and Weighted Standard Errors 24 Type 1 error Applies to NLTS2 as to any other study Temptation to perform many tests drastically increases the probability of spurious results Use caution

26. 9. Weighting and Weighted Standard Errors Analysis recommendations Two analysis recommendations Start simple! Before you start more complicated analyses,  Understand who’s in and who’s out of the analyses.  Understand the distributions.  Understand the bivariate relationships. 25

27. 9. Weighting and Weighted Standard Errors 26 Closing Topics discussed in this module  Sampling weights overview  Creating NLTS2 sampling weights  Sampling weight example  Which sampling weight to use  Obtaining correct standard errors by correcting for design effects  Type 1 error  Analysis recommendations

28. 9. Weighting and Weighted Standard Errors 27 Closing Next module:  10. NLTS2 Documentation Overview

29. 9. Weighting and Weighted Standard Errors 28 Important information  NLTS2 website contains reports, data tables, and other project-related information http://nlts2.org/http://nlts2.org/  Information about obtaining the NLTS2 database and documentation can be found on the NCES website http://nces.ed.gov/statprog/rudman/http://nces.ed.gov/statprog/rudman/  General information about restricted data licenses can be found on the NCES website http://nces.ed.gov/statprog/instruct.asphttp://nces.ed.gov/statprog/instruct.asp  E-mail address: nlts2@sri.com