How survey design affects analysis Susan Purdon Head of Survey Methods Unit National Centre for Social Research

Two general principles Weighting affects both survey estimates and standard errors Complex sample design (clustering, stratification) affects standard errors but not survey estimates (as long as survey isn’t weighted).

Why weights are used Two main reasons: Because sample units are selected with non-equal probabilities of selection To reduce non-response bias

Weighting for non-equal probabilities Non-equal probabilities of selection can be used: For good statistical reasons: stratification with disproportionate allocation (e.g over-sampling by region) For convenience: because sampling frame is of larger units than survey units, and need to sub-sample within units (e.g selection of one person per household). Weights are calculated as the inverse of the probability of selection.

Example 1: WERS98

Effect of selecting one adult per household

Weighting for non-response Many surveys use weights to reduce non-response bias. Up-weight low responding groups; down-weight high responding groups.

Weighting for non-response

Effect of weights on estimates Weighting changes almost all survey estimates (means, percentages, odds ratios, correlation coefficients, regression coefficients etc.)

Effect of weights on estimates Unweighted estimate= (25%x %x102+15%x27+9%x8) /( )=20% Weighted estimate= (25%x38x1 + 17%x102x2+15%x27x3+9%x8x4) /(38x1+102x2+27x3+8x4)=18%

Effect of weighting on standard errors Standard errors for weighted and unweighted estimates are not the same. Weighting because of non-equal probabilities of selection tends to increase standard errors. Weighting for non-response sometimes increases/sometimes decreases ses. Impact tends to be smaller.

Impact of other design features - clustering Most face-to-face surveys are clustered Clustering doesn’t change estimates, but it does increase standard errors Degree of increase depends on cluster size and cluster homogeneity To account for clustering need to identify the primary sampling unit (psu) on dataset.

Impact of other design features - proportionate stratification Most surveys use proportionate stratification (either overall or within regions) Does not affect estimates. Tends to reduce standard errors. Degree depends on choice of stratifiers.

In summary: To get unbiased estimates need to use survey weights. To get correct standard errors need to take into account survey design, in particular weighting, clustering and stratification.