QMSS, Lugano, Lynn Control of Sampling Error Peter Lynn Institute for Social and Economic Research, University of Essex, UK

QMSS, Lugano, Lynn Presentation Structure Objectives of Survey Design Survey Error Framework Coverage Error: Control Sampling Error: Control Design Effects: a key tool Examples from European Social Survey

QMSS, Lugano, Lynn Survey Design: Objectives Appropriate accuracy (cf. budget): For what estimates? What is appropriate? Estimates: Typically, descriptives for sub- domains and total, and comparisons between sub- domains (inc. models) Appropriate: high enough, cost effective

QMSS, Lugano, Lynn Survey Errors ProcessError source Population ↓Coverage Sampling Frame ↓Sampling Sample ↓Non-response Responding sample ↓Measurement Data

QMSS, Lugano, Lynn Coverage Error Ideal aim: Complete coverage Implies zero coverage error Practical aim: Very high coverage Similar coverage for each sub- domain Hopefully similar (and small) coverage bias

QMSS, Lugano, Lynn Example: ESS Target population in each nation (domain): all persons 15 years or older resident in private households within the borders of the nation, regardless of nationality, citizenship, language or legal status. Under-coverage in practice: If language is barrier to interviewing; If some addresses/households excluded (e.g. If electoral registers used as frame of addresses); If illegal residents excluded (e.g. If population register used as frame).

QMSS, Lugano, Lynn Sampling Error Aim: Maximum precision of estimates of between-domain differences Implication: same precision for each domain (for a given estimate)

QMSS, Lugano, Lynn Sampling Error Affected by: Sample size (n) Population variance (S 2 ) Sample clustering Sample stratification Variable sampling fractions For SRS: Var(y)=(S 2 /n)(1-(n/N)) Effect of other 3 design features can be summarised by design effect

QMSS, Lugano, Lynn Design Effect: A useful tool where ;

QMSS, Lugano, Lynn Example: Use of Design Effects on ESS and where and So, challenge was to predict and for each nation

QMSS, Lugano, Lynn Sample Design Process ESS sampling panel set up Each nation allocated to a panel member for bilateral liaison Panel met 3 times and communicated regularly, to ensure consistency of approach Each design had to be approved by whole panel

QMSS, Lugano, Lynn Predicting Unclustered designs (5) trivial: For other designs, necessary to decide upon n and number of clusters and to predict eligibility rate and response rate Under-estimated if RR under-estimated. E.g. Greece Over-estimated if RR over-estimated. E.g. Italy, Spain, Czech Rep

QMSS, Lugano, Lynn Predicting Many countries assumed default value of 0.02 A few countries assumed values between 0.03 and 0.05, either based on estimates from earlier surveys or because clustering units were particularly small Post-fieldwork estimates showed large range across variables

QMSS, Lugano, Lynn Predicting Planned variation in sampling fractions over strata (only NR is uncertain); Additionally, 3 forms of uncertainty encountered: Re. Distribution of # persons aged 15+ per household Re. Relationship between a proxy size measure used at PSU level and actual size measure of relevance Re. Relationship between a proxy size measure at hhd/address level and actual number of persons aged 15+

QMSS, Lugano, Lynn Predictions of Zero variation (SE, DK, FI, HU, SI, BE) Known SFs only (DE, NO, PL) From recent surveys (CH, NL, UK) PSU size measures & SFs (IL) Hhold size dist. (AT, CZ, ES, GR) Hhold size measure & SF (LU)

QMSS, Lugano, Lynn Overall Deff Predictions Good for many countries Some under-estimates compensated by larger m (GR, SI) Some modest under-estimates, e.g. CZ AT Two severe under-estimates: IL NO Poor RR prediction also affected m, e.g. IT, CZ

QMSS, Lugano, Lynn Quality Improvement Round 1 procedures improved quality in several countries and cross-nationally Round 1 estimates will influence round 2 predictions (etc.) Guidelines to be amended in light of round 1 experience (e.g. m, dual designs, default roh)

QMSS, Lugano, Lynn Final Comments None of these ideas are unique to cross- national surveys: apply to any comparative survey (= all surveys) We are forced explicitly to consider domain precision aims when fieldwork is organised separately for each domain But we should always do this: sample design should be appropriate for analysis aims

QMSS, Lugano, Lynn Control of Sampling Error Peter Lynn Institute for Social and Economic Research, University of Essex, UK