Practical Survey Design Strategies for Minimizing MSE Lars Lyberg and Bo Sundgren Statistics Sweden

The Survey Process Research Objectives Sampling Design Data Collection Analysis/Interpretation Concepts Population Mode of Administration Questions Questionnaire revise Data Processing

New goals and organization at Statistics Sweden Standardized processes, methods and tools Standardized processes, methods and tools Centralized methodological and IT staff Centralized methodological and IT staff User orientation User orientation World class vision World class vision Subject-matter departments responsible for survey design Subject-matter departments responsible for survey design

The typical survey at Stats Sweden Used to have extensive freedom regarding systems, methods and tools Used to have extensive freedom regarding systems, methods and tools Suboptimization due to methodologists and culture Suboptimization due to methodologists and culture Now a very clear responsibility is given to the survey managers Now a very clear responsibility is given to the survey managers –Use standard methods and tools –Decide a proper resource allocation

The idea: A survey design handbook for survey managers Simple language Simple language Educational Educational Rules of the road Rules of the road

Contents The iterative process The iterative process Talking to the client Talking to the client Methods available Methods available General assessment of survey situation General assessment of survey situation The planning criterion The planning criterion An ideal approach An ideal approach Trade-off situations Trade-off situations The final compromise design The final compromise design Information needs Information needs –Pilots and pretests –QC based on paradata –Evaluation –Error and cost structures Responsive designs Responsive designs Documentation Documentation Creating a design team Creating a design team

The typical survey textbook Lack of a real planning theory Lack of a real planning theory Emphasis on sampling design Emphasis on sampling design Handling of specific sources of error Handling of specific sources of error –Reducing, weighting, estimating via survey models and latent class analysis Design checklists and process design one by one Design checklists and process design one by one Handbooks by Eurostat, ABS, SNZ, UN FAO Handbooks by Eurostat, ABS, SNZ, UN FAO

Talking to the client Research problem and analytical needs Research problem and analytical needs Budget Budget Information available Information available Various briefings of design developments Various briefings of design developments What to do when the client situation is fuzzy What to do when the client situation is fuzzy

General assessment of the survey situation Characteristics of alternative methods Characteristics of alternative methods Similar surveys done before? Similar surveys done before? Is info available elsewhere? Is info available elsewhere? Information needs Information needs Constraints Constraints Error and cost structures Error and cost structures Special features such as population distribution, sensitivity, lack of options, new research field, etc. Special features such as population distribution, sensitivity, lack of options, new research field, etc. Risk assessment Risk assessment To do or not to do To do or not to do

The planning criterion Smallest possible MSE for a given budget Smallest possible MSE for a given budget Smallest possible MSE for a given budget and further constraints such as timeliness, comparability, accessibility, coherence, response burden, etc. Smallest possible MSE for a given budget and further constraints such as timeliness, comparability, accessibility, coherence, response burden, etc.

Trade-off situations Ideal seldom realistic Ideal seldom realistic One quality dimension vs another One quality dimension vs another Cost vs errors Cost vs errors Bias vs variance Bias vs variance Quality vs cost Quality vs cost Error in one process step vs another Error in one process step vs another Mode switch vs new error structures Mode switch vs new error structures New technology vs new error structures New technology vs new error structures One method vs another One method vs another The issue: How are multiple trade-offs handled? The issue: How are multiple trade-offs handled?

The final design the result of: Continuing discussions between team and client Continuing discussions between team and client Adoption of certain rules of the road Adoption of certain rules of the road Appreciation of the fact that optimum is often flat Appreciation of the fact that optimum is often flat Prioritizing between multiple purposes Prioritizing between multiple purposes No major steps or error sources ignored No major steps or error sources ignored Proper resource allocation Proper resource allocation

General thinking Think upstream Think upstream Use expertise and prior experience Use expertise and prior experience Analyse reduction of error as a function of cost Analyse reduction of error as a function of cost Those correlated variances Those correlated variances Develop a plan Develop a plan

Rules of the road Use known reliable methods Use known reliable methods Use information on errors and costs Use information on errors and costs Allocate resources and assign responsibilities Allocate resources and assign responsibilities Collect information about quality as survey progresses Collect information about quality as survey progresses Collect paradata for QC and responsive efforts Collect paradata for QC and responsive efforts Document and disseminate info on quality to users and producers Document and disseminate info on quality to users and producers Use experts and best practices Use experts and best practices Use guidelines and standards Use guidelines and standards

References Dalenius 1971 Dalenius 1971 Fellegi and Sunter 1974 Fellegi and Sunter 1974 Groves 1989, 2006 Groves 1989, 2006 Linacre and Trewin 1993 Linacre and Trewin 1993 Holt and Jones 1998 Holt and Jones 1998 Weeks Weeks Biemer and Lyberg 2003 Biemer and Lyberg 2003 Campanelli 2006 Campanelli 2006 Heeringa and Groves 2007 Heeringa and Groves 2007 O’Muircheartaigh 2008 O’Muircheartaigh 2008 Couper 1997, 1998 Couper 1997, 1998