1. Survey phases, survey errors and quality control system
Measurement of the quality of statistics
3-5 October 2012
Marina Signore
Istat
Division "Metadata, Quality and R&D Projects", Chief

2. Topics
Survey phases and survey errors
Quality control system

3. Survey phases
Set of homogenous operations from an organisational and chronological point of view
Each phase is a potential source of errors
Error sources and methods to prevent or reduce errors impact for each phase of the statistical production process

4. GSBPM: Generic Statistical Business Process Model

5. GSBPM main features
Developed within the Joint UNECE/Eurostat/OECD Work Session on Statistical Metadata (METIS)
Born as a documentation model, with the aim of standardise terminology
Organised in four levels with increasing level of detail
Over-arching processes
Flexibility
Final version April 2009

6. Over-arching processes
The Generic Statistical Business Process Model
Level 0
Over-arching processes
Level 1
Level 2
5.2. Classify and code - This sub-process classifies and codes the input data. For example automatic (or clerical) coding routines may assign numeric codes to text responses according to a pre-determined classification scheme.
Level 3

7. Questionnaire design and testing
planning
Sample selection
Data processing
Data collection
Documentation
Quality control system

8. Quality control system
Survey phases
Planning
Questionnaire design and testing
Sample selection
Data collection
Data processing
Documentation
Quality control system

9. Planning Determine needs for information Design outputs
Design variables descriptions
Design data collection methodology
Design frame and sample methodology
Design production system & workflow
Design the quality control system
Take into account constraints in terms of available resources, costs, timeliness, national or international regulations.

10. Planning Determine needs for information Design outputs
Impact on relevance, comparability and coherence
Planning
Determine needs for information
Design outputs
Design variables descriptions
Design data collection methodology
Design frame and sample methodology
Design production system & workflow
Impact on accuracy but also on timeliness

11. Survey phases Questionnaire design and testing Sample selection
Data collection
Data processing
Impact on accuracy but also on timeliness

12. Questionnaire design and testing
The questionnaire is a communication tool highly influential for data quality
question wording
response categories and their order
the way and context in which questions are presented
Source of measurement errors and item non response
In some cases also of unit non response

13. Sample selection
The selection from a frame of the population units given the sample design and size
the population to be investigated should have been specified and the frame from which the units are sampled and contacted should have been chosen
Source of coverage or frame errors

14. Data collection
data collection is any process whose purpose is to acquire or assist in the acquisition of data*
the impact of data collection is both direct and critical as these data are the primary input. The quality of the operation has a very high impact on the quality of the final product, in particular on accuracy
the mode of administration of the questionnaire (e.g. mail, telephone, in person) has an influence on data quality. Respondents may answer questions differently in the presence of an interviewer, over the phone, or by themselves
Source of unit non response and measurement errors (respondent and interviewer errors)
But also data collection mode effect
*Statistics Canada (2003)

15. Data processing - data entry - editing and imputation - coding
Coding: operation of converting an actual response for a survey question into a category (open-ended responses e.g. “other specify” or “specify occupation”
Data entry: process of transferring collected data to an electronic medium
Editing and Imputation: data editing is the application of checks to detect missing, invalid or inconsistent entries or to point to data records that are potentially in error and the imputation is the process used to determine and assign replacement values for missing, invalid, inconsistent data that have failed edits

16. Data processing Source of processing errors
the errors range from simple recording errors to mispecification of the editing and imputation model.
different impact for different data collection mode.
e.g. different coders are likely to interpret and code the same response differently. Even the same coder may change over time (get experienced or bored). They affect the accuracy of final estimates
Source of processing errors

17. Survey phases and survey errors
Questionnaire design
Measurement errors
Item non response
Sample selection
Coverage or frame errors
Data collection
Unit non response
Data processing
Processing errors

18. Quality Control System
Set of tools and methods to improve data quality
Set of quality control actions for improving the quality of each survey phase and operations
Quality control activity should be part of the survey process itself

19. Quality Control Actions
Preventive actions: actions put in place in order to avoid potential errors usually before the operation is carried out
Monitoring actions: actions put in place in order to reduce errors during the execution of an operation
Ex-post actions: actions put in place in order to estimate the impact on the final data of non sampling errors

20. Some examples Quality control system Training of interviewers
Advance letter to respondents
Interviewers response rate
Questionnaire testing
Follow-up on non respondent units
Edit and imputation procedures
Sampling variance estimation

21. Quality Control System
Planning quality
Prevent
Monitor
Evaluate
Run the checks
Process Indicators
Direct Measures
Analyse results

22. Quality Measures Direct measures Quality indicators
+ estimate of error impact
- very expensive
- no overall estimate
Quality indicators
- indirect estimate of error impact
+ by-product of the production process
+ alarm bell

23. Documentation
It is important to document the survey operations and procedures, the quality controls done and their results
It is a demanding and time-consuming task and usually has not a high priority for survey managers

24. Documentation It should be considered as an investment:
Save time in future survey occasions
Easier to train new staff
Share good practices and know-how
Transparency

25. References
Biemer P. and Lyberg L. (2003), Introduction to Survey Quality, J. Wiley, N.Y.
Groves R.M. (1989), Survey Errors and Survey Costs, J. Wiley, N.Y.
Lyberg L. and Kasprzyk D. (1991), “Data Collection Methods and Measurement Errors: An Overview” in Measurement errors in surveys, Biemer et alt. (eds.), J. Wiley, N.Y.
Lyberg L. and Kasprzyk D. (1997), “Some Aspects of Post-Survey Processing” in Survey Measurement and Process Quality, Lyberg et alt. (eds.), J. Wiley, N.Y.
ONS (2004), Guidelines for Measuring Statistical Quality, version 1.0,
Statistics Canada (2003), Quality Guidelines, fourth edition 25 25