1. Data Collection techniques Marina Signore
Measurement of the quality of statistics
3-5 October 2012
Marina Signore
Istat
Division "Metadata, Quality and R&D Projects", Chief

2. Data collection
data collection is any process whose purpose is to acquire or assist in the acquisition of data
the impact of data collection is both direct and critical as these data are the primary input. The quality of the operation has a very high impact on the quality of the final product, in particular on accuracy:
- nonresponse errors
- measurement errors (respondent errors, interviewer errors)
the mode of administration of the questionnaire (e.g. mail, telephone, in person) has an influence on data quality. Respondents may answer questions differently in the presence of an interviewer, over the phone, or by themselves (data collection mode effect)

3. Data collection
the choice of the data collection technique is a complex decision that depends on several aspects:
- survey purposes (e.g. sensitive questions, collection of retrospective information or daily data)
- observational units
- availability of frames
- questionnaire length
- the length of the data collection period
- the budget
- the expected response rate and expected measurement error
Overview of different data collection modes with their impact on data quality*:
*Lyberg L and Kasprzyk (1991)

4. Data collection techniques
with interviewer
- face to face
- telephone
without interviewer
- mail
- web
- diary

5. Face-to-face interview
an interviewer administers a structured questionnaire to respondents
PAPI: paper and pencil interview
CAPI: interviewers with laptop computers
+ allows for more complex interviews
+ respondent identity can be controlled
+ support to understand the questions (e.g. visual aids)
+ automatic checks: editing, completeness and correctness (CAPI)
+ high flexibility (open ended questions,…)
+ probing by experienced interviewers
- very expensive
- complex fieldwork organisation
- interviewer effect
- presence of other people during the interview
- difficult to answer to sensitive questions
- “social desirability” effect: respondents want to appear “good”

6. Telephone interview
an interviewer administers a structured questionnaire to respondents over the telephone rather than in person:
CATI (usually centralised)
possible also with paper questionnaires
+ faster data collection process
+ less expensive
+ possibility of monitoring the interviewers (immediate feedback)
+ reduced interviewer effect (due to anonymity)
+ automatic checks: editing, completeness and correctness
- less flexibility (no complicated questions, visual aids, open ended questions)
- difficult to control the respondent identity
- the interviewer still plays a central role
- coverage problems

7. Self-administrated mail surveys
the questionnaires are mailed to the observation units which complete and mail back the questionnaires.
Most commonly used mode for economic surveys.
+ less expensive
+ simple organisation
+ no interviewer effect
+ less risk of social desirability effects
- high nonresponse rates
- high number of cases in which it is not possible to distinguish between a frame error and a nonresponse (unresolved units)
- respondents errors (misreading and misinterpretations)
- no control of who is answering the questionnaire
*Couper M. (2001)

8. Web surveys the questionnaires is accessible on a web site
On-line compilation
Off-line compilation and uploading
Increasing use in recent years in economic surveys but recently also for households (e.g. Italian Population census).
+ less expensive
+ simple organisation
+ no interviewer effect
+ less errors due to ordering of questions and response categories
+ less risk of social desirability effects
- high nonresponse rates
- respondents errors (misreading and misinterpretations)
- no control of who is answering the questionnaire

9. Diary surveys
the respondents are asked to record the events right after they occur to avoid recall errors.
Interviewers are used to contact the respondent, deliver the diary, explain the procedures, collect the diary.
Topics that require detailed behaviour reporting over a period of time (expenditure, time-use,…).
+ more accurate than asking retrospective questions
+ initial contact with the interviewer
+ no interviewer effect
- active participation
- respondents give insufficient attention to recording events and fail to record them in their memories
- respondents may change behaviour as a result of using a diary
- complex structure of diary
- need for incentives

10. Summary Mail and web Telephon Person Contact Respondent environment
Presentation letter and questionnaire
Anoynimous interviewer
Interviewer
Respondent environment
no control of who is answering the questionnaire
A certain control of who is answering the questionnaire.
No big influence of the presence of other people
Possibility of controlling who is answering the questionnaire. The presence of other people might influence the interview

11. Summary Mail and web Telephon Person Interviewer and aids
No aids but also no interviewer effect
Small interviewer effect
Interviewer support and other visual aids but also large interviewer effect
Sensitive questions
Possible to include them
Possible to include them but special training on interviewers
Special techniques have to be used

12. Summary Mail and web Telephon Person Pressure to answer
Respondent has the time to answer and no pression is put on him/her
A moderate pressure can lead to vague or not precise answers
Pressure on providing adequate answers
Lenght
No more than pages
No more than half an hour
No more than one hour

13. Unit non-response rate by data collection mode
CAPI and CATI surveys are included in the Face to face group

14. Mixed modes
two or more modes are used to save money, improve coverage, improve response rates, reduce measurement errors.
- use of two or more frames to improve coverage
- use of different data collection modes for nonresponse follow-ups to improve response rates
- use a telephone interview to screen subpopulations of interest to be interviewed by mail or face-to-face afterwards.
- in panel surveys, a mixed mode strategy is used to reduce costs and take advantage of the best features of each mode

15. Istat surveys 148 Type of survey Status Administrative Directe Mixed*
Total
Active 47 93 9
149
No more active 25 29 63
Temporary inactive 3 7 1 11 75
129 19
223
148
*partly administrative and partly direct

16. Istat surveys distribution of direct and mixed surveys (148)
Characteristics
Number %
Periodicity
Annual 54
36.5
Infra- Annual 50
33.8
Pluriannual 44
29.7
Observation unit
Households 24
16.2
Institutions 55
46.6
Busnesses 69
37.2
EU Regulation
Yes 72
48.6 No 76
51.4

17. Istat surveys: data collection mode
single data collection mode – current surveys (n=87*)
Data collection mode
# surveys
CATI 17
Web 4
Mail 9
PAPI 3
CAPI 1
Totale 51
*excluded censuses and occasional surveys

18. Istat surveys: data collection mode
multi data collection modes – current surveys (n=87*)
Data collection techniques
# surveys
Mail and web 24
PAPI + Diary or self-administered quest. 6
CATI and CAWI 2
PAPI and CAPI 1
CAPI and CATI
Mixed electronic means
Totale 36
*excluded censuses and occasional surveys

19. Istat surveys: changes in data collection mode
from 1998 to 2012
23 busness surveys changed data collection mode
(2) CATI
(17) Mail and web
(4) web
Mail

20. Trend Over Time in Response Rates at Istat, by Data Collection Mode (Multi vs. Single-mode)
Signore et al .( 2008)

21. Riferimenti bibliografici
Biemer, P.P., and L.E. Lyberg (2003), Introduction to survey quality, Hoboken, NJ: Wiley
Couper M.P. (2001), “Web Surveys: A Review of the Problem”, Journal of the Market Research Society, 22(2), pp.77-95
De Leeuw, E.D. (2005), “To Mix or Not to Mix Data Collection Modes in Surveys”, Journal of Official Statistics, 21, 2, pp DillmanD. (1991), “The Design and Administration of Mail Surveys”, Annual Review of Sociology,17, pp
Groves R.M. (1989), Survey Errors and Survey Costs, J. Wiley, N.Y.
Lyberg L. and Kasprzyk D. (1991), “Data Collection Methods and Measurement Errors: An Overview” in Measurement errors in surveys, Biemer et alt. (eds.), J. Wiley, N.Y.
Signore M., Brancato G., Simeoni G. (2008), “Multi-mode data collection: What can still be expected?”, invited paper, Proceedings of Statistics Canada Symposium 2008, Data Collection: Challenges, Achievements and New Directions.