1. Multiple Indicator Cluster Surveys Survey Design Workshop
Fieldwork:
Survey Quality Control
MICS Survey Design Workshop

2. Objectives
Identify factors affecting the accuracy and reliability of survey data
How to prevent and correct errors
The essential role of supervision in the field and providing feedback to the team

3. Accuracy and reliability
The accuracy (or validity) of a measurement is concerned with the net difference between the mean of the measurements obtained and the true value (related to the size of the bias)
The reliability (or precision) of a measurement refers to the degree to which repeated measurements give consistent values (related to the size of the confidence interval)

4. Accuracy and reliability
Unreliable but accurate
imprecise but unbiased
Unreliable & inaccurate
imprecise & biased
Reliable but inaccurate
precise but biased
Reliable & accurate
precise & unbiased
 BIASED

5. Overview of presentation
Data collection
Field supervision

6. Data Collection Organisation of daily work,
Security of staff and equipment

7. Data Collection: Implementation of sample
Non-response: failure to obtain information for selected households, eligible women, or children
A potentially serious bias that can be minimized
Interviewers will need to make return visits to households (call-backs)

8. Data Collection: Implementation of sample
Types of non-response:
Interviewer is unable to do the selected household
Interviewer unable to meet eligible respondents
Respondent refuses to be interviewed

9. Data Collection: Implement. of sample
Types of non-response:
Interviewer is unable to do the selected household.  Why?
Structure not found
Occupied structure inaccessible
Structure non-residential, vacant, or demolished

10. Data Collection: Implement. of sample Types of non-response:
Interviewer unable to meet eligible respondents.  Why?
No one at home at time of call
Respondent temporarily absent
Will need call-backs

11. Call-backs
Interview that is not completed requires a “call-back” or follow-up visit
Three call-backs required (different times and days)
Supervisors and interviewers keep track using control sheets
Requires good tracking of work to ensure that all planned interviews are completed before leaving cluster

12. Data Collection: Implement. of sample Types of non-response:
Respondent refuses to be interviewed. Why?  What to do?
Approach respondent from her point of view
Postponed interview to another day
Have supervisor/field editor revisit the respondent
The number of refusals should be closely monitored. Need to find out reasons for frequent refusals.

13. Data Collection: Monitoring field work
Training is a continuous process
Observation and supervision throughout the fieldwork are a part of the training
Team supervisors and field editors play very important roles in continuing this training and in ensuring the quality of MICS data

14. Data Collection: Monitoring field work Fieldwork control sheets
Direct observation of interviews
Review of completed questionnaires (editing)
Spot-checking households and HH composition

15. Systematic Spot Checking
How to spot-check household composition?
Supervisors should complete columns 2-6 in the HH questionnaire and compare with that of the interviewer
Check about 5% of households (5-6 per week)
All team members must be spot-checked; provide feedback if necessary

16. Fieldwork Control Sheets
Interviewers’ work is monitored and evaluated by keeping accurate record of assignments
Both supervisors and interviewers have control forms to maintain
These forms should be returned to the director of field operations along with the completed questionnaires
Interviewer is responsible for ensuring that control sheet is up-to-date

17. Observing Interviews, part 1
To evaluate and improve interviewer performance
To look for errors and misconceptions that cannot be detected through editing
Why? Precise but inaccurate answers
Who observes? The supervisors or field editors
Who should be observed? Every interviewer
To check if interviewer is editing h/er own work before leaving HH

18. Observing Interviews, part 2
How often? 5-6 interviews per week, more at start of fieldwork
How? Just take notes without disrupting the interview
What to do after? Supervisors reviews questionnaires with interviewer, highlight issues and proposes solution and training

19. Editing Questionnaires in the Field
Fundamental to survey quality
Need to ensure:
Accurate and complete information in each questionnaire
Correct count of questionnaires

20. Editing Questionnaires in the Field
Done daily by field editor; supervisor can assist
Editing of all questionnaires must be completed BEFORE leaving cluster
Results and errors are discussed with interviewers
Interviewers should go back to HH for correction

21. Evaluating Interviewers Performance
Daily discuss the quality of interviewers work
Point out mistakes discovered during observation of interviews or noticed during editing
Discuss examples of actual mistakes, but be careful not to embarrass individual interviewers.
Re-read relevant sections from the Interviewer's Manual with the team to resolve problems.
Encourage the interviewers to talk about any situations they encountered in the field
Discuss whether situations are handled properly, and how to do it in the future

22. Overview of presentation
Data collection
Field supervision

23. Field supervision Who should go to Field supervision?
Supervision team, Senior staff from Implementing agency, stakeholders, UNICEF staff (except if strategic or political, avoid “supervision tourism”)
ToR for the supervision team (around Quality control) with report after each mission
Very important to put in place within the first week of fieldwork (if possible, start fieldwork in one central location)

24. Field Supervision Why Bring equipment, money, questionnaires …
Visit teams to observe interviews, review work
Re-visit selected clusters; spot-check households
Bring back completed and edited questionnaires to central location for data entry

25. Field Supervision: Field Check Tables (FCT)
FCT are an essential tool for field supervision
Based on already entered questionnaires brought back from fieldwork by supervision missions
Generated by data entry teams on a weekly basis
Provide a full range of information about the quality of the data already collected
Provide information on the work of each team and each interviewer
To be shared on a regular basis with RO and HQ
Survey Coordinators: Don’t go out without them

26. Field Check Tables
FC-2W: Eligible women per household
Mean number of eligible women per household, according to interviewer team
Team
Urban
Rural
Number of completed households
Number of eligible women in those HHs
Mean number of eligible women per HH
Target not met
Team 1 46 65
1.41 86 73
0.85
Team 2
172
243
214
225
1.05
Team 3
139
158
1.14 82
119
1.45
Team 4
197
236
1.20
120
112
0.93
Team 5
116
131
1.13
161
190
1.18
Total
670
833
1.24
663
719
1.08
Note: Number of women that are expected to be found per HH is country-specific and defined in the sample design (it usually differs by urban/rural areas).
The target is the minimum mean number of de facto eligible women per HH that we hope to find,
and should be > 80% of what was expected at the time of sample design.
Thus, if we expected to find 1.2 women per HH at the time of sample design, teams should be finding a minimum of 0.96 women per HH.
Survey managers should provide data processors with the country-specific targets.
Example: MICS sample was drawn based on the expectation of finding 1.46 women per HH in urban areas and 1.24 women per HH in rural areas. Targets for this table are for teams to find at least 1.17 women per HH (80% of 1.46) in urban areas and 0.99 women per HH (80% of 1.24) in rural areas.

27. Age ratio (women 15/ women 14)
Field Check Tables
FC-4W: Age displacement: women
Number of women age years listed in the household schedule by single year of age and age ratio 15/14, according to interviewer team
Team
Women's age ( years)
TOTAL
Age ratio (women 15/ women 14)
Target not met 12 13 14 15 16 17 18
Team 1 10 11 8 7 9 64
0.73
Team 2 68
0.78
Team 3 79
1.18 -
Team 4 5 6 67
0.38
Team 5 74
Total 59 62 56 46 44 40
352
0.83
Note: Target is an age ratio of women age 15 / women age 14 > 0.8

28. FCT : What else ?
USE THEM!