1 Quality Control for Field Operations

2 Overview Goal To ensure the quality of survey field work Purpose To detect and deter interviewer errors and falsification

3 Overview (cont.) What do we check in QC for field work? Interviewer procedures Forms Interviewer productivity Interviews Address listing

4 Overview (cont.) What are the tools for performing QC on field work? Interviewer observations Formal / Informal reviews Monitoring reports Reinterview Address listing QC

5 Interviewer Observations What are the goals of interviewer observations? To assess effectiveness of interviewer training To ensure interviewers understand and follow prescribed interviewing procedures To provide interviewers with feedback to encourage continuous improvement

6 Interviewer Observations (cont.) What are interviewer observations? Supervisor “shadows” interviewer during initial assignments Observe interviewer’s work to ensure appropriate procedures are followed Evaluate interviewer’s work and provide positive and negative feedback to interviewer See Handout on Field Control Form Information

7 Interviewer Observations (cont.) What are the results of observations? Supervisor decides if interviewer’s performance is acceptable If so, allow interviewer to continue work independently If not, re-train interviewer on procedures or release interviewer from the operation

8 Formal / Informal Reviews What are the goals of reviews? To ensure the quality of the data collected on forms and questionnaires To ensure interviewers and clerks understand and follow procedures for completing forms and questionnaires

9 Formal / Informal Reviews (cont.) How are informal reviews performed? Supervisor meets with employees to review forms and/or questionnaires Review for legibility, completeness, and accuracy Provide positive and negative feedback to employee for continuous improvement Repair errors where necessary

10 Formal / Informal Reviews (cont.) How are formal reviews performed? Usually performed in a field office by office staff Review questionnaires or listings for legibility, completeness, and accuracy Repair errors or return materials to the field when necessary

11 Formal / Informal Reviews (cont.) What work products are subject to reviews? Any form or questionnaire that requires a quality check (e.g., payroll forms, questionnaires, address listing forms, etc.)

12 Formal / Informal Reviews (cont.) What are the results of informal reviews? Supervisor determines if the form or questionnaire meets quality standards If so, the form/questionnaire is accepted and submitted If not, repair the form/questionnaire prior to submission

13 Monitoring Reports What are the goals of monitoring reports? To track status of operations To inform management of progress To identify problems early for timely rectification

14 Monitoring Reports (cont.) How do we know what reports to monitor for QC purposes? Define quality goals for the operation (e.g., response rates, productivity rates, cost, etc.) Identify variables that provide information on quality goals Design report(s) that present the important variables for quick review and assessment

15 Monitoring Reports (cont.) How do we monitor the reports for QC purposes? Determine necessary frequency for monitoring (usually daily) Look for outlier values (e.g., low response rates, high costs, etc.) Determine reason(s) for outlier values Take appropriate action

16 Reinterview What is reinterview? Quality check on interviewing activities Primary tool for ensuring quality of interviewers’ field work Independent, objective review of interviewing work

17 Reinterview (cont.) What types of reinterview can be used? Random reinterview Administrative reinterview Supplemental reinterview

18 Reinterview (cont.) What decisions are made in reinterview? Falsification (release interviewer) Interviewer error (retraining/feedback) Respondent error (no action) Rework may be necessary for falsification or other serious errors

19 Random Reinterview What is the purpose of the random reinterview component? Provides broad protection over entire workload and all interviewers Serves as an effective deterrent for interviewer falsification

20 Random Reinterview (cont.) What is the sampling design for the random reinterview component? Select random sample of cases within every enumerator’s workload, or Two-stage design: take a sample of interviewers, then a sample of cases within those interviewers’ workloads

21 Administrative Reinterview What is the purpose of the administrative reinterview component? Provides “targeted” approach to identifying suspect work Provides valuable performance data to field supervisors

22 Administrative Reinterview (cont.) How does the administrative model work? Accumulates data on work characteristics for all interviewers Vacant housing units Partial interviews Single-person households Deleted housing units Average population per housing unit

23 Administrative Reinterview (cont.) How does the administrative model work? (cont.) Compare accumulated data for each interviewer to the data for all other interviewers in their assignment area Identify and flag interviewers who are significantly different than their area average

24 Administrative Reinterview (cont.) How does the administrative model work? (cont.) Provide field supervisors with a list of “outlier” interviewers and what variables caused the flag Field supervisors determine if the interviewers’ “outlier” status is explicable More of suspect interviewer’s cases put into reinterview

25 Supplemental Reinterview What is the purpose of the supplemental reinterview? Provides field supervisors with the capability of checking work for selected interviewers Tool for use during investigation of suspected falsification

26 Supplemental Reinterview (cont.) What is the sampling design for the supplemental reinterview? Select a random sample of cases from the interviewer’s workload Field supervisor can specify how many cases to check

27 Address Listing QC What is the goal of address listing QC? To ensure the quality of address listing work To protect against undercoverage and overcoverage

28 Address Listing QC (cont.) What are the tools for performing address listing QC? Dependent QC Advance listing Suppression

29 Address Listing QC (cont.) What decisions are made in address listing QC? If errors exceed acceptable limit, recanvass assignment area Repair minor mistakes Listers may be released or retrained depending on severity of errors

30 Dependent QC What is dependent QC? Random, independent check of addresses listed by survey listers Field supervisors verify existence of housing units or validity of housing unit deletes

31 Dependent QC (cont.) What is the sampling design for dependent QC? Select a random sample of housing units to verify from the listers’ workload Sample should include all types of listings (e.g., verifications, adds, deletes)

32 Advance Listing What is advance listing? Field supervisor does initial listing of assignment areas prior to making assignments to listers Gives supervisor knowledge of the area so they are able to detect faulty or inadequate work

33 Suppression What is suppression? Selected addresses are purposely left off listings Listers’ ability to find and add the suppressed addresses is assessed Provides confidence in the listers’ ability to perform adequate canvassing