1. RTI International RTI International is a trade name of Research Triangle Institute. www.rti.org Computer Audio Recording: A Practical Technology for Managing Survey Quality M. Rita Thissen, Hyunjoo Park, Mai Nguyen International Total Survey Error Workshop Québec City, QC, June 2011

2. RTI International Overview  Survey process steps  Technology and methodology of CARI*  Incorporating CARI into the survey process  Addressing structural and functional errors  CARI adoption by the US Census Bureau – Overall implementation plan – American Community Survey Content Test use of CARI for behavior coding  User feedback and future possibilities *CARI: Computer Audio-Recorded Interviewing

3. RTI International Survey Process Steps 1. Define research objectives 2. *Choose methods of data collection 3. *Detailed planning, including quality assurance plan 4. *Construct and pretest questionnaire 5. *Collect data 6. *Analyze and report * All except the first step may be affected by use of audio recording methods

4. RTI International CARI Technology  Collect audio recordings during actual interview  Capture screen image and response data for comparison with recorded audio  Monitor recordings centrally after the interviewing session ends  Quantify content through coding  Review coding results and choose what action to take, if any

5. RTI International CARI System Concepts  Support for total error management  Multimode  Multi-purpose  More efficient and less burdensome than real-time monitoring

6. RTI International Survey Step 2: Choose Data Collection Methods ISSUE  Concern about mode effects between telephone (CATI) and in-person (CAPI) interviews  Determine what data points might be sensitive to mode effects  Difficulty of data capture in semi-structured interviews or free-response items  Standardized coding, such as occupation and industry, or drug and disease codes APPROACH  Audio recordings for comparison across “electronic” modes  Plan to record those questions and responses, plus “buffering” by recording adjacent items  Capture open-ended responses through audio recording for later coding  Allow standardized coding by centrally-located experts instead of by interviewers

7. RTI International Survey Step 3: Quality Assurance (QA) Plan ISSUE  QA plan is for detection, control and remediation of error  Data entry challenges: “Other, specify” questions, long lists of response options, “Code all that apply”, or free-form responses  Effort and expense of live- monitoring, field observation and verification APPROACH  Build CARI capabilities into QA processes for data quality and performance management  Can use audio recordings for primary data entry or as a form of double data entry to confirm/correct data from CATI or CAPI  May be able to reduce costs through use of recordings for performance review

8. RTI International Survey Step 4: Construct and Pretest the Questionnaire ISSUE  Question-level problems: Wording, ambiguity, lack of clarity  Translation effects: Dialects, regionalisms or country- specific vocabulary may not be recognized until production  Key data points need extra attention APPROACH  Use recordings and behavior coding during pretest and production to identify issues  Review of production responses by bilingual staff can confirm correctness or identify problems  Select questions to record, including buffer zones as needed.

9. RTI International Survey Step 5: Collect Data ISSUE  Managing interviewer performance & adherence to protocol  Motivating interviewers when the going gets rough, especially late in a cycle  Responsive design or adaptive total design; when subpopulations don’t respond APPROACH  Frequent review of a sample of cases helps “keep an eye” on interviewing  Positive feedback can reinforce good behavior and provide encouragement  Review of recordings from targeted subpopulations may give insight into reluctance

10. RTI International Survey Step 6: Analyze and Report ISSUE  Quantifying operational data for methodology reports  Explaining difficulties or unusual situations encountered  Estimating survey error APPROACH  CARI coding contributes to paradata metrics  Audio recordings may provide insight into or detailed examples of the challenges  Coded monitoring results offer a way to standardize measurement of ‘subjective’ error, such as consistency of questionnaire delivery

11. RTI International Addressing Structural Errors  Translation errors  Regional effects, such as vocabulary differences (Do you drink soda, pop or soft drinks? Would you drink from a water fountain?)  Logic errors: complex questionnaire logic may hide errors, in question sequences rarely followed  Values computed based on earlier responses and then used as fills; errors may be hard to detect during testing

12. RTI International Addressing Functional Errors  Human error: Data entry mistakes, presentation or mode effects, respondent reactions  Advance warning of mode effects, indications prior to analysis  Additional information about item-level non- response, including both “don’t know” answers and refusals

13. RTI International CARI Adoption by the US Census Bureau  CARI Interactive Data Access System  Multiple-year project, 2009 - 2012 Stage 1: Develop behavior coding and quality assurance (QA) interfaces (completed) Stage 2: Field test for behavior coding, recommendations for revision (completed) – Stage 3: Enhancements and revisions for BC and QA and addition of a Coaching component (in progress) – Stage 4: Field test for QA and Coaching (future)

14. RTI International Behavior Coding Field Test  American Community Survey (ACS) Content Test  Data collection, 2010 – Mail: August – September – CATI: October – CAPI: November  Behavior coding immediately after data collection – Dec 2010 – March 2011  Data analysis in progress at the Census Bureau (Contact: Joanne Pascale)

15. RTI International ACS Content Test CATICAPI Sample size23,67315,202 Completed + partial cases4,5236,384 Recording consent rate88.5%64.8% Interviews recorded4,0054,137 Interviews selected for coding807708 Training cases320 Cases coded out as inaudible487 Interviews coded727701 Recordings coded27,16322,676

16. RTI International Coding Page Compact, feature-rich interface Playback for audio Option to view image Navigation bar at left (blue) for non-sequential access to recordings Next / Prev buttons for sequential access Dynamic code lists are structured into categories, groups and codes Grid display summarizes codes assigned Room for notes

17. RTI International General Feedback Features that the CARI System users found most valuable included the following:  Unobtrusive nature of CARI  Ability to select cases (or questions) for review from among those recorded  Flexibility in defining what behaviors to code  Image display offering the exact display of wording (including fills) as well as the actual data entry value  Real-time monitoring of coding  Data available for extraction at any time

18. RTI International Future Possibilities for Error Management  Greater availability of recordings as a routine part of data collection  Expanded use of recordings in performance management and training or retraining of interviewers  Expanded use of CARI paradata in survey management  Development of low-burden performance metrics for interviewers and coders  Creation of an “error profile” for a survey  And… whatever else you can imagine!

19. RTI International Acknowledgements This project would not have been possible without the support of many people at the Census Bureau and RTI. We especially thank Sherry Thorpe, CARI Project Manager, Joanne Pascale, and the ACS Office of the Census Bureau. For more information, contact: Rita Thissen – rthissen@rti.orgrthissen@rti.org Hyunjoo Park – mpark@rti.orgmpark@rti.org Mai Nguyen – mnguyen@rti.orgmnguyen@rti.org