Cognitive Interview Standards and Guidelines at the U. S Cognitive Interview Standards and Guidelines at the U.S. Census Bureau: Implementation and use across Languages Patricia Goerman, U.S. Census Bureau Presented at the Washington Statistical Society Seminar on Implementing the New OMB Cognitive Interviewing Standards and Guidelines Washington, D.C. Thursday, February 23, 2017 Disclaimer: This presentation is intended to inform people about research and to encourage discussion. The views expressed are those of the author and not necessarily those of the U.S. Census Bureau.

U.S. Census Bureau Survey Methodology Research Groups Household surveys: Center for Survey Measurement Questionnaire Development and Evaluation Group Social Science Research Group Language and Cross-Cultural Research Group Human Factors and Usability Research Group Emerging Methods Group Economic surveys: Economic Statistical Methods Division Data Collection Methodology and Research Branch Economic Management Division Respondent Management and Promotion Branch

History of Cognitive Interviews at Census Both CSM and ESMD have gone through several name changes and iterations Cognitive testing and other pretesting research since the 90s Census Bureau Standard: Statistical Quality Standard A2: Developing Data Collection Instruments and Supporting Materials https://www.census.gov/about/policies/quality/standards/standarda2.html 

Census Bureau Pretesting Standard One or more of the following pretesting methods must be used: Cognitive interviews, Focus groups (if group completes self administered instrument followed by discussion), Usability techniques (if focused on respondent understanding), Behavior coding of interviewer/respondent interactions, Respondent debriefings (in conjunction with field test or actual data collection), Split panel tests Expert reviews* (in special circumstances only)

Complexity of Implementation in Diverse Situations New OMB standards address importance of: “Examining issues of comparability – for example, the accuracy of translations or equivalence across socio-cultural or other relevant subgroups Language and Cross-Cultural Research Group at Census: great illustration of issues agencies can face in implementing cognitive testing standards

A.1: Methodological Plan CSM has long developed methodological plans prior to the beginning of a study but they have not always been a formal “deliverable” or product. Usability lab typically begins each product with a written study plan Includes: Study objective Sampling plan Recruitment of respondents Location of interviews Interview protocol Analysis plan

A.2: Sample Selection Particularly complex in the case of non-English projects Example: American Community Survey testing of Spanish instrument Generally for Spanish speakers: need for range of national origins: Central America, South America, Puerto Rico, Cuba, Dominican Republic Geographic diversity required to most easily find all groups Respondents who are Spanish dominant

Sample Characteristics ACS lengthy instrument: tested in segments Need to look at types of question being tested to help choose respondent types Segment 1: Household roster, Demographics: Relationship question, Hispanic origin, race, other living quarters questions Need for people who would be expected to choose variety of response options

Sample Size Depends on type of questions being tested, types of respondents relevant. Race question: Are you White; Black, African American; American Indian or Alaska Native; Asian; Native Hawaiian or Other Pacific Islander; or Some other race? Is this the only question being tested? Are there other questions where respondent characteristics might affect interpretation?

Recruitment Plan Recruitment plans are crucial with multi-lingual projects especially when working with contractors. Where are we most likely to find respondents that speak particular languages? Plan for geographic diversity What kinds of characteristics can overlap? Diverse responses to race question Household size and relationship types

A.3: Interview Guides Complexity with multi-lingual project Decisions to make related to type of probing: Think aloud (difficulty for some populations) Scripted v. emergent probes Concurrent v. retrospective probing Translation/ adaptation of cognitive interview protocol Simultaneous development v. one language and then translation

A.4: Systematic Analysis 1. Conducting interviews 2. Interview summaries 3. Comparing summaries across respondents Complexity in multi-lingual projects with multiple teams 4. Comparing themes across subgroups Use of qualitative analysis software 5. Making conclusions 6. Preparing initial recommendations

A.5: Transparent Analysis Ability for study findings to be traced to original data collected in interviews Documenting steps in analytic process Spreadsheet, text document, qualitative analysis software Long used qualitative software in contracts, has not typically been a deliverable Reporting on details of findings

ACS Testing Example Original Q: Does this <housing unit type fill>have a stove or range? Spanish: ¿Tiene <esta casa fill> una estufa? Round 1: 27 Spanish speakers said “yes”. Almost all understood as intended 2 Peruvians: confusion about “estufa”: “Space heater/ serving tray”. Their suggestions: “hornilla” (oven) or “cocina” (kitchen) Round 2 test: estufa u hornilla (2 synonyms) Results: Confusion amongst other Spanish speakers. Thinking it was an either/or choice rather than synonyms. Confusion. Final solution: Estufa para cocinar (stove for cooking)

A.6: Final Reports Reports containing: Study objectives Sampling Data collection, interviewers, methods Analysis Results Credibility and transferability of results Issue of quick report/ report to sponsor v. publication online for public

A.7: Reporting Results Survey data documentation should include links to cognitive testing reports Could be done directly on website of survey for which cognitive interviews were conducted or another public website or database Something that needs to be addressed at a higher level within agencies

Contracting Language and Cross-cultural Research Group often does not have language capability for every language required Common study languages: Spanish, Chinese, Vietnamese, Korean, Russian, Arabic Decennial census often requires additional languages New contracts will contain reference to standards/guidelines to be sure that contractors are aware Need for additional deliverables, such as documentation of analysis

Summary and Conclusions New standards and guidelines codify a number of practices that many agencies have been doing more informally for years Census Bureau is in the process of verifying whether any changes to our internal Pretesting Standards need to be made to align with the new OMB standards Helpful guidance for agencies who are just starting to do pretesting

Cognitive Interview Standards and Guidelines at the U. S Cognitive Interview Standards and Guidelines at the U.S. Census Bureau: Implementation and use across Languages Patricia Goerman, U.S. Census Bureau For more information: E-mail: Patricia.L.Goerman@census.gov