Standards and Guidelines for Conducting Cognitive Interviews Jennifer Edgar WSS Seminar February 23, 2017

OMB Standards Intended to document principles and practices Federal agencies must adhere to Not many have been produced (15) Conducting statistical surveys CI Standards are the 2nd addendum to that Not a roadmap or playbook, agencies still need experienced practitioners to carry them out

Standards and Guidelines Standards – methodological requirement What needs to be done Guidelines – best practices on how to fulfill the standards How to do it

Cognitive Interviewing Standards Methodological Plan Sample Protocol for Interview Analysis Transparency Reporting Public Accessibility

1. Methodological Plan A plan must be developed prior to conducting a CI Study Plan should include at least Study objective Sampling plan Recruitment plan Location of interviews Development of interview guide Analysis plan

2. Sample Selection Sample selection must be done in a purposeful way to ensure that respondents have the characteristics necessary to provide data required to meet the study objectives

Sampling Guidelines Sample Size Sample Characteristics Recruitment Plan

3. Interview Protocol A protocol is required and must be designed to collect the processes by which a respondent interprets and responds to a question as well as any difficulties experienced by a respondent in providing and answer

Protocol Guidelines Iterative Process Interview as Analyst Mode and Context Effects

4. Systematic Analysis The data produced must undergo a systematic analysis Ensure no particular case is overemphasized and that findings represent the full range of responses

Analysis Guidelines Empirical Evidence Process for Conducting Analysis

5. Transparent Analysis Analysis must be transparent such that an outsider can understand and assess the findings Each step must be documented Findings can be traced back to raw data Allows readers to examine and judge the quality of the data and results

Transparent Analysis Guidelines Audit Trail Maintenance of Raw Data

6. Final Report Methods, results and conclusions must be documented in a final report Improves the rigor and credibility of a study Maximizes transparency of analysis Offers clear and conceptually adequate descriptions of all aspects of the study and its execution

Report Guidelines Issues addressed in report Contents of a report

7. Publically Available Final reports must be made available to the public if results are referenced in publications or data collection documentation