1. 1

2. 2 Issues in the Design and Testing of Business Survey Questionnaires: Diane K. Willimack U.S. Census Bureau Economic Census 10 11 The International Conference on Reshaping Official Statistics What We Know Now that We Didn’t Know Then – and What We Still Don’t Know

3. 3 Issues in the Design and Testing of Business Survey Questionnaires: Diane K. Willimack U.S. Census Bureau Economic Census 10 11 The International Conference on Reshaping Official Statistics Lessons from 10 Years of Pretesting Business Surveys at the U.S. Census Bureau

4. 4 Outline  Background –U.S. Census Bureau economic programs –Characteristics of economic surveys Response process  Pretesting procedures and methodology  Issues and recommendations

5. 5 Economic Survey Programs at the U.S. Census Bureau  More than 100 economic survey programs plus economic census  Authority – mandatory vs. voluntary  Data collection modes

6. 6  Survey organization’s perspective –Population, data, and survey design  Respondent’s perspective –Response process Characteristics of Economic Surveys

7. 7 The Response Process in Economic Surveys Cognitive response model PLUS Organizational processes

8. 8 Cognitive Response Model (Tourangeau, 1984) 1. Comprehension 2. Retrieval 3. Judgment 4. Communication Survey

9. 9 Response Process Model for Establishment Surveys (Sudman et al., 2000 ICES-2) Organizational Aspects 4. Comprehension 5. Retrieval 6. Judgment 7. Communication 1. 1. 2. 2. 3. 3. 8. 8. Business Survey

10. 10 Response Process Model for Establishment Surveys Encoding in Memory / Record Formation Selection / Identification of Respondent(s) Assessment of Priorities (Motivation) 4. Comprehension 5. Retrieval 6. Judgment 7. Communication 8. 8. 1. 1. 2. 2. 3. 3. Release of the Data Business Survey from Memory and / or Records from Memory and / or Records

11. 11 Pretesting Goals  Identify potential response error, e.g. --- –Question clarity –Contents of records –Response strategy  Suggest improvements to survey questions

12. 12  Dedicated staff  Consultant – client relationship with survey programs –More than 25 separate programs plus the economic census –Variety of survey types and topics  Electronic mode –Usability testing Economic Survey Pretesting at the U.S. Census Bureau

13. 13  Collaborative approach  Voluntary participation by respondents  Site visits  Post-testing follow-through  Multiple rounds – test, revise, re-test Pretesting Procedures

14. 14 Pretesting Methodology  Qualitative research  Traditional cognitive interviewing methods  Adaptations –Hypothetical questions / probes –Funnel approach: general  specific –Directive questions / probes  In-depth unstructured interviews

15. 15 Six Unresolved or Persistent Issues for Economic Surveys  Consequences  Methodology  Shortcomings  Recommendations

16. 16 Issue #1: Business survey response is labor-intensive, burdensome & costly  Consequences –Respondents unwilling or unable to complete a draft questionnaire prior to or during the pretest interview. –Researchers unable to use some traditional cognitive methods with draft questionnaires

17. 17  Pretesting methodology –Hypothetical probes  Shortcoming –Hypothetical report  Actual behavior Issue #1: Business survey response is labor-intensive, burdensome & costly

18. 18  Recommendations –Respondent debriefings during field period Actual behavior –“Pilot” test new questionnaires in the field Conduct respondent debriefings Evaluate data quality Issue #1: Business survey response is labor-intensive, burdensome & costly

19. 19 Issue #2: “Omnibus” surveys require multiple respondents or data sources  Consequences –No single “right” respondent –Hindrances to gathering data Knowledge Authority Motivation –Poor data quality

20. 20  Pretesting methodology –Detailed probes about data sources and communication practices  Shortcoming –Unable to meet with or interview downstream informants Issue #2: “Omnibus” surveys require multiple respondents or data sources

21. 21  Recommendations –New methodologies needed to research --- Data locations Company policies and practices for exchanging data internally Strategies for internal company communication –Develop data collection procedures that encourage, facilitate and support respondents’ data-gathering Issue #2: “Omnibus” surveys require multiple respondents or data sources

22. 22 Issue #3: Mismatch between survey concepts and company records  Consequences –Response error –Response burden

23. 23  Pretesting methodology –Directive questions  Shortcoming –Cognitive interviews devolve into a hybrid of cognitive probes and exploratory questions. Issue #3: Mismatch between survey concepts and company records

24. 24  Recommendations –Involve methodologists in content development –Conduct in-depth exploratory interviews to identify the “business model” for the concept of interest Use directive questions Issue #3: Mismatch between survey concepts and company records

25. 25 Issue #4: Questions request technical or financial data  Consequences –Methodologists  subject area experts –Methodologists  financial accountants –Cognitive interviews may not capture discrepancies between respondents’ answers and the intent of the question

26. 26  Pretesting methodology –Collaboration between methodologists and experts in subject matter or accounting  Shortcomings –Different approaches Methodologists: how the respondent answers questions without help Subject area / accounting experts: instruct respondents on how to report correctly Issue #4: Questions request technical or financial data

27. 27  Recommendations –Improved collaborative methods Experts aid methodologists in developing the interview protocol Methodologists report respondent behaviors; experts assess adequacy of response Experts participate in interviews as observers Issue #4: Questions request technical or financial data

28. 28 Issue #5: Many / most economic surveys are self-administered  Consequences –No interviewer –Heavy reliance on questionnaire to convey technical requirements of the data request –Need for detailed instructions Business Survey

29. 29  Pretesting Methodology –Probes on wording, language, visual design and placement of instructions  Shortcoming –Questionnaire design cannot fix every problem Issue #5: Many / most economic surveys are self-administered

30. 30  Recommendation –Encourage new paradigms in --- Questionnaire development –Top-down and bottom-up –Identify collectable data with identifiable shortcomings Use of data –Consider context and utility of reported data –Consider shortcomings in data analysis and interpretation Issue #5: Many / most economic surveys are self-administered

31. 31 Issue #6: Heavy reliance on cognitive interviewing to identify and correct questionnaire problems  Consequence –Improper or unrealistic expectations –Difficult to evaluate effectiveness

32. 32  Pretesting Methodology –Multiple iterations of design and testing – test, revise, re-test  Shortcomings –Cognitive testing is --- Qualitative methodology Somewhat artificial Issue #6: Heavy reliance on cognitive interviewing to identify and correct questionnaire problems

33. 33  Recommendations –Use multiple complementary methods Helps gain confidence in results –Develop and monitor meaningful objective measurements of data quality Issue #6: Heavy reliance on cognitive interviewing to identify and correct questionnaire problems

34. 34 Conclusion: 10 Years of Pretesting Business Surveys at the U.S. Census Bureau  Cognitive testing –Useful –Integrated into survey development

35. 35 Conclusion: The Next 10 Years of Pretesting Business Surveys  Challenges –Multiple research methods –Quantitative evaluation of effectiveness –Organizational behavior and the survey response process

36. 36  Thank you.  Questions?? Comments?? Diane K. Willimack U.S. Census Bureau diane.k.willimack@census.gov