1. Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison

2. 2 Outline Introduction & Background Development of the Guidelines Preliminary Guidelines Application of Guidelines Conclusion & Next Steps

3. Introduction & Background

4. 4 U.S. Census Bureau: –…leading source of quality data…nations people and economy –Tries to minimize burden of data collection Burden = time, level of effort Reduce cognitive burden through visual design, structure of data request

5. 5 Introduction & Background (contd) Paper does not focus on specific question wording and order issues. Paper presents preliminary guidelines in questionnaire design – formatting, layout, navigation, instructions, and data request phrasing and style.

6. 6 Development of the Guidelines

7. 7 Development of the Guidelines: Census Bureau Agency standards for various survey quality issues No standard for design of questionnaires Economic surveys moving toward consistent look and feel

8. 8 Development of the Guidelines: ESMS Establishment Survey Methods Staff –Group of survey methodologists –In-house consultants to economic programs –Noticed differences across questionnaires –Survey programs work with ESMS on questionnaire design and pretesting –Catalyst towards consistency

9. 9 Development of the Guidelines: Methods Questionnaire design principles from standard texts Special attention to literature on visual design Based on pretesting a variety of questionnaires with respondents

10. 10 Development of the Guidelines: Considerations List of design elements to consider Used to assess tradeoffs within specific survey conditions Not a cookbook

11. Preliminary Guidelines

12. 12 Guideline A: Text styles Be aware of text styles, and how they are used for emphasis within a survey instrument.

13. 13 Guideline A: Example 2004 Annual Survey of Local Government Finances

14. 14 Guideline A: Example

15. 15 Guideline A: Text styles These are favored by respondents: –Print item numbers in reverse-print bubbles, e.g.,,, –Print questions in bold black text. –Print instructions in plain text or italics. –Use an 8-point font or larger.

16. 16 Guideline A: Text styles Answer spaces: –Open, non-delineated vs. delineated –Respondents do not seem to have a strong preference –Be consistent

17. 17 Guideline A: Text styles Key codes / Punch codes: –De-emphasize processing codes for respondents –Print in a darker shade of background color –Example:

18. 18 Guideline B: Reduce visual clutter. Use natural navigational paths. Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents.

19. 19 Guideline B: Reduce visual clutter. Use natural navigational paths.

20. 20 Guideline B: Reduce visual clutter. Use natural navigational paths. Format in 1997 Economic Census confusing: –Lines were speedbumps to navigation –Not clear how respondent was to navigate two adjoined columns Format changed for 2002 –Single column of questions –Generally, a single column of response options

21. 21 Guideline B: Reduce visual clutter. Use natural navigational paths. Two columns of questions –Occasionally done when questions: Are shorter Do not involve extensive instructions Do not ask for numerical information –Survey of Business Owners (SBO) Information collected is categorical or ordinal, not interval or ratio

22. 22 Guideline B: Reduce visual clutter. Use natural navigational paths. Spread out the questions across more pages Use bulleted lists More open space = more user-friendly

23. 23 Guideline C: Instructions Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.

24. 24 Guideline C: Instructions Instructions convey specifications, intent of question Respondents tend not to pay attention to instructions, or only look when they think they need them

25. 25 Guideline C: Instructions 2 Goals for instructions: –Eliminate, or reduce, amount of instructions located separate from question –Place instructions/information where it is most needed

26. 26 Guideline C: Instructions Convert instructions into questions so respondents attend to them –When content critical to correct interpretation of later questions –When it helps clarify/correct reported data Example (2004 Medical Expenditure Panel Survey):

27. 27 Guideline D: Avoid sentence fragments and key words Phrase data requests as questions or imperative statements, not as sentence fragments or key words.

28. 28 Guideline D: Avoid sentence fragments and key words Types of data requests –Question (question word, question mark) –Imperative statement (report, enter, add) –Sentence fragment (key words, no verb)

29. 29 Guideline D: Avoid sentence fragments and key words Example, 2002 Economic Census: –Is this establishment physically located inside the legal boundaries of the city, town, village, etc.? Versus: –Type of municipality where this establishment is physically located

30. 30 Guideline D: Avoid sentence fragments and key words Cognitive interview study: respondents prefer questions over sentence fragments Survey methodology grad students: questions more effective, imply respondent has to do something

31. 31 Guideline E: Simple vs. complicated questions Ask additional, simple questions, rather than fewer, more complicated ones.

32. 32 Guideline E: Simple vs. complicated questions Length and complexity of question affects how long it takes for respondent to understand it More complicated questions might also be more likely to be double-barreled May be easier for respondents to answer series of shorter, simpler questions

33. 33 Guideline E: Simple vs. complicated questions Example: 2002 Industrial Research & Development Survey

34. 34 Guideline E: Simple vs. complicated questions 2006 R&D Survey

35. 35 Guideline F: Use matrices judiciously Use matrices judiciously, and consider the likely respondents background when deciding whether or not to use them.

36. 36 Guideline G: Be consistent! Items should be numbered in the same way Bold and italicized print should always mean the same thing Navigation path should remain constant Data requests should be in the same form Variation can be confusing for respondents

37. 37 Application of Guidelines

38. 38 Guideline A: Text Styles Work in progress among estab surveys at Census Bureau Economic Census automation required consistent design features –Plain text: questions –Italics: instructions, definitions, include/exclude lists –Bold: Separating items into categories, sums of added lines, emphasis within questions

39. 39 Guideline A: Text Styles Survey of Business Owners: –Questions: plain –Instructions: bold and/or italics –Emphasis within questions: bold and/or italics

40. 40 Guideline B: Reduce visual clutter. Use natural navigational paths. Survey of Residential Alterations and Repairs (SORAR) –Two columns to one column –Removed or lightened lines –Use of bulleted lists –More open space

41. 41 Guideline B: Reduce visual clutter. Use natural navigational paths. Old SORAR

42. 42 Guideline B: Reduce visual clutter. Use natural navigational paths. New SORAR

43. 43 Guideline B: Reduce visual clutter. Use natural navigational paths. Two columns: Survey of Business Owners

44. 44 Guideline B: Reduce visual clutter. Use natural navigational paths. BEA quarterly foreign direct investment (FDI) Complex navigational path Crowded text

45. 45 BEA form revised Guideline B: Reduce visual clutter. Use natural navigational paths.

46. 46 Guideline C: Instructions Economic census continues to use separate instruction sheets and booklets Two questionnaires moved many instructions to questionnaire: –2007 Commodity Flow Survey –BEA quarterly FDI form

47. 47 Guideline C: Instructions

48. 48 Guideline D: Avoid sentence fragments and key words Mixed success in application across surveys

49. 49 Guideline D: Avoid sentence fragments and key words Commodity Flow Survey uses mix of questions and imperative statements

50. 50 Guideline E: Simple vs. complicated questions Example: 2002 Survey of Business Owners

51. 51 Guideline E: Simple vs. complicated questions Example: 2007 Survey of Business Owners (still in cognitive pretesting)

52. 52 Guideline F: Use matrices judiciously Example: BEA quarterly FDI

53. 53 Guideline F: Use matrices judiciously

54. 54 Conclusion & Next Steps

55. 55 Conclusion & Next Steps Guidelines are preliminary, based on principles of questionnaire design Further refinement is necessary before they can be applied to establishment surveys at Census Bureau

56. 56 Conclusion & Next Steps Other agencies/organizations may develop guidelines that work better for their environment There is no cookbook for questionnaire design

57. 57 Thank you! Rebecca L. Morrison 301-763-7595 Rebecca.L.Morrison@census.gov

58. 58 Preliminary Guidelines A: Be aware of text styles, and how they are used for emphasis within a survey instrument. B: Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents. C: Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.

59. 59 Preliminary Guidelines D: Phrase data requests as questions or imperative statements, not as sentence fragments or key words. E: Ask additional, simple questions, rather than fewer, more complicated ones. F: Use matrices judiciously, and consider the likely respondents background when deciding whether or not to use them.