2. Collaborating for Quality: A Cross-Discipline Approach to Questionnaire Content Evaluation in Business Surveys Diane K. Willimack Peter Gibson U.S. Census Bureau

3. 3 Outline Background –Business surveys –Response process Questionnaire development, pretesting & evaluation –Why collaborate? –An example –Benefits of the collaborative approach

4. 4 Nature of the collected data –Financial –Technical definitions Expect data to be in records Business Surveys (1)

5. 5 Nature of response –Person answers survey questions for the business –Cognitive response process Business Surveys (2)

6. 6 The Response Process in Business Surveys Cognitive response model PLUS Organizational processes

7. 7 Tourangeau’s (1984) Cognitive Response Model 1. Comprehension 2. Retrieval 3. Judgment 4. Communication Survey

8. 8 Organizational Aspects 4. Comprehension 5. Retrieval 6. Judgment 7. Communication 1. 1. 2. 2. 3. 3. 8. 8. Business Survey Response Process in Business Surveys (1) (Sudman et al., 2000)

9. 9 Encoding in Memory / Record Formation Selection / Identification of Respondent(s) Assessment of Priorities (Motivation) 4. Comprehension 5. Retrieval 6. Judgment 7. Communication 8. 8. 1. 1. 2. 2. 3. 3. Release of the Data Business Survey from Memory and / or Records from Memory and / or Records Response Process in Business Surveys (2) (Sudman et al., 2000)

10. 10 Encoding in Memory / Record Formation Selection / Identification of Respondent(s) Assessment of Priorities (Motivation) 4. Comprehension 5. Retrieval 6. Judgment 7. Communication 8. 8. 1. 1. 2. 2. 3. 3. Release of the Data Business Survey from Memory and / or Records from Memory and / or Records Response Process in Business Surveys (3) (Sudman et al., 2000)

11. 11 Management needs Regulatory requirements Accounting standards Knowledge domain of experts in accounting practicesKnowledge domain of experts in accounting practices Record Formation

12. 12 Respondent Records Accountant Putting the Pieces Together for Quality Response

13. 13 Answers survey questions for the business –Cognitive response process Expect respondent to know – –What is in records –How to get them Knowledge domain of cognitive survey methodologistsKnowledge domain of cognitive survey methodologists The Respondent in Business Surveys

14. 14 Encoding in Memory / Record Formation Selection / Identification of Respondent(s) Assessment of Priorities (Motivation) 4. Comprehension 5. Retrieval 6. Judgment 7. Communication 8. 8. 1. 1. 2. 2. 3. 3. Release of the Data Business Survey from Memory and / or Records from Memory and / or Records Response Process in Business Surveys (4) (Sudman et al., 2000)

15. 15 Questionnaire Development, Pretesting & Evaluation Challenges –Technical definitions –Data availability –Labor-intensive response process Expertise –Subject matter –Accounting practices –Cognitive survey methodology

16. 16 Respondent Records Cognitive Survey Methodologist Accountant Putting the Pieces Together for Quality Response: Cross-Discipline Collaboration

17. 17 Questionnaire Development & Pretesting Methodologist role Cognitive testing –Non-directive probes get at cognitive steps Shortcoming: –Limited by labor- intensive response process –Unable to access records Accountant role Aid stakeholders in specifying constructs –Improve initial questions –Align with records Review draft form & protocol Shortcoming: –Directive approach

18. 18 Questionnaire Evaluation Methodologist role Respondent debriefings –Post-collection –Response strategy Shortcoming: –Does not re-create construction of the response –Does not validate the response Accountant role Review reported data –Comparison to public data sources –Aid identifying & correcting reporting errors Shortcoming: –Lacks contact with respondent

19. Cross-Discipline Collaboration: Methodologist + Accountant An Example 19

20. Example Draft Question 1.What is the amount of licensing fees or royalty income received by your company’s domestic operations? 2.Of the amount reported in item 1, how much was from: a)Patents? b)Copyrights? c)Computer Software? d)Other (specify _______________) 20

21. Example Cognitive Protocol 1.Tell me in your own words what the terms “licensing fees” and “royalty income” mean to you in the context of this question. 2.Please explain how these are similar or different. 3.What sorts of “royalty income” or “licensing fees” does your company receive? 4.How would you go about reporting “royalty income” or “licensing fees” for your domestic operations? What records would you use? 5.How would you go about reporting the amounts earned from patents, copyrights, etc.? What records would you use? 21

22. Example Technical Issues Identified by the Accounting Expert 1.Forms of Intellectual Property (IP) tracked in the corporate legal entity that directly owns the IP. 2.The legal office structures the IP licensing arrangements (and not the Financial area). 3.Contracts consist of several forms of IP. 4.The company’s IP might be enhanced by the licensee and resulting income combines the original with the enhanced. 5.Attributing income to forms of IP (e.g., patents, copyrights) is a theoretical exercise. Actual pricing reflects opportunity costs, present value, profit potential, value of future development. 22

23. Example Enhanced Protocol with Technical Probes from the Accountant 1.Do you have “patents, trademarks, copyrights?” 2.Are these IP owned directly by the domestic legal entity’s corporate or subsidiary business units? 3.Is any of this IP owned by a foreign subsidiary? 4.Do any of the “licensing fee or royalty income” accounts contain a coding for income tax purposes? 5.Would there be a difference between the amount you would report for “world-wide” or “domestic?” 6.Who handles licensing arrangements in sub-units? 23

24. Example Outcomes from the Collaborative Interview Burden: –Reporting total domestic “licensing fees and royalty income” depends on data access. –Analysis of each licensing arrangement needed to attribute income to individual forms of IP (e.g., patents, copyrights). –Would not provide robust data. Clarification of the construct being measured –Stakeholder reduced requirements, dropped questions Protocol –Cognitive methodologist strengthened probes of technical issues vis-à-vis data availability 24

25. 25 Methodologist + Accountant = Robust Pretest Interviews Methodologist role More comfortable with technical nuances More effective probing Accountant role Gained insights into variety of respondents and reporting issues

26. 26 Methods Probing anticipated problems Prepared to investigate hypotheses about specific terms and concepts Results Respondent’s initial response augmented with reaction to possible alternatives Insights into burden and data quality Methodologist + Accountant = Robust Pretest Interviews

27. 27 Benefits of Cross-Discipline Collaborative Approach Improved draft questionnaire for pretesting More effective pretest interviews More efficient pretesting strategy More robust results and useable recommendations Additional expertise for review of reported data

28. Thank you! Diane K. Willimack U.S. Census Bureau diane.k.willimack@census.gov