From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler United States Bureau of Economic Analysis

2 Overview – Redesign of BEA Form BE-605 Reimbursable project with BEA Multiple, intensive stages of research, design, testing, revision Qualitative and quantitative evaluations New BE-605 form mailed out in March 2007

3 Outline Background – BEA Form BE-605 Summary of stages of research Findings Next steps

4 Background BE-605 (quarterly) and BE-15 (annual) Foreign Direct Investment (FDI) – >10% of voting rights in an incorporated entity (or equivalent interest in unincorporated entity) Sample – foreign-owned businesses in US

5 Research stages and methods Background investigations –Expert review of form –Focus groups with survey analysts –Observations of callbacks –Respondent debriefings

6 Research stages and methods Cognitive testing of mockups

7 Research stages and methods Pilot test –Respondent debriefings –Evaluation questions

8 Findings from background research Data collection and design issues: Dense formatting Economic accounting concepts differ from US GAAP Separate instructions

9 Recommendations Formatting recommendations –Letter size pages (not legal) –More “open” visual design –Incorporate instructions into form: within questions adjacent to questions on facing page key instructions become questions –Diagrams to augment questions

10 Original form

11 Recommendation: More “open” design

12 Recommendation: Embedded instructions

13 Recommendation: Adjacent instructions

14 Recommendations: Questions from instructions, and diagrams

15 Cognitive testing Problematic sections mocked-up and tested (not entire forms) 5 rounds of cognitive interviews Iterative process: –mock-ups revised between rounds –design features modified & re-tested

16 Findings from cognitive testing New design features were effective: –Majority of respondents liked the more open format –Embedded instructions would likely improve data quality –Some “Instructions  Questions” and diagrams required extensive modification, though most did not

17 Developing 605 pilot Redesigned entire BE-605 form Close collaboration with BEA survey staff

18 Pilot test Mailed 2 nd quarter 2006 Sent to sub-sample (n = 653; survey sample ~4,000) Completing pilot form optional Response rate: 53% (350 returned forms) during initial 8-week processing period

19 Pilot form evaluation Respondent debriefings Evaluation questions in form

20 Findings – R debriefings Favorable impressions of new form: –More pleasing to the eye –White reporting spaces easier to see against colored background –Larger font, bold text, more space –Letter size pages easier to handle –Diagrams were helpful –Embedded instructions helpful

21 Findings – R debriefings Drawbacks: –More pages –Booklet format makes it harder to fax –Took longer to complete than original form (disruption of reporting routine)

22 Limitations Stymied by experienced Rs, reporting routines… …though even experienced Rs were helped by new formatting and instructions.

23 Findings – Evaluation questions Four questions appended to survey Each item received between 279 and 290 responses

24 Findings – Evaluation questions Instruction placement? 79.3% - Near questions 18.3% - In separate booklet 2.4% -Not applicable – I haven’t completed the usual BE-605

25 Findings – Evaluation questions Open space and number of pages? 48.7% - less open space, fewer pages 47% - more open space, more pages 4.3% - Not applicable – I haven’t completed the usual BE-605

26 Findings – Evaluation questions Use of diagrams? 65.4% - very helpful or somewhat helpful 11.7% - only a little helpful 18% - did not need them 3.9% - confusing

27 Findings – Evaluation questions Easy/difficult compared to usual form? 50.5% - harder 42.5% - easier 7%- Not applicable – I haven’t completed the usual BE-605

28 Findings – Evaluation questions Instructions preferred near questions Organization charts helpful Easier or more difficult? Preference for more/fewer pages?

29 Summary Redesigned BE-605 went into production in March 2007 –Recommended analysis of edit checks during form processing

30 Summary Multi-method approach to comprehensive survey redevelopment

31 Thanks! Alfred D. Tuttle Rebecca L. Morrison David Galler