1. 1 RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box 12194 ■ Research Triangle Park, North Carolina, USA 27709 Phone 919-990-8456e-mail mcominole@rti.orgFax 919-541-7014 Challenges of Designing and Implementing Multimode Instruments Jennifer Wine, Melissa Cominole, Ruth Heuer, and John Riccobono March 15, 2006

2. 2 Why Multimode?  NCES postsecondary studies – primarily CATI in the past, with some CAPI  NCES required the addition of the web self-administered option to help offset the decline in survey response rates  Characteristics of our postsecondary populations  Technologically savvy  Unconventional schedules  Cell phones replacing land lines  Time and money

3. 3 Data Collection Methodologies Used for the Postsecondary Studies

4. 4 Typical Timeline for Data Collection Source: BPS FS March2006 March2006 April2006 June 2006 October2006 Send data collection announcement mailing Begin web,self-administered data collection Begin telephone interviewing (CATI) Begin field interviewing (CAPI) End data collection

5. 5 Multimode Design Challenges and Adaptations

6. 6 Interview Specifications -- All Modes Source: NPSAS FS

7. 7 Selected Interview Screen – Web Mode Source: NPSAS FS

8. 8 Selected Interview Screen – CATI Mode Source: NPSAS FS

9. 9 Multimode Design Challenges and Adaptations

10. 10 Interviews Completed during Early Response Incentive Period* *When evaluated in the FT studies, early response incentives were shown to increase the likelihood of a response (p<.05).

11. 11 Design for Telephone Prompting Experiment Source: BPS FT

12. 12 Interview Participation, by Prompting Status Type of prompting Response rates at end of early period Prompted21.6* Not prompted 10.4 *p<.01 Source: BPS FT

13. 13 Prompting and Base Year Response Status Response rates at end of early period Base year respondents Prompted25.0 Base year nonrespondents Prompted20.8* Not prompted15.1* *p<.05 Source: BPS FT

14. 14 Average Call Count per Completed Interview, by Mode p<.0001 Source: BPS FT

15. 15 Multimode Design Adaptations

16. 16 Proportion of Time in the Interview and in Transit for Web and CATI Respondents  Time on-screen  Time in transit *p<.0001 Source: B&B FS Web CATI Total 34.4 34.8 On-screen24.4 29.1* Transit10.0 5.7*

17. 17 Web Connection Speed

18. 18 Multimode Design Adaptations

19. 19 Help Desk Calls

20. 20 On-Line Help Text Source: BPS FT

21. 21 Rate of Help Text Access by Mode p<.0001 Source: NPSAS FS

22. 22 Multimode Design Adaptations

23. 23 Major/Field of Study Coder

24. 24 Multimode Design Adaptations

25. 25 Mode Differences in Item Reliability  Very few items with significant differences between modes, with no patterns evident  Some studies – NSOPF FT and BPS FT – showed no mode differences on any items

26. 26 Multimode Design Adaptations

27. 27 Percent of Indeterminate Responses, by Mode  38 interview questions, and over 26 percent of those in the finance section, had nonresponse rates of at least 10 percent  CATI/CAPI respondents provided an indeterminate response for nearly 2.5 percent of all responses.  Web respondents provided an indeterminate response for nearly 5 percent of all responses (p<.05). Source: B&B FT 5% 2.5%

28. 28 Response Conversion Text for Selected Items Source: B&B FS

29. 29 Use of Conversion Text  Percent converted to an explicit response ranged from 0 percent to 100 percent in the B&B FS; comparable rates (18 to 100 percent) were observed in the BPS FT  Items less likely to convert: dollar amounts related to income, monthly expenses, and overall debt (28 to 73 percent)  Web respondents in the BPS FT were more likely than interview respondents to convert on two items – race and undergraduate loan debt; no other mode differences were observed

30. 30 Next Steps?  Different incentive amounts at different points in data collection  Prompting difficult subgroups  Evaluate the use of help text relative to item-level nonresponse and reliability  Continue efforts to assess differences in data quality across modes

31. 31 Contact Information Melissa B. Cominole RTI International 919-990-8456 mcominole@rti.org Slides and paper available at: www.rti.org/fedcasic