1 RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box ■ Research Triangle Park, North Carolina, USA Phone Challenges of Designing and Implementing Multimode Instruments Jennifer Wine, Melissa Cominole, Ruth Heuer, and John Riccobono March 15, 2006

2 Why Multimode?  NCES postsecondary studies – primarily CATI in the past, with some CAPI  NCES required the addition of the web self-administered option to help offset the decline in survey response rates  Characteristics of our postsecondary populations  Technologically savvy  Unconventional schedules  Cell phones replacing land lines  Time and money

3 Data Collection Methodologies Used for the Postsecondary Studies

4 Typical Timeline for Data Collection Source: BPS FS March2006 March2006 April2006 June 2006 October2006 Send data collection announcement mailing Begin web,self-administered data collection Begin telephone interviewing (CATI) Begin field interviewing (CAPI) End data collection

5 Multimode Design Challenges and Adaptations

6 Interview Specifications -- All Modes Source: NPSAS FS

7 Selected Interview Screen – Web Mode Source: NPSAS FS

8 Selected Interview Screen – CATI Mode Source: NPSAS FS

9 Multimode Design Challenges and Adaptations

10 Interviews Completed during Early Response Incentive Period* *When evaluated in the FT studies, early response incentives were shown to increase the likelihood of a response (p<.05).

11 Design for Telephone Prompting Experiment Source: BPS FT

12 Interview Participation, by Prompting Status Type of prompting Response rates at end of early period Prompted21.6* Not prompted 10.4 *p<.01 Source: BPS FT

13 Prompting and Base Year Response Status Response rates at end of early period Base year respondents Prompted25.0 Base year nonrespondents Prompted20.8* Not prompted15.1* *p<.05 Source: BPS FT

14 Average Call Count per Completed Interview, by Mode p<.0001 Source: BPS FT

15 Multimode Design Adaptations

16 Proportion of Time in the Interview and in Transit for Web and CATI Respondents  Time on-screen  Time in transit *p<.0001 Source: B&B FS Web CATI Total On-screen * Transit *

17 Web Connection Speed

18 Multimode Design Adaptations

19 Help Desk Calls

20 On-Line Help Text Source: BPS FT

21 Rate of Help Text Access by Mode p<.0001 Source: NPSAS FS

22 Multimode Design Adaptations

23 Major/Field of Study Coder

24 Multimode Design Adaptations

25 Mode Differences in Item Reliability  Very few items with significant differences between modes, with no patterns evident  Some studies – NSOPF FT and BPS FT – showed no mode differences on any items

26 Multimode Design Adaptations

27 Percent of Indeterminate Responses, by Mode  38 interview questions, and over 26 percent of those in the finance section, had nonresponse rates of at least 10 percent  CATI/CAPI respondents provided an indeterminate response for nearly 2.5 percent of all responses.  Web respondents provided an indeterminate response for nearly 5 percent of all responses (p<.05). Source: B&B FT 5% 2.5%

28 Response Conversion Text for Selected Items Source: B&B FS

29 Use of Conversion Text  Percent converted to an explicit response ranged from 0 percent to 100 percent in the B&B FS; comparable rates (18 to 100 percent) were observed in the BPS FT  Items less likely to convert: dollar amounts related to income, monthly expenses, and overall debt (28 to 73 percent)  Web respondents in the BPS FT were more likely than interview respondents to convert on two items – race and undergraduate loan debt; no other mode differences were observed

30 Next Steps?  Different incentive amounts at different points in data collection  Prompting difficult subgroups  Evaluate the use of help text relative to item-level nonresponse and reliability  Continue efforts to assess differences in data quality across modes

31 Contact Information Melissa B. Cominole RTI International Slides and paper available at: