Presented at the 2006 American Educational Research Association Annual Meeting April 7 – 11 San Francisco, CA Techniques for Generating Higher Response Rates Presented at the 2006 American Educational Research Association Annual Meeting April 7 – 11 San Francisco, CA Kevin Kenward, PhD Director of Research National Council of State Boards of Nursing Lynn C. Webb, Ed.D. Testing Consultant

Increasingly, [people] are refusing to participate in consumer surveys. A truly representative survey may be unattainable… The New York Times June 2004

Status of Survey Nonresponse Falling cooperation rates in the developed world (de Leeuw and de Heer, 2002) Greater decline in one-time surveys vs. longitudinal surveys (AAPOR, 2003) Greater decline in unit nonresponse than item nonresponse

An electronic survey was conducted to compare the roles of two types of Advanced Practice Registered Nurses Nurse Practitioners (NP) Clinical Nurse Specialists (CNS) Job Analysis

The professional sophistication of Advanced Practice RNs led us to believe that an electronic survey would be appropriate, and the electronic format offered advantages over paper.

The history of nurse response rates 40% led us to believe we would have strong participation.

The response rate was disappointingly low. 13% In desperation, nurses were randomly selected from the mailing list and asked why they did not participate in the study. TOO BUSY was the recurring theme in the explanations offered.

How do We Get People to Complete Our Surveys and Provide us with Good Information ??

Techniques for Generating Higher Response Rates l Preliminary or Advance Notification l Follow-ups l Appeals l Postage and Mailing Classes l Personalization l Incentives

Decreasing Response Rates QUESTIONNAIRE LENGTH - MAIL 4 PAGES OR LESS No Effect MORE THAN 4 PAGES - 8%

MONETARY INCENTIVES Prepaid - Creates Trust and Obligation Amount - Symbolic Token of Appreciation Understanding the Techniques

Phone Call Reminder Mail Survey Long Form Version 1 Mail Survey Long Form Version 2 Mail Survey Short Form N=1,500 N=1,000

Mail Survey Long Form Version 1 + Photo on Cover Letter Mail Survey Long Form Version 1 + Photo on Cover Letter

Mail Survey + $1 Incentive Mail Survey + $1 Incentive Mail Survey + $2 Incentive Mail Survey + $2 Incentive Mail Survey + $5 Incentive Mail Survey + $5 Incentive Monetary Incentives N=250

Mail Survey + Starbucks $3 Gift Certificate Mail Survey + $250 Drawing N=250 N=1,000

2% 26% Results

Mail Survey Long Form Version 1 Mail Survey Long Form Version 2 Mail Survey Short Form 24% 26% 30% Results

Increasing Response Rates Monetary Incentives RESPONSE AMOUNT RATE $1.0043% $2.0042% $5.0054% SOURCE: Yu and Cooper, JMR.

Increasing Response Rates NON-MONETARY INCENTIVES Starbucks $3 Gift Certificate42% $250 Drawing 28%

Effective Nonresponse Rate Reduction Methods Incentives reduce nonresponse rates higher effects with increasing incentives prepaid incentives more powerful than promised cash more powerful than in-kind incentives can reduce total survey costs

Sample Size Mailing cost Response Rate Cost per Complete 10,000 $14,000 $5 Incentive No Incentive $64,000 14%54% $10.00$11.85

2 nd Mailing 10,000 – 1,400 = 8,660 5% Response Rate Cost = $12,040

Mailing cost Response Rate Cost per Complete 18,60010,000 $26,040 $5 Incentive No Incentive $64,000 18%54% $14.23$11.85 Total Mailed To

Returned Incentives $1 $2 $5 Starbucks # People