Slide 1 Incentives in Surveys with Farmers Third International Conference on Establishment Surveys Montreal, Quebec, Canada June 19, 2007 Slide Kathy Ott and Dan Beckler presented by Norman Bennett USDA, National Agricultural Statistics Service

Background Information - NASS  The National Agricultural Statistics Service (NASS) surveys Agricultural Operations (farmers/ranchers)  Collects information from the US agricultural sector on acreage, production, yield, economics, labor, etc.  Conducts over 400 surveys per year  Technically, these are establishment surveys, but have some similarities to household surveys Slide 2

Background Information - ARMS  Agricultural Resource Management Survey (ARMS) is a long data collection, typically conducted by face-to-face interview  Asks detailed information on what is produced, economic items (assets, debt, income, expenses), and operator characteristics  Average interview time = 90 minutes  Typical national response rate = 60 to 65%  Very complex survey design with multiple versions. For this project, just two questionnaire versions used Slide 3

Incentive Use Goals for ARMS: Raise overall response rate Contain data collection costs “Shift” to mail completion  Two Incentive Studies completed at NASS; both on the ARMS Mail-out/Mail-back Study 2005 Face to Face Interview Study Slide 4

2004 Mail-out/Mail-back Study  2004 Reference Year, conducted early 2005  “Core” Questionnaire Version  Administered in 15 states, sample: ≈16,000  Mail-out/Mail-back data collection with face-to-face interview follow-up for all nonrespondents  Pre-Survey Letter  First Questionnaire Mailing  Postcard Reminder/Thank You  Second Questionnaire  Face-to-face Interview follow-up on all mail nonrespondents Slide 6

2004 Mail-out/Mail-back Study  Used a “monetary” incentive in the form of a $20 ATM card  Administered by:  1 st class and priority mail  Prepaid and promised  Treatment Groups receiving incentives included cover letters that:  Explained the incentive as a “Thank You”  Described uniqueness of ARMS  Justified incentive by overall cost-savings to government Slide 7

ATM Cards  Cards loaded with $20 plus $4 for transaction fees  ATM Cards came pre-activated  ATM Card was affixed to an instruction sheet Slide 8

Treatment Group 1 st Questionnaire Mailing 2 nd Questionnaire Mailing 1 (Control) 1 st Class, no incentive 2 (1 st, Prepaid) 1 st Class, $20 ATM1 st Class, no incentive 3 (Priority) Priority, no incentive 4 (Priority, Prepaid) Priority, $20 ATMPriority, no incentive 5 (1 st, Promised) 1 st Class, promise $20 ATM  All groups also received a pre-survey letter, a post-card reminder, and face- to-face follow-up 2004 Treatment Groups  Slide 9

2004 Response Rate Results Treatment GroupNMail Returns Face-to-Face Completes Overall Completes 1 (Control)1, %33.3%63.4% 2 (1 st, Prepaid)1, %29.6%70.4% 3 (Priority)1, %31.8%64.7% 4 (Priority, Prepaid)1, %28.5%72.4% 5 (1 st, Promised)1, %31.2%68.4% 1,3 1,3,5 1,3 1  Red numbers indicate Treatment Groups to which this percent was significantly greater (at α = 0.05 from t-tests with Bonferroni adjustment). Slide 10

2004 ATM Card Usage Treatment Group Mail Respondents Overall Respondents Overall Nonrespondents Overall Card Recipients 2 (1 st, Prepaid)60.4%41.1%4.7%30.3% 4 (Priority, Prepaid)58.9%40.8%5.4%31.0% 5 (1 st, Promised)71.4%61.4%3.1%61.3%  These people initially returned questionnaires which were later deemed inadequately completed.  Slide 11

2004 Overall Costs Treatment Group Postage & NPC Costs All ATM Costs Face- to- Face Costs Total Average per Sample Average per Complete 1 (Control)$11,520-0-$176,291$187,811$96.41$ (1 st, Prepaid) $13,128$14,845$154,154$182,127$93.83$ (Priority)$21,722-0-$170,951$192,673$99.57$ (Priority, Prepaid) $23,250$15,342$148,356$186,948$95.77$ (1 st, Promised) $13,939$20,486$160,475$194,900$100.15$ Face-to-face follow-up costs. Includes all ATM/POS withdrawals, all fees, and administrative costs. Includes postage, printing, and NPC administrative fees. Slide 12

Summary of 2004 Mail-out/Mail-back Study  Offering $20 ATM card incentives for the ARMS Core improved both mail and overall response rates  The incentives reduced overall costs, the average cost per sample and the average cost per complete.  The prepaid incentives performed better than the promised.  Priority mail further increased response rates (albeit non- significantly) when combined with the incentive but alone was ineffective. Slide 13

What to do next?  Implemented use of prepaid incentives for entire Core version mail-out/mail-back sample (approximately 16,000 records)  For 2005, extended incentive research for face-to-face interview component of the data collection using the Cost and Returns Report (long) version of the ARMS questionnaire Slide 14

Slide Face-to-Face Incentive Study  “Long” (32 pages) questionnaire version only  All 48 states included  Face-to-face interview only, no mail data collection  Presurvey letter was mailed with the incentive, followed by face-to- face interview data collection  Stimuli:  Prepaid Monetary Incentive ($20 ATM card)  “Prepaid” Non-monetary Incentive (Wall clock)  Promised Individual Farm Analysis (IFA)

Treatment Group First ContactInterview Contact Pre-survey letter, plus:Face-to-face interview, plus: 1 (Control) No incentiveNo Incentive 2 (ATM Card) $20 ATM CardNo Incentive 3 (IFA) Mention IFAPromise IFA 4 (Clock) Non-monetary clockNo Incentive 5 (ATM Card, IFA) $20 ATM Card Mention IFA Promise IFA 2005 Treatment Groups Slide 16

Slide 20 Summary of 2005 Face-to-Face Study  As implemented, Prepaid Monetary, “Prepaid” Non-monetary, and Individual Farm Analyses Incentives:  Had no significant impact on response rates, although response rates did go up slightly  Increased costs  It’s possible that a different method of distributing a non-monetary incentive or a different non-monetary incentive all together would have been more effective

Slide 21 Recommendations  Continue the use of monetary incentives for ARMS “Core” mail- out/mail-back data collection methodology and include a control group at least every other year  Do not use prepaid monetary or “prepaid” non-monetary incentives for face-to-face data collections as a means of increasing response rates  Track the card cashing rates of respondents and nonrespondents every year to maintain cost effectiveness of incentive methodology for mail data collection – part of studying the long-term effects on NASS’ entire survey program.

Contacts: Dan Beckler: Kathy Ott: Norman Bennett: Slide 23

References: Beckler, D. K. Ott, P. Horvath. (2005). Indirect Monetary Incentives for the 2004 ARMS Phase III Core. Research and Development Division Research Report RDD National Agricultural Statistics Service, United States Department of Agriculture. Beckler, D. K. Ott (2005). Indirect Monetary Incentives With a Complex Agricultural Establishment Survey. Proceedings of the Joint Statistical Meetings of the American Statistical Association. McCarthy, J.S. D. Beckler, K. Ott (2006). The Effect of Incentives on Response in 2005 ARMS Phase III Interviews. Research and Development Division Research Report RDD National Agricultural Statistics Service, United States Department of Agriculture. Slide 24