Jaki S. McCarthy, Daniel G. Beckler, and Suzette M. Qualey Slide 1Slide Slide 1 International Conference on Establishment Surveys III Montreal June 18-21, 2007 United States Department of Agriculture National Agricultural Statistics Service An Analysis of the Relationship Between Survey Burden And Non-response: If we bother them more, are they less cooperative?

Common Perceptions of Burden  BURDEN = BAD!  Many components of burden – vary by person – many aren’t measurable  Burdens commonly believed to reduce cooperation: Length of interview or questionnaire Length of prior survey request Number of prior requests Prior burdensome contact Frequent survey requests Slide 1Slide Slide 2 International Conference on Establishment Surveys III Montreal June 18-21, 2007

Establishments may be particularly burdened How about agricultural operations?  Examined a 4 year period (Jan 2000 – Dec 2003)  Included all NASS national surveys (184 total possible survey contacts)  Mostly telephone (but included limited face-to-face and mail contacts)  Surveys collected a variety of data including crop and livestock inventory and production, farm economics, and farm labor  For this study, equate “contact” with “sampled for the survey” Slide 1Slide Slide 3 International Conference on Establishment Surveys III Montreal June 18-21, 2007

NASS Burden (over 4 years)  Total number of US agricultural operations: ~2.1 million (2002 U.S. Census of Agriculture)  Total number of operations contacted: 579,531 (27%)  Most common number of contacts: 1  Highest number of contacts for single operation: 103  72% of sampled operations (419,363) contacted 4 or less times in 4 year period Slide 1Slide Slide 4 International Conference on Establishment Surveys III Montreal June 18-21, 2007

Does Burden = Lower Response?  Examined effect of burden in 10 surveys at end of 4 year period  Compared burden measures for cooperators and refusals to the 10 surveys  Same operators not necessarily sampled for all 10 surveys.  Surveys were among NASS’ primary survey programs and included a variety of crops, livestock, agricultural labor, and economics surveys Slide 1Slide Slide 5 International Conference on Establishment Surveys III Montreal June 18-21, 2007

Prediction: higher number of prior survey contacts = more refusals Predicted Effect No Significant Difference Opposite Effect 5 surveys2 surveys3 surveys Largest Effect 8.2 vs 11.1 (2.9 surveys) Largest Effect 51.4 vs 48.3 (3.1 surveys) Slide 1Slide Slide 6 International Conference on Establishment Surveys III Montreal June 18-21, 2007

Prediction: longer previous contact = more refusals Predicted Effect No Significant Difference Opposite Effect 6 surveys2 surveys Largest Effect 12.5 vs (1.9 minutes) Largest Effect 23.3 vs (2.1 minutes) Slide 1Slide Slide 7 International Conference on Establishment Surveys III Montreal June 18-21, 2007

Prediction: longer total length of prior contacts = more refusals Predicted Effect No Significant Difference Opposite Effect 6 surveys2 surveys Largest Effect 61.6 vs (57.3 minutes) Largest Effect vs (37.9 minutes) Slide 1Slide Slide 8 International Conference on Establishment Surveys III Montreal June 18-21, 2007

Prediction: prior burdensome contact (ARMS) = more refusals Predicted Effect No Significant Difference Opposite Effect 3 surveys2 surveys5 surveys Largest Effect 35.1 vs (8.6 percent. points) Largest Effect 62.5 vs (19.1 percent. points) Slide 1Slide Slide 9 International Conference on Establishment Surveys III Montreal June 18-21, 2007

Prediction: fewer days since prior contact = more refusals Predicted Effect No Significant Difference Opposite Effect 3 surveys 4 surveys Largest Effect vs (20.7 days) Largest Effect vs (65.5 days) Slide 1Slide Slide 10 International Conference on Establishment Surveys III Montreal June 18-21, 2007

Does Burden = BAD?  In our set of contacts, increased “burden” does not universally reduce response  Effects are small in many cases  Effect in direction opposite predictions can be large  Analysis does not lead to obvious strategies to increase response  Caution: these results were not based on a controlled experiment Slide 1Slide Slide 11 International Conference on Establishment Surveys III Montreal June 18-21, 2007