Like – But Oh, How Different: The effect of different questionnaire formats in the 2005 Census of Agriculture Content Test. Jaki S. McCarthy National Agricultural Statistics Service, United States Department of Agriculture

Census of Agriculture conducted every 5 years Includes all known operations with agriculture potential of any type (~3mil) Information collected on acreage, production, sales and expenses, production practices, and demographics Primarily mail out/mail back form, with follow up mailings, limited CATI and interview follow up Reporting is mandatory

Prior to Census, Census Content Test is Conducted Test used to finalize forms, test data collection and processing procedures Small scale mail out/mail back with follow up reminder Follow up cognitive interviews conducted with subset of respondents Reporting was not mandatory

2005 Census Content Test (CCT) Two forms tested (long and short) In 2007 Census, short form will be targeted to specific types of operations For CCT, split sample of all farms used for analysis

Both CCT Forms collected the same information Differences in: – Format –Layout –Number of pages

Why use a short form? May increase response rates May be faster and/or easier to complete Costs less to print and mail

Why use a long form? More specific examples and instructions can be included Explicit screening questions and routing instructions can be incorporated Common items are pre-printed Flexibility with layout can allow related items to be placed together

Differences Between Forms LONG FORM 24 pages Acres operated by land type collected immediately after acres operated by ownership Separate sections for different commodities; Most common items pre-printed, others listed Sales collected separately from production in list Government programs listed SHORT FORM 12 pages Acres operated by land type collected several pages after acres operated by ownership All crops and livestock collected in open write-in tables; Only a few items included as examples Value of sales collected with production in open tables Government program info collected in abbrev. tables Qs on production contracts and irrigated pasture omitted

Short Form, Field Crops and Hay Table

Long Form, Field Crop Section

Short and Long Forms Split sample used to send short and long forms to all types of operations Response rate for each form tracked Content was the same, so comparisons of length were not confounded with content Data compared across forms Information from cognitive interviews compared across forms

Less pages ≠ Higher response FormTotal MailedReceiptsResponse Rate % (Weighted*) Short (52.73) Long *short form response weighted for comparability to long form subgroup sizes

Long Form: page 2 Short Form: page 2 Long Form: Facing on Page 3 Short Form: 3 pages later on page 5

If you want people to report the same number twice, don’t ask them 3 pages apart Form% Total Land 1 missing (n) % Total Land 2 missing (n) % Total land 1 <> Total land 2 (n) Short6.3% (364) 14.6% (841) 39.61% (2180) Long5.1% (358) 11.4% (790) 26.54% (1779) Problems reporting Total Acres

Quantity Unit preprinted Calculated yields = 1 FormCornSoy Short11%13% Long2% If you want bushels, ask for bushels

Calculated yields = 1 FormCornSoy Short11%13% Long2% Quantity Unit must be written by respondent

Long Form Short Form

If you want something reported, ask for it explicitly Total reporting horses in prior census Horses reported in prior census and in CCT % (n) Horses reported in prior census and 0 in CCT % (n) Short % (356) 67.9% (752) Long % (1056) 25.0% (353)

Other problems Short Form: Fewer gov. payments reported Short Form: More manual processing required (extra sheets, uncodable responses, more than one number entered in a cell) Short Form: Slightly more omitted agriculture reported in cognitive interviews (9.9% vs 7.4%) Long Form: Self reported completion times slightly longer (44.46 vs minutes)

Changes based on these findings Added additional types of operations to exclude from short form (e.g. horses, specialties, etc.) Use of more long form question formats on short form (e.g. government payments) Use of additional edit checks on short form

Conclusions Decreasing # of pages does not increase response rate More data problems with the short form, particularly underreporting Forms may collect the same data, but their performance is NOT equivalent— Like, but different

Plans for use of Short Form Short Form will be used ONLY with limited set of operations (e.g. operations known to have 4 or less commodities; operations without specialty commodities; operations not previously reporting horses, goats, sheep, bees, etc) With over 3 million mailing addresses, cost savings for lighter mailing can be substantial

YES, it was the mountain Echo, Solitary, clear, profound, Answering to the shouting Cuckoo, Giving to her sound for sound! Unsolicited reply To a babbling wanderer sent; Like her ordinary cry, Like--but oh, how different! --W. Wordsworth