1. Using Paradata to Track the Incidence of Respondent Substitution and its Effect on Survey Quality Carl Ramirez, U.S. GAO FedCASIC 2009

2. Paradata created by survey process: Name / title /role / contact information of the originally targeted respondent vs. Actual respondent of record Difference or Lack of Difference between:

3. Application Business or household surveys List frame surveys with pre-identified reporters or informants Identity of reporter or informant is significant to data collection

4. Purpose For survey managers & designers To monitor and analyze phenomenon of respondent substitution To understand link between substitution and data quality

5. Is Substitution Important? In longitudinal establishment surveys – yes Substitution can be good: Correction of misdirected survey requests Facilitators for multiple respondents necessary Replacement of absent or delaying reporter Or bad: Delegation to unqualified recipients Interception by gatekeepers Lack of commitment to the task

6. Research Response Models for Business Surveys Edwards & Cantor (1991) Tomaskovic-Devey (1994) Willimack et. al (1999-2001) Tuttle (2008) Proxy Reporting

7. Survey Datasets 1.Army spare parts inventory Respondent: managers of sampled parts 2.Head Start Grantees Respondent: program officials 3.Healthy Marriage Grantees Respondent: program officials 4.Protégé Firms Respondent: Corporate officers

8. Substitution Rates 1.Army spare parts inventory: 3% 2.Head Start Grantees: 34% 3.Healthy Marriage Grantees: 56% 4.Protégé Firms: 31%

9. Nature of Substitution Delegation: typically from higher- level titles to subordinate titles 1.Head Start Titles with term director: 88% of originally targeted 32% of substitutes

10. Nature of Substitution 2. Healthy Marriage Grantees - Executive Directors: 28% vs. 3% (original vs. substitute) - Substitutes more often had subordinate title terms: assistant, deputy, associate

11. Nature of Substitution 3. Protégé Firms - Originals: more often President, CEO - Substitutes: more often Vice President, Chief Financial Officer

12. Nature of Substitution Specialization: typically from general executive titles to more specialized titles Analyst, Project Director of ____, Fatherhood Advocate, Chief Procurement Officer, QA Director

13. Quality Measures Item nonresponse Nonsubstantive answers Volume of open-ended text provided Navigational & other response errors

14. Answer Text Volume 1.Head Start Grantees Original > Substitute in 5 of 6 qns. 2.Healthy Marriage Grantees Original > Substitute in all 5 qns. 3.Protégé Firms Original > Substitute in 1 of 4 qns

15. Response Errors 1.Head Start Grantees Original > Substitute skip errors in 2 of 4 qns. 2.Healthy Marriage Grantees Original > Substitute in satisficing in 1 of 1 qns. (estimating 1 st of month in date qn.) 3.Protégé Firms Original > Substitute skip errors in 2 of 4 qns

16. Item Nonresponse 1.Head Start Grantees Original > Substitute in 23 of 30 qns. 2.Healthy Marriage Grantees Original > Substitute in 10 of 34 qns. 3.Protégé Firms Original > Substitute in 5 of 7 qns

17. Nonsubstantive Answer 1.Head Start Grantees Original > Substitute in 3 of 19 qns. 2.Healthy Marriage Grantees No data 3.Protégé Firms Original > Substitute in 2 of 6 qns.

18. Further Issues Multiple Respondents Population-specific characteristics Manipulating substitution rates

19. Thank you! The views and statements expressed are the authors own and do not necessarly reflect official policies of the U.S. Government Accountability Office. Questions? ramirezc@gao.govramirezc@gao.gov