The Implications of Misreporting for Longitudinal Studies of SNAP Preliminary work. Do not cite without permission. The Implications of Misreporting for Longitudinal Studies of SNAP Brian Stacy Erik Scherpf The findings and conclusions in this preliminary publication have not been formally disseminated by the USDA and should not be construed to represent any agency determination or policy. This research was supported in part by the intramural research program of the USDA, Economic Research Service.

Introduction Researchers often rely on survey datasets to examine SNAP Several studies have noted misreporting of SNAP participation Many researchers use longitudinal estimators of SNAP To study dynamics of SNAP participation To possibly help with selection bias However, longitudinal methods can exacerbate bias from misreporting

Objectives & Findings This paper examines the misreporting of SNAP on causal effects Particularly focused on longitudinal estimators Link SNAP Administrative Data to CPS ASEC survey records Create panel dataset for individuals Three main findings: Find evidence of severe errors in changes in reported SNAP participation Of those switching onto SNAP from one period to next based on surveys, 86% are false Application: Effects of SNAP on work effort Find severe bias in longitudinal estimators Find evidence that some methods to correct estimators for misreporting perform well Conditional distribution method (CDM) works well for first-difference OLS estimator CDM can be applied by researchers without access to admin records

Background Meyer, Mittag, & Goerge (2018) find substantial under-reporting of SNAP in CPS ASEC survey in cross-section Close to 50% of SNAP participants don’t report receipt in CPS ASEC Little literature on errors in changes in reported SNAP participation over time, which identify many longitudinal estimators of SNAP effects Paper is similar to Bound & Krueger (1991); Bound, Brown, Duncan, Rogers (1994); Freeman (1984) These papers examine effects on longitudinal estimators of misreporting of earnings and union participation

Data Combine SNAP administrative records on SNAP receipt to CPS ASEC survey We assume admin reports are “true”, mismatch with survey is false report Links based on PIK created by Census Bureau staff Can create two-period panel datasets using longitudinal structure of CPS ASEC Can directly assess errors in changes in reported SNAP participation Create linkages for nine anonymous states from 2005-2015 Unfortunately, not all states have data available for all years. From 2010-2013, states make up 32% of individual SNAP caseload In 2005, when fewest states were available, made up 5% of caseload

Results on Misreporting in Cross-Section Using our linked SNAP admin records and CPS ASEC records, examine misreporting Cross-sectional Misreporting: Find under-reporting rate of 43.9% Find over-reporting rate of 2.9% Find evidence that misreporting related to individual characteristics Those who work are more likely to under-report SNAP Females more likely to under-report Those with more education more likely to under-report Meyer et al. (2018) found higher income individuals more likely to under-report

Results on Misreporting Longitudinally Using linked data, also examine changes in reported SNAP participation Errors in changes in reported SNAP participation: Four groups: Switch off SNAP: 86% of survey based changes are false Most of these people were on SNAP in both periods or off SNAP both periods On SNAP – Both Periods: 27.5% of survey reports false Even though over-reporting rare, off SNAP group makes up 88.5% of records Off SNAP – Both Periods: 4.7% of survey reports false Switch on SNAP: 87% of survey based changes are false Again, most were either off SNAP or on SNAP both periods Find errors in changes related to characteristics Employment, age, race

Empirical Application: Effects of SNAP on Work  

Cross Sectional w/ OLS: Reported SNAP -0.291*** -0.222*** -13.82*** Table 4. Regression Estimates of Effects of SNAP on Work and Hours Worked using Reported SNAP Measure from CPS ASEC versus Admin Verified SNAP Measure. Results from 16 Separate Regressions.   (1) (2) (3) (4) Estimated Effects Worked Past Year Hours Worked Cross Sectional w/ OLS: Reported SNAP -0.291*** -0.222*** -13.82*** -9.912*** (0.007) (0.257) (0.267) Admin Verified SNAP -0.286*** -0.221*** -13.60*** -9.556*** (0.006) (0.238) (0.251) First Differences w/ OLS: -0.040*** -0.039*** -2.373*** -2.330*** (0.010) (0.413) (0.419) 0.050*** 0.048*** 1.769*** 1.710*** (0.015) (0.607) (0.612) Covariates NO YES Covariates include: age, gender, race, education, marital status, unemployment rate. Robust standard errors in parenthesis. *** p<0.01, ** p<0.05, * p<0.1

Solutions for Researchers Find Conditional Distribution Methods work well (Mittag,2018) Can be used by researchers using only public use data No admin data link required Two-Step approach: Researchers with access to admin data publish estimates of conditional distribution of “true” SNAP participation conditional on covariates in public use data Researchers using public use data then impute SNAP participation based on these covariates Approach similar to multiple imputation Key is that imputed SNAP participation has same covariance structure as “true” SNAP participation measure Assumptions: Conditional distribution must be same for linked data and public use data Parametric model used to estimate conditional distribution must be properly specified

*** p<0.01, ** p<0.05, * p<0.1 Table 5. Regression Estimates of First Differences in Work and Hours Worked on First Differences in SNAP on using OLS with Reported SNAP Measure, OLS with Admin Verified Measure, and the Mittag (2018) Conditional Distribution Method   (1) (2) (3) (4) Estimated Effects Worked Past Year Hours Worked First Differences w/ OLS: Reported SNAP -0.040*** -0.039*** -2.373*** -2.330*** (0.010) (0.413) (0.419) Admin Verified SNAP 0.050*** 0.048*** 1.769*** 1.710*** (0.015) (0.607) (0.612) Conditional Distribution Method 0.054 *** 0.047 *** 1.947*** 1.698*** (0.006) (0.003) (0.288) (0.126) Covariates NO YES Observations 36,000 Bootstrap standard errors in parenthesis from 100 bootstrap replications. In each bootstrap replication, both the multinomial logit estimation and regression estimation steps are performed. Covariates include controls for age, education, race, marital status, and the county unemployment rate. *** p<0.01, ** p<0.05, * p<0.1

Conclusions Many researcher make use of survey data and apply longitudinal methods CPS ASEC supplement frequently used and has substantial misreporting Find errors in changes in SNAP participation worse than cross-section We characterize bias for cross-sectional and longitudinal estimators Find severe bias in longitudinal estimator of effects of SNAP on Work Solutions: Conditional Distribution Methods seem promising Researchers with access to admin linkages may wish to publish conditional distribution estimates

Questions?

Misreporting and Bias  

Non-random Misreporting and Bias