A modeler’s perspective on Total Survey Error Rod Little

Outline Total Survey Error – strengths and weaknesses, according to Groves and Lyberg A modeler’s view of TSE An application: multiple imputation for regression involving covariates with measurement error, including data from a calibration sample Epidemiological, but implications for survey practice Involving calibration samples, heteroscedastic measurement errors Compare with classical calibration methods Modeler’s view of TSE

My philosophy: Calibrated Bayes Bayes for inference, frequentist for model development and assessment – seek inferences that are “frequentist calibrated” In surveys this often leads to “weak, robust” models with “reference” prior distributions” “model-based, design-assisted inference” (Little 2012) Bayes inference is optimal under well-specified models Frequentist calibration creates resistance to “bad models” E.g. “design-consistency” forces models that account for survey design unlike e.g. models in Hansen, Madow and Tepping (1983) Frequentist simulations suggest this approach can yield better repeated-sampling properties than design-based approaches Bayes propagates error in estimating parameters Modeler’s view of TSE

Total Survey Error (TSE) Groves and Lyberg (GL, 2010) TSE paradigm is the conceptual foundation of the field of survey methodology. Quality properties of survey statistics are functions of essential survey conditions that are independent of the sample design. More successful as an intellectual framework than a unified statistical model of error properties of survey statistics. More importation of modeling perspectives from other disciplines could enrich the paradigm. In this case, biostatistics and epidemiology Modeler’s view of TSE

Strengths of TSE (GL) Explicit attention to the decomposition of errors Separation of phenomena affecting statistics in various ways Success in forming the conceptual basis of the field of survey methodology, pointing the direction for new research. Modeler’s view of TSE

Current weaknesses of TSE (GL) Key quality concepts are not included (notably those of user) Quantitative measurement of many components burdensome and lagging Has not led to enriched error measurement in practical surveys Assumptions required for some estimators of error terms are frequently not true Mismatch between existing error models and theoretical causal models of the error mechanisms Misplaced focus on descriptive statistics, and failure to integrate error models developed in other fields Modeler’s view of TSE

Current weaknesses of TSE (GL) Key quality concepts are not included (notably those of user) Quantitative measurement of many components burdensome and lagging Has not led to enriched error measurement in practical surveys Assumptions required for some estimators of error terms are frequently not true Mismatch between existing error models and theoretical causal models of the error mechanisms Misplaced focus on descriptive statistics, and failure to integrate error models developed in other fields Unified modeling addresses these points Modeler’s view of TSE

TSE Components Linked to Steps in Measurement and Representational Inference Process (Groves et al. 2004) Representation Measurement Inferential Population Construct Validity Target Population Coverage Error Measurement Sampling Frame Measurement Error Sampling Error Response Sample Processing Error Nonresponse Error Edited Data Respondents Errors of observation Errors of nonobservation Survey Statistic Modeler’s view of TSE

Commentary 1 Dual inferential approaches – model-based for measurement, design-based for analysis – inhibits integration of the two streams into a unified analysis “The isolation of survey statisticians and methodologists from the mainstream of social statistics has, in our opinion, retarded the importation of model-based approaches to many of the error components in the total survey error format.” (GL) The great disappointment regarding the TSE perspective is that it has not led to routine fuller measurement of the statistical error properties of survey statistics. While official statisticians and much of social science have accepted the probability sampling paradigm and routinely provide estimates of sampling variance, there is little evidence that the current practice of surveys in the early 21st century measures anything more than sampling variance routinely. Modeler’s view of TSE

Commentary 1 Dual inferential approaches – model for measurement, design for analysis – inhibits integration of the two streams into a unified analysis “There are exceptions worth noting. The tendency for some continuing surveys to develop error or quality profiles is encouraging (Kalton, Winglee, and Jabine, 1998; Kalton, Winglee, Krawchuk, and Levine, 2000; Lynn, 2003). These profiles contain the then-current understanding of statistical error properties of the key statistics produced by the survey. Through these quality profiles, surveys with rich methodological traditions produce well-documented sets of study results auxiliary to the publications of findings. None of the quality profiles have attempted full measurement of all known errors for a particular statistic.” (GL) Modeler’s view of TSE

Commentary 2 The standard decomposition of RMSE into components of bias and variance in the TSE approach implies a particular model that may not be realistic, and is restricted to simple statistics like means “3.6 Assumptions Patently Wrong for Large Classes of Statistics. Many of the error model assumptions are wrong most of the time. For example, the Kish (1962) linear model for interviewer effects assume that the response deviations are random effects of the interviewer and respondent, uncorrelated to the true value for the respondent. However, for example, reporting of drug usage has been found to have an error structure highly related to the true value.” (GL) “3.7 Mismatch between Error Models and Theoretical Causal Models of Error. The existing survey models are specified as variance components models devoid of the causes of the error source itself… Missing in the history of the TSE formulation is the partnership between scientists who study the causes of the behavior producing the statistical error and the statistical models used to describe them.” (GL) Modeler’s view of TSE

Commentary 2 The standard decomposition of RMSE into components of bias and variance in the TSE approach implies a particular model that may not be realistic, and is restricted to simple statistics like means “3.8 Misplaced Focus on Descriptive Statistics. Hansen, Hurwitz, and Pritzker (1964); Hansen, Hurwitz, and Bershad (1961), Biemer and Stokes (1991), and Groves (1989) all describe error models for sample means and/or estimates of population totals… survey data in our age are used only briefly for estimates of means and totals. Most analysts are interested more in subclass differences, order statistics, analytic statistics and a whole host of model-based parameter estimates.” (GL) A unified model for sampling and nonsampling error makes assumptions explicit, and allows inference for parameters other than means Modeler’s view of TSE

TSE as a missing data problem (Rubin, 1974) Z A X Y1 Y2 …YJ D1 … DK Errors of observation concern columns Experimental Units ? ? √ √ × …× ? … ? ? ? √ × × …√ ? … ? Sample Respondents √ √ × √ × …× ? … ? √ √ × × × …√ ? … ? Unit Nonrespondents √ √ × × × …× × …× Errors of non-observation concern rows Nonsampled Units √ × × × × …× × …× Z = frame/design variables, A = meta-data (e.g. mode) X = unobserved true, Yj = observed for mode j Dk=kth variable not subject to major measurement error √ = observed, × = missing, ? = observed or missing Modeler’s view of TSE

Application: measurement error in epidemiology Many variables in epidemiology are measured with error (dietary intake, biomarkers, …) Measurement error attenuates effect of variables, distorts inferences for other variables E.g. effect of dietary intake on cancer Existing methods (e.g. regression calibration) assume measurement variance is constant, but this is often a poor assumption Proposed approach: multiple imputation under a Bayesian model with non-constant variance (Guo and Little 2011, Guo Little and McConnell, 2011) Modeler’s view of TSE

Data for two vitamins Modeler’s view of TSE

Measurement Error Model This model links unobserved covariate X with error-prone measurement Y, considering potentially nonlinear mean functions and heteroscedastic measurement error with , the function g to model heteroscedasticity. specifically, we assume that Modeler’s view of TSE

Estimates of for eight analytes, linear model. Residual Regression ML Bayes Estimate Post. Mean 95% HPD Gamma tocopherol 0.56 (0.51,0.61) Lutein 0.61 0.63 (0.58,0.68) Alpha tocopherol 0.62 (0.57,0.67) Delta tocopherol 0.65 (0.53,0.77) Beta Cryptoxanthin (BC) 0.64 (0.58,0.71) Lycopene 0.70 0.72 0.71 (0.67,0.76) Retinol 0.68 0.67 (0.63,0.72) Carotene 0.77 (0.67,0.77) Modeler’s view of TSE

Regression on Covariates with measurement error (a) External Calibration design (b) Internal Calibration design D X Y D X Y Calibration sample D measured in calibration sample Main sample X: covariate of interest but unobserved Y: observed error-prone measurement related to X D: response variable, interest in regression of D on X (more generally D can include other covariates). Modeler’s view of TSE

Analysis Model This model links unobserved covariate X with outcome D. For simplicity we assume the model where , although more generally nonlinear relationships between Y and X can be modeled. Our aim is to estimate the unknown regression parameters, taking into account the measurement error in X. Modeler’s view of TSE

(B) Non-differential measurement error (NDME) assumption X Y Calibration Main Study (a) External calibration design D X Y Calibration Main Study (b) Internal calibration design Non-differential measurement error: D is independent of Y given X With external calibration design, assumption is needed to identify parameters With internal calibration design, assumption is not needed but improves efficiency if assumed and true Modeler’s view of TSE

Conventional Calibration (CA) fits an appropriate curve to the calibration data estimates the true value of X by inverting the fitted calibration curve (usually assumes a linear association) Regresses D on calibration estimates of X X Y Modeler’s view of TSE

Regression Calibration (RC) Estimates the regression of X on Y using the calibration data Replaces the unknown values of X in main study with predictions Simple and easy to be applied (Carroll and Stefanski,1990) Standard errors: Asymptotic formula or bootstrap both data sources Modeler’s view of TSE

“Efficient” Regression Calibration (ERC) When the internal calibration data is available, direct estimates of the regression D on X are available from the calibration sample These can be combined with the RC estimates, weighting the two estimates by their precision Spiegelman et al. (2001) call this efficient regression calibration Modeler’s view of TSE

Multiple Imputation (MI) Multiply impute all the values of X using draws from their predictive distribution given the observed data. We develop MI methods based on a fully Bayes model and measurement error model Main study model prior distribution Modeler’s view of TSE 24

Comparisons with constant measurement error variance Freedman et al. (2008) evaluate the performance of CA, RC, ERC and MI for the case of internal calibration data. CA biased, ERC better than RC, MI But: ERC assumes non-differential measurement error, and MI based on a model that does not make this assumption This accounts for superiority of ERC A limitation of their work is that it assumes the variance of the measurement errors is constant. As discussed, in many real applications, the variance of the measurement error increases with the underlying true value. We compare methods when measurement variance is not constant Modeler’s view of TSE

Weighted Regression Calibration (WRC) An alternative to RC, taking into account heteroscedastic measurement error. We reformulate the measurement error model as estimate λ as the slope of a simple regression of logarithm of the squared residuals of the regression on X on Y on the logarithm of the squared Y using the calibration data. estimating and by weighted least squares. substituting unknown values X in main study with estimates, Modeler’s view of TSE

Multiple Imputation (MI) MI is applied with a measurement error model that incorporates non-constant variance of form . (a) Full posterior distribution requires Metropolis step for draws of (b) Approximation based on weighted least squares avoids Metropolis step Modeler’s view of TSE 27

Prior distributions Noninformative prior distributions for the marginal distribution of X and the parameters. Specifically, we assumed where the prior distribution of X is normal with mean 0 and variance 1000, and Modeler’s view of TSE

Simulation Study Factors varied: study design: external calibration and internal calibration measurement error size outcome-covariate relationship Twelve simulation scenarios were generated by combining the following choices of parameters: analysis model: measurement error model: Main study sample size = 400, calibration data sample size = 80 Modeler’s view of TSE 29

Table 1. Empirical bias *1000 of estimators of γx with internal calibration data based on 500 simulations, when measurement error is heteroscedastic. (Empirical standard deviation *1000 in parenthesis). Modeler’s view of TSE

Table 2. Coverage of 95% confidence interval of the estimator of γx with the internal validation calibration data based on 500 simulations. Modeler’s view of TSE

Conclusions Conventional approach is very biased and the estimate is attenuated when measurement error increase. Regression calibration works poorly in presence of heteroscedastic measurement error. Weighted regression calibration is better, but still biased with below nominal coverage Multiple imputation yields satisfactory results with small biases and good coverage. Similar findings for external calibration, under NDME Modeler’s view of TSE

Implications for survey research Combining data from measurement error experiments and main survey yields better inferences Survey applications are more complex: Including other covariates is an easy extension Include complex design features in model, or apply design-based analysis to multiply-imputed data Relax normal assumptions Models needed for categorical data Bayesian prediction framework incorporates sample design and measurement features seamlessly Already starting to happen … Modeler’s view of TSE

A Method of Correcting for Misreporting Applied to the Food Stamp Program Nikolas Mittag (U.S. Census Bureau Dissertation Fellow), University of Chicago March 28, 2013 Using administrative and survey data I show that survey misreporting leads to biases in common statistical analyses. Standard corrections for measurement error cannot remove these biases. I develop a method to obtain consistent estimates by combining parameter estimates from the linked data with publicly available data… Administrative data on SNAP receipt and amounts linked to American Community Survey data from New York State show that survey data can misrepresent the program in important ways… The conditional density method I describe recovers the correct estimates using public use data only… Extrapolation to the entire U.S. yields substantive differences to survey data and reduces deviations from official aggregates by a factor of 4 to 8 compared to survey aggregates. Modeler’s view of TSE

References Groves, R.M. and Lyberg, L. (2010). Total Survey Error: Past, Present, and Future. POQ, 74 (5): 849-879. Guo, Y. & Little, R.J. (2011). Regression Analysis Involving Covariates with Heteroscedastic Measurement Error. Statistics in Medicine, 30, 18, 2278–2294. Guo, Y., Little, R.J. and McConnell, D.S. (2011). On Using Summary Statistics from an External Calibration Sample to Correct for Covariate Measurement Error. Epidemiology, 23(1), 165-174. Hansen, M.H., Madow, W.G. & Tepping, B.J. (1983), “An evaluation of model-dependent and probability-sampling inferences in sample surveys” (with discussion), JASA 78, 776-793. Little, R.J. (2012). Calibrated Bayes: an Alternative Inferential Paradigm for Official Statistics (with discussion and rejoinder). J. Official Statist., 28, 3, 309-372. Rubin, D. B. (1974), "Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies," J. Educ. Psych., 66, 688-701. Modeler’s view of TSE