1. Eurostat Statistical Data Editing and Imputation

2. Presented by Sander Scholtus Statistics Netherlands

3. Introduction Data arrive at a statistical institute... IDsize class number of employees turnover (x €1000) labour costs (x €1000) other costs (x €1000) total costs (x €1000) 0001large21349,827030,479 0002large364,933 0003medium421,462511,513 0004medium296,3018916,350 0005small4875,00098,000547,000645,000 0006small81,716175998 0007small061447153570

4. Introduction Data arrive at a statistical institute... –…containing errors and implausible values –…containing missing values To produce statistical output of sufficient quality, these data problems have to be treated –Statistical data editing deals with errors –Imputation deals with missing values

5. Statistical data editing Overview –Goals –Edit rules –Different editing methods and how to combine them –Modules in the handbook

6. Statistical data editing – goals Traditional goal of editing: –Detect and correct all errors in the collected data Problems: –Very labour-intensive –Very time-consuming –Highly inefficient: measurement error is not the only source of error in statistical output

7. Statistical data editing – goals Modern goals of editing: 1.To identify possible sources of errors so that the statistical process may be improved in the future. 2.To provide information about the quality of the data collected and published. 3.To detect and correct influential errors in the collected data. 4.If necessary, to provide complete and consistent micro-data. sources: Granquist (1997), EDIMBUS (2007)

8. Statistical data editing – edit rules

9. Examples of edit rules: –Turnover ≥ 0 (non-negativity edit, hard) –Profit = Turnover – Total costs (balance edit, hard) –IF (Size class = “Small”) THEN (0 ≤ Number of employees < 10) (conditional edit, soft) –IF (Economic activity = “Construction”) THEN (a < Turnover / Number of employees < b) (ratio edit, soft)

10. Statistical data editing – methods deductive editing selective editing not selected selected manual editing automatic editing macro- editing statistical microdata raw microdata

11. Statistical data editing – methods Deductive editing –Directed at systematic errors –Deterministic detection and amendment  if-then rules  algorithms –Examples:  unit of measurement errors (e.g. “4,000,000” instead of “4,000”)  sign errors (e.g. “–10” instead of “10”)  simple typing errors (e.g. “192” instead of “129”)  subject-matter specific errors

12. Statistical data editing – methods deductive editing selective editing not selected selected manual editing automatic editing macro- editing statistical microdata raw microdata

13. Statistical data editing – methods

14. deductive editing selective editing not selected selected manual editing automatic editing macro- editing statistical microdata raw microdata

15. Statistical data editing – methods Manual editing –Requires:  Human editors (subject-matter specialists)  Dedicated software (interactive editing)  Edit rules (hard and soft)  Editing instructions –Re-contacts with businesses are sometimes used –Important as a source for improvements in future rounds of a repeated survey

16. Statistical data editing – methods deductive editing selective editing not selected selected manual editing automatic editing macro- editing statistical microdata raw microdata

17. Statistical data editing – methods Automatic editing –Obtain consistent micro-data for non-influential records –Paradigm of Fellegi and Holt (1976): Data should be made consistent with the edit rules by changing the fewest possible (weighted) number of items.  Leads to error localisation as a mathematical optimisation problem  Imputation of new values as a separate step –Requires:  (Hard) edit rules  Dedicated software (e.g.: Banff by Statistics Canada; SLICE by Statistics Netherlands; R package editrules )

18. Statistical data editing – methods deductive editing selective editing not selected selected manual editing automatic editing macro- editing statistical microdata raw microdata

19. Statistical data editing – methods Macro-editing –Also known as output editing –Same purpose as selective editing –Uses data from all available records at once –Aggregate method:  Compute high-level aggregates  Check their plausibility  Drill down to suspicious lower-level aggregates  Eventually: Drill down to suspicious individual records  Feedback to manual editing –Graphical aids (scatter plots, etc.) to find outliers

20. Statistical data editing – modules Modules in the handbook: 1.Main theme module 2.Deductive editing 3.Selective editing 4.Automatic editing 5.Manual editing 6.Macro-editing 7.Editing administrative data 8.Editing for longitudinal data

21. Imputation Overview –Missing data –Imputation methods –Special topics –Modules in the handbook

22. Imputation – missing data Missing data may occur because of –Logical reasons  A particular question does not apply to a particular unit –Unit non-response  No data observed at all for a particular unit –Item non-response  Unit is not able to answer a particular question  Unit is not willing to answer a particular question –Editing  Originally observed value discarded during automatic editing

23. Imputation – missing data Imputation: filling in new (estimated) values for data items that are missing Commonly used for missing data due to item non-response and editing Obtain a completed micro-data file prior to estimation –Simplifies the estimation step –Prevents inconsistencies in the output

24. Imputation – methods Deductive imputation Model-based imputation Donor imputation Assumption: All observed values are correct –Imputation applied after error localisation

25. Imputation – methods Deductive imputation –Derive (rather than estimate) missing values from observed values based on  logical relations (edit rules)  substantive imputation rules –Can be very useful as a first imputation step IDturnover (sales) turnover (services) turnover (other) turnover (total) 100115410166 1002147 IDturnover (sales) turnover (services) turnover (other) turnover (total) 1001154102166 1002147 IDturnover (sales) turnover (services) turnover (other) turnover (total) 1001154102166 100214700

26. Imputation – methods Model-based imputation –Imputations based on a predictive model –Model fitted on the observed data, then used to impute the missing data

27. Imputation – methods

28. Model-based imputation –Choice of model depends on intended use of data  Estimating means and totals: mean or ratio imputation may be sufficient  General purpose micro-data: important to model relationships –Multivariate model-based imputation  Multivariate regression imputation (joint model for all variables)  Sequential regression / chained equations (separate model for each variable, conditional on the other variables)

29. Imputation – methods Donor imputation –Missing values imputed by ‘borrowing’ observed values from other (similar) units  Unit with observed value: donor  Unit with missing value: recipient –Hot deck: donor and recipient in the same data file

30. Imputation – methods Donor imputation –Special cases:  Random hot deck imputation Donor selected at random (within classes) Use auxiliary variables to define imputation classes  Nearest-neighbour imputation Donor selected with minimal distance to recipient Use auxiliary variables to define distance  Predictive mean matching Special case of nearest-neighbour imputation Distance based on predicted values from a regression model

31. Imputation – special topics Choice of method/model/auxiliary variables –General problem in multivariate analysis –Auxiliary variables should explain  the target variable(s)  the missing data mechanism –Compare model fit among item respondents  Can be misleading (“imputation bias”) –Simulation experiments with historical data

32. Imputation – special topics Imputation for longitudinal data –Repeated cross-sectional surveys –Panel studies Special imputation methods for longitudinal data –Last observation carried forward –Interpolation –Extrapolation –Little and Su method

33. Imputation – special topics Imputations are estimates –Imputed values should be flagged Variance estimation with imputed data –Variance likely to be underestimated when…  …imputations are treated as observed variables  …model predictions are imputed without a disturbance term  …single imputation is used –Alternative approach: Multiple imputation  Not often used in official statistics (yet)

34. Imputation – special topics Imputed values may be invalid/inconsistent –Examples:  Turnover = –100 (invalid)  Labour costs = 0, Number of employees = 15 (inconsistent) –Need not be a problem for estimating aggregates –Can be a problem if micro-data are distributed further Imputation under edit constraints –One-step method: constrained imputation model –Two-step method: imputation followed by data reconciliation

35. Imputation – modules Modules in the handbook: 1.Main theme module 2.Deductive imputation 3.Model-based imputation 4.Donor imputation 5.Imputation for longitudinal data 6.Little and Su method 7.Imputation under edit constraints

36. Thank you for your attention!

37. References EDIMBUS (2007), Recommended Practices for Editing and Imputation in Cross-Sectional Business Surveys. Fellegi, I.P. and D. Holt (1976), A Systematic Approach to Automatic Edit and Imputation. Journal of the American Statistical Association 71, pp. 17–35. Granquist, L. (1997), The New View on Editing. International Statistical Review 65, pp. 381–387.