1. Eurostat On the use of data mining for imputation Pilar Rey del Castillo, EUROSTAT

2. Outline Imputations to solve non-response in surveys; new problems for mass imputations State of the art: model-based imputations => MI Introduce data mining methods (for continuous data) Compare results in a simulation exercise following different criteria Raise questions on mass imputation (should data mining methods be considered?) 2 Eurostat

3. Imputations to solve non-response Replace each missing-value with an estimate Current problems in sample surveys –Small area estimation-> provide values for non- sampled units –Statistical matching-> provide joint statistical information based on 2 or more sources  A complete data set providing a basis for consistent analysis?...  Mass imputation as possible solution Model-based procedures making inferences based on the posterior distribution  Multiple Imputation (MI) (suited for computing variances) 3 Eurostat

4. Multiple Imputation 4 Eurostat ImputationAnalysisCombination Incomplete data Imputed data Statistics Combined statistic

5. Simulation exercise EU-SILC 2009: microdata on income, poverty, social exclusion and living conditions (Spain, Austria) Wages numerical variable to be imputed; Covariates (15) gender, age, country of birth, marital status, region, degree urbanisation of residential area, economic activity, highest level education, managerial position, occupation, temporary job, part-time job, hours usually worked per week, years education & years in main job Methods to be compared: –Least Median Squared Error Regressor (LMS) –M5P algorithm (M5P) –Multilayer Perceptron Regressor (MLP) –Radial Basis Function (RBF) –Regression (REG) –Predictive Mean Matching (PMM) 5 Eurostat

6. Least Median Squared Error Regressor (LMS) Outliers affect classical LS linear regression: squared distance accentuates influence of points far away from regression line More robust: minimise median of squares of differences from regression line Standard linear regression, solution with smallest median-squared errors 6 Eurostat

7. M5P algorithm (M5P) Decision tree: supervised classifier with uses a tree- like graph or model of decisions and their possible consequences (decision nodes, leaves…) Model tree: for continuous variables, with a linear regression model at each leaf Reconstruction of Quinlan's algorithms 7 Eurostat

8. Multilayer Perceptron Regressor (MLP) Neural networks based on structure of the brain; learning by adjusting connections MLP Feed forward network 1 hidden layer Delta rule as learning algorithm  w ij = -   E(w ij )/  w ij Logistic function as transfer function f(x) = 1/(1+e -x ) Output layer: 1 node with linear activation 8 Eurostat

9. Radial Basis Function (RBF) Neural network similar to MLP Differing in way hidden layer performs computations Activation for an input depends on distance to hidden unit Parameters to be learnt weights + centres 9 Eurostat

10. Regression (REG) Regression forecast for each input of covariate variables from regression estimated using training set Categorical treated by constructing appropriate dummy variables for each category Baseline for comparisons 10 Eurostat

11. Predictive Mean Matching (PMM) Similar to regression For each missing imputes a value randomly chosen from the set of observed values having the closest predicted value to the forecast obtained by the regression model Identified as providing best imputations 11 Eurostat

12. Data mining evaluation criteria 12 Correlation coefficient Mean Absolute Error Root Mean Squared Error Relative Absolute Error Root Relative Squared Error Eurostat

13. 13 COUNTRYMETHODCorrelationMAERMSERAERRSE ESLMS0.74435.8708.359.168.0 ESM5P0.75431.3694.558.466.7 ESMLP0.73449.6718.860.969.0 ESPMM0.55634.8982.586.094.3 ESRBF0.75430.0696.258.366.8 ESREG0.73443.8716.960.168.8 ATLMS0.53648.51551.663.684.6 ATM5P0.55636.31529.162.483.2 ATMLP0.44751.71733.773.796.1 ATPMM0.33944.52067.192.7116.6 ATRBF0.53643.71543.163.184.0 ATREG0.52655.71561.964.385.2 Eurostat

14. Statistical inference evaluation criteria 14 Output of mean & other parameters estimates, e. g. Similarity between original distribution & that with imputed values,, Eurostat

15. 15 COUNTRYMETHODMeanModeMedianSTD ESORIGINAL18201400157510.5 ESLMS1780140015958.9 ESM5P1777140015929.2 ESMLP1782140015879.3 ESPMM18191305157210.5 ESRBF1776140015869.2 ESREG1775140016059.0 ATORIGINAL22871800195527.3 ATLMS22281915199821.8 ATM5P22091915199321.5 ATMLP22381915198923.1 ATPMM22881500196825.9 ATRBF22141915199421.8 ATREG22051915199721.6 Eurostat

16. 16 Imputation errors for the original Wages variable in one of the simulated files using M5P imputation method Shrinkage to the mean!! Eurostat

17. 17 CountryMethodHellinger distance Kolmogorov-Smirnov distance ESLMS0.0500.031 ESM5P0.0430.028 ESMLP0.0360.023 ESPMM0.0150.009 ESRBF0.0410.027 ESREG0.0520.035 ATLMS0.0490.028 ATM5P0.0500.030 ATMLP0.0360.022 ATPMM0.0180.012 ATRBF0.0450.026 ATREG0.0500.030 Eurostat

18. 18 Histograms of the Log (wages) variable Eurostat

19. But… When the purpose is obtaining complete files free of missing data… What happens with the results at a more detailed level of disaggregation? Do the comparative advantages and disadvantages remain? 19 Eurostat

20. Example (region of Extremadura in Spain)(1) 20 METHODCorrelationMAERMSERAERRSE LMS0.85317.2489.954.857.1 M5P0.83313.5489.454.257.1 MLP0.80337.8521.858.360.8 PMM0.66504.8731.887.586.0 RBF0.84314.8480.154.456.0 REG0.82339.1504.758.658.9 Eurostat

21. Example (region of Extremadura in Spain)(2) 21 METHODMeanModeMedianSTD LMS1477137213485.7 M5P1471137213376.0 MLP1476137213406.1 ORI1492140013176.8 PMM1557137313746.9 RBF1467137213236.0 REG1519137213935.9 METHODHellinger distance Kolmogorov-Smirnov distance LMS0.0830.055 M5P0.0680.045 MLP0.0670.044 PMM0.0760.063 RBF0.0620.038 REG0.0880.086 Eurostat

22. Thus… Results at a more detailed level of disaggregation can be reversed…!!! 22 Eurostat

23. Final remarks (1) Data mining procedures provide imputations which reproduce the original individual values sign. better PMM produces sign. better estimates of means & other statistical parameters for the whole population Imputations by regression are slightly worse than those of data mining procedures 23 Eurostat

24. Final remarks (2) Paradoxical result: Given an original distribution one imputed-population has more similar individual values another imputed-population has more similar distribution parameters PMM produces random imputations (from regressions) designed to improve   estimates: at the cost of closeness to individual values!! Different possibilities to improve data mining imputations Might it be worth considering also individual one-to-one likeness when assessing similarities between distributions? 24 Eurostat Maybe valid inference in the era of data integration, data matching, small area estimation… should be another thing?

25. 25 Eurostat Thanks for your attention !!

26. 26 Donald B. Rubin, "Multiple Imputation After 18+ Years", JASA, vol. 91, no. 434, June 1996 "…Judging the quality of missing data procedures by their ability to recreate the individual missing values (according to hit- rate, mean square error, etc.) does not lead to choosing procedures that result in valid inference, which is our objective"