1. www.stat.gov.lt Oslo, 24–26 September 2012 Work Session on Statistical Data Editing APPLICATION OF THE DEVELOPED SAS MACRO FOR EDITING AND IMPUTATION AT STATISTICS LITHUANIA Jurga Rukšėnaitė Chief specialist Methodology and Quality division

2. www.stat.gov.lt TOPICS I.Methods of detection of errors and outliers II.Methods of data imputation III.The use of the developed SAS Macro at Statistics Lithuania (practical example) Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

3. www.stat.gov.lt Oslo, 24–26 September 2012 Work Session on Statistical Data Editing I Methods of detection of errors and outliers For quantitative variables 1.Universal method 2.Interval method 3.Standard deviation rule 4.Testing of hypothesis

4. www.stat.gov.lt Oslo, 24–26 September 2012 Work Session on Statistical Data Editing I.1 Universal method

5. www.stat.gov.lt I.2 Interval method Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

6. www.stat.gov.lt Oslo, 24–26 September 2012 Work Session on Statistical Data Editing I.3 Standard deviation rule

7. www.stat.gov.lt I.4 Testing of hypothesis Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

8. www.stat.gov.lt II Imputation 1.Imputation using distributions 2.Imputation using donors 3.Imputation using models Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

9. www.stat.gov.lt II.1 Imputation using distributions (1) Figure 1. Statistical models Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

10. www.stat.gov.lt II.1 Imputation using distributions (2) Oslo, 24–26 September 2012 Work Session on Statistical Data Editing The study variableChosen distribution Two different valuesBernoulli distribution Three to eight different valuesDiscrete random variable More than eight different values Continuous distributions (uniform, normal, lognormal, and exponential)

11. www.stat.gov.lt II.2 Imputation using donors  Historical (cold-deck) imputation replaces the missing value of an item with a constant value from an external source (previous survey).  Hot-deck imputation replaces missing data with comparable data from the same data set.  Nearest neighbor imputation replaces missing data with the donor value. The right donor is found by calculating the distance function from a set of auxiliary information. Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

12. www.stat.gov.lt II.3 Imputation using models Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

13. www.stat.gov.lt Practical examples Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

14. www.stat.gov.lt Example 1. Detection of outliers  Quarterly statistical survey on short-term statistics on service enterprises  The study variable is income in each quarter (PAJ3),  The auxiliary variable is the number of employees. The output Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

15. www.stat.gov.lt Example 2. Verification of imputation for quantitative data The verification table shows the percentage difference between the predicted and the real value Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

16. www.stat.gov.lt Example 3. Verification of imputation for qualitative data Simulated data was used. The study variable y4 has two possible values: 1 and 2. Three auxiliary variables: x1, x2, and x3. Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

17. www.stat.gov.lt Conclusions and future work  SAS Macro program consists of five parts: detection of errors, detection of outliers, imputation using the nearest neighbor method, imputation using models, and imputation using distributions.  Several trainings were organized for the employees of Statistics Lithuania. 37 employees attended the training of this program. Half of them is using or going to use the SAS Macro in their work.  The program was tested using real data. The results showed that time spent for data editing/imputation was reduced.  The program not only gives a new data set with imputed values but also calculates several statistics (sample mean before and after imputation, standard deviation before and after imputation), which can be used to assess the quality of imputation.  The latest improvement to this program enables the identification of strata variable. This improvement allows finding errors or outliers and imputing missing values separately in each stratum, group or domain.  The methods programed now are the simplest one; therefore, later, more complicated methods for the imputation and detection of outliers will be added to the program. Oslo, 24–26 September 2012 Work Session on Statistical Data Editing

18. www.stat.gov.lt Oslo, 24–26 September 2012 Work Session on Statistical Data Editing Questions?

19. www.stat.gov.lt References 1.Chen J. and Shao J. Nearest neighbor imputation for survey data. Journal of Official Statistics, 16: 113–131, 2000. 2.Čekanavičius V., Murauskas G. Statistika ir jos taikymai // 1 dalis. TEV, Vilnius, 2000. 3.Čekanavičius V., Murauskas G. Statistika ir jos taikymai // 2 dalis. TEV, Vilnius, 2002. 4.Granquist L. Macro-editing. A review of some methods for rationalizing the editing of survey data. http://www.unece.org/stats/publications/editing/SDE1chB.pdfhttp://www.unece.org/stats/publications/editing/SDE1chB.pdf 5.McFadden, D. Conditional logit analysis of qualitative choice behavior. In Frontiers in Econometrics, ed. P. Zarembka, New York: Academic Press: 105-42. 1974. 6.Krapavickaitė, D., Plikusas, A. Imčių teorijos pagrindai. Vilnius: Technika, 2005. 7.Little R.J.A. and Rubin D. B. Statistical analysis with missing data. Wiley, 1987. 8.Luzi O., et al. Recommended Practices for Editing and Imputation in Cross-Sectional Business Surveys. EDIMBUS-RPM, 2007. http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/RPM_EDIMBUS.pdf http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/RPM_EDIMBUS.pdf 9.Nordholt E. S. Imputation: Methods, Simulation Experiments and Practical Examples. International Statistical Review, 66: 157–180, 1998. 10.Statistical data editing. Methods and techniques. Vol. 1, United Nations, 1994. 11.Statistical data editing. Impact on data quality. Vol. 3, United Nations, 2006. Oslo, 24–26 September 2012 Work Session on Statistical Data Editing