1. The development of a data editing and imputation tool set UN/ECE Work Session on Statistical Data Editing Topic (ii): Global solutions to editing Claude Poirier Oslo, Norway, 24 – 26 September 2012

2. Outline  Context  Desired features of a tool set  A basic tool set  Future work 27/02/2016 Statistics Canada Statistique Canada 2

3. Context  HLG-BAS strategy From little data to an abundance of data Increased need for quick statistics Towards an industrialised environment  Statistical Network Using EDIMBUS recommended practices E&I standards and guidelines developed by NSIs 27/02/2016 Statistics Canada Statistique Canada 3

4. Desired features of a tool set  Functionality Editing process: on-line edits, flow edits, fatal edits, distribution edits, outlying edits, selective editing, deductive edits, minimum change, processing sub- populations, macro editing Imputation process: rule-based imputation, deductive imputation, model-based, donor-based, proration, processing sub-populations Estimation process: Variance due to imputation 27/02/2016 Statistics Canada Statistique Canada 4

5. Desired features of a tool set (cont’d)  Quality criteria Relevance: Meets the real needs Accessibility: Is easy to use Interpretability: Is easy to understand Coherence: Offers standardization and interoperability Accuracy: Produces expected outcome Timeliness: Meets performance requirements 27/02/2016 Statistics Canada Statistique Canada 5

6. Desired features of a tool set (cont’d)  Important Software Characteristics Adaptability: Isolates specific statistical functions Reliability: Offers robustness and trust Maintainability: Enables enhancements Interoperability: Offers the «plug and play» feature 27/02/2016 Statistics Canada Statistique Canada 6

7. A basic tool set  BANFF Linear programming to identify the minimum change Imputation methods: Deductive; Donor; Estimator  CANCEIS Mixtures of categorical and numerical census data Minimum change while ensuring plausible imputation  SELEKT Suspicion level; potential impact; pseudo-bias Control on the importance of variables 27/02/2016 Statistics Canada Statistique Canada 7

8. Functionality of the tool set 27/02/2016 Statistics Canada Statistique Canada 8 EDITINGBanffCANCEISSelekt On-line edits Construction of groups Editing within groups Fatal edits Distribution edits Outlying edits Selective editing (scores) Deterministic edits Fellegi-Holt (min change) Editing of macro data Graphical editing IMPUTATIONBanffCANCEISSelekt Imputation within groups Rule-base imputation Deterministic imputation Model-based imputation Donor-based imputation Prorating imputation ESTIMATIONBanffCANCEISSelekt Variance due to imputation

9. Functionality of the tool set 27/02/2016 Statistics Canada Statistique Canada 9 EDITINGBanffCANCEISSelekt On-line edits Construction of groups Editing within groups Fatal edits Distribution edits Outlying edits Selective editing (scores) Deterministic edits Fellegi-Holt (min change) Editing of macro data Graphical editing IMPUTATIONBanffCANCEISSelekt Imputation within groups Rule-base imputation Deterministic imputation Model-based imputation Donor-based imputation Prorating imputation ESTIMATIONBanffCANCEISSelekt Variance due to imputation

10. Future work  To consider other tools Macro and Graphical editing  To investigate survey platforms POSS; BESt; 27/02/2016 Statistics Canada Statistique Canada 10

11. Thank you Merci  For more information,Pour plus d’information, please contact:veuillez contacter : Claude.Poirier@statcan.gc.ca 27/02/2016 Statistics Canada Statistique Canada 11