Administrative Data Challenges: Joining Forces A view from the UK DIME – June 15, 2018

Vision and ambition Shared vision with other NSIs: To make better use of administrative and transaction data To do this, and to ensure statistics can be transparent and reproducible, we need to develop new methods: catalyse a paradigm shift.

Administrative Data Challenges: Joining Forces GSS Methodology Advisory Committee Extraordinary Meeting –28 February 2018, London 68 key stakeholders and experts in administrative data participated in a parallel strand workshop, discussing the challenges identified by Prof David Hand in Statistical Challenges of Administrative and Transactional Data https://rss.onlinelibrary.wiley.com/doi/full/10.1111/rssa.12315

The Challenges summarised… Challenge 1. Statistics teaching should cover data quality issues Challenge 2. Develop detectors for particular quality issues. Challenge 3. Construct quality metrics and quality scorecards for data sets. Challenge 4. Audit data sources for quality. Challenge 5. Be aware of time series discontinuities arising from changing definitions. Challenge 6. Evaluate the impact of data quality on statistical conclusions.

The Challenges summarised… Challenge 7. Explore potential sources of non‐representativeness in the data. Challenge 8. Develop and adopt tools for adjusting conclusions in the light of the data selection processes. Challenge 9. Explore how suitable the administrative data are for answering the questions. Identify their limitations, and be wary of changes of definitions and data capture methods over time. Challenge 10. Report changes and time series with appropriate measures of uncertainty, so that both the statistical and the substantive significance of changes can be evaluated. The measures of uncertainty should include all sources of uncertainty which can be identified

The Challenges summarised… Challenge 11. Be aware that administrative data are observational data, and exercise due caution about claiming causal links. Challenge 12. Be aware of the risks that are associated with linked data sets and the potential effect on the accuracy and validity of any conclusions. Recognize that quality issues of individual databases may propagate and amplify in linked data. Develop better measures of overall combined data quality.

The Challenges summarised… Challenge 13. Continue to develop statistically principled and sound methods for record linkage and evidence assimilation, especially from non‐structured data and data of different modes. Challenge 14. Develop improved methods for data triangulation, combining different sources and types of data to yield improved estimates. Challenge 15. Continue to explore anonymization and deidentification methods.

Translating the challenges to workstreams Development of frameworks for measurement and reporting of quality in administrative data Representativeness/Design of structured data collection to measure errors Record linkage Estimation Missing values: Imputation/adjustment Disclosure control

Development of frameworks for measurement and reporting of quality in administrative data   Strand 1 Challenge 1 Statistics teaching should cover data quality issues Challenge 2 Develop detectors for particular quality issues Challenge 3 Construct quality metrics and quality scorecards for data sets Challenge 4 Audit data sources for quality Challenge 5 Be aware of time series discontinuities arising from changing definitions Challenge 6 Evaluate the impact of data quality on statistical conclusions Challenge 9 Explore how suitable the administrative data are for answering the questions. Identify their limitations, and be wary of changes of definitions and data capture methods over time Challenge 11 Be aware that administrative data are observational data, and exercise due caution about claiming causal links.

Representativeness/ Design of structured data collection to measure errors Strand 2 Challenge 7 Explore potential sources of non-representativeness in the data Challenge 8 Develop and adopt tools for adjusting conclusions in the light of the data selected Challenge 9 Explore how suitable the administrative data are for answering the questions. Identify their limitations, and be wary of changes of definitions and data capture methods over time

Record linkage Strand 3 Challenge 12 Be aware of the risks that are associated with linked data sets and the potential effect on the accuracy and validity of any conclusions. Recognise that quality issues of individual databases may propagate and amplify in linked data. Challenge 13 Continue to develop statistically principled and sound methods for record linkage and evidence assimilation, especially from non-structured data and data of different modes.

Estimation Strand 4 Challenge 6 Evaluate the impact of data quality on statistical conclusions Challenge 8 Develop and adopt tools for adjusting conclusions in the light of the data selected Challenge 10 Report changes and time series with appropriate measures of uncertainty, so that both the statistical and substantive significance of changes can be evaluated. The measures of uncertainty should include all sources of uncertainty that can be identified. Challenge 14 Develop improved methods for data triangulation, combining different sources and types of data to yield improved data sources.

Missing values: Imputation/adjustment Strand 5 Challenge 8 Develop and adopt tools for adjusting conclusions in the light of the data selected Challenge 9 Explore how suitable the administrative data are for answering the questions. Identify their limitations, and be wary of changes of definitions and data capture methods over time

Disclosure control Strand 6 Challenge 15 Continue to explore anonymization and de-identification

What next? Establishing an Admin Data Methods Branch in the Methodology Division. Incorporating the workstreams into a plan. Developing expertise: blended approach to solutions: in-house, commissioned, developed in partnership (loosely modelled on ESCoE). Publication to ensure credibility.

Any questions?

Appendix

Admin data experts University of Edinburgh University of Bristol Liverpool University University of Huddersfield University of Leicester Queens University, Belfast Swansea University University of Surrey Lancaster University Valtech Oxford University National Centre for Social Research SAIL, Swansea Imperial College London Demographic Queen Mary University, London Southampton University University of Leeds

Admin data experts cont. Greater London Authority Department for Work and Office for National Statistics Pensions (DWP) NHS Digital Department for Existing the EU (DExEU) Ministry of Justice(MoJ) ISTAT UK Statistics Authority Department for Business, Energy and Industrial Strategy (BEIS) NRS Scotland Her Majesty Revenue & Customs (HMRC) Department for Education (DfE) Local Government Association NISRA Statistics Norway