Big Data Quality, Partnerships and Privacy Teams

Agenda  Introduction  Approach  Results of 3 task teams Quality Partnerships Privacy

Approach  Analysis of best practice and available documentation  Work in virtual teams on two-week schedule  Testing in Sandbox  Virtual sprints  Workshops

Quality  Quality framework(s) for Big Data.  Testing the framework(s).  Indicators and associated metadata requirements. Approach - the concept of hyperdimensions was taken from the administrative data quality framework.

Conclusions for Quality  There is a need for quality assessment covering the entire business process. Input quality can be explored and assessed by using and elaborating existing input quality frameworks. Throughput quality can be maintained by following quality processing principles but quality dimensions need to be further developed for Big Data processing. Additions have been proposed to output quality dimensions from existing frameworks, to make them suitable for Big Data applications.

Partnerships  Task: Explore current experiences and produce guidelines for partnerships  Sources: Experiences from the Sandbox Experiences from Task Team participants / organisations Survey information: partnership questions added to a UNSD survey on Big Data for Official Statistics  Different types of partnerships - data providers design and analysis, technology partners…

Conclusions for Partnerships  A project can only exist if a working partnership can be forged with a data provider  For multinational data sources partnership agreements need to be drafted that can be used by all statistical offices  Operational guidelines for forging Big Data partnership agreements are needed

Privacy  To give an overview of existing tools for risk management in view of privacy issues  To describe how risk of identification relates to Big Data characteristics  To draft recommendations for NSOs on the management of privacy risks related to Big Data

Conclusions for Privacy  Existing tools are well-developed  Privacy risk can be linked to Big Data characteristics  Recommendations have been formulated on: information integration and governance statistical disclosure limitation/control managing risk to reputation  But: not much experience yet with Big Data privacy issues

More Information  UNECE Wiki  Presentations at NTTS Conference Quality 17A Partnerships 4A Privacy 9B