United Nations Oslo City Group on Energy Statistics OG7, Helsinki, Finland October 2012 ESCM Chapter 8: Data Quality and Meta Data 1

2 Introduction  IRES Chapter 9: deals with Data Quality Assurance and Meta Data  Under IRES, countries are encouraged to: Develop national quality assurance programs Document these programs Develop measures of data quality Make these available to users

3 Prerequisites of Data Quality  Institutional and organizational conditions, including:  Legal basis for compilation of data  Adequate data-sharing and coordination between partners  Assurance of confidentiality and security of data  Adequacy of resources – human, financial, technical  Efficient management of resources  Quality awareness

4 Promoting Data Quality  Make quality a stated goal of the organization  Establish standards for data quality  Track quality indicators  Conduct regular quality assurance reviews  Develop centres of expertise to promote quality  Deliver quality assurance training

5 What is a Quality Assurance Framework?  All planned activities to ensure data produced are adequate for their intended use  Includes: standards, practices, measures  Allows for: Comparisons with other countries Self-assessment Technical assistance Reviews by international and other users  See Figure 8.1 for examples of quality frameworks

6 Quality Assurance Framework  Six Dimensions of Data Quality, based on ensuring “fitness for use” 1.Relevance 2.Accuracy 3.Timeliness 4.Accessibility 5.Interpretability 6.Coherence

7 Quality Measures and Indicators  Should cover all elements of the Quality Assurance Framework  Methodology should be well-established, credible  Must be easy to interpret and use  Should be practical – reasonable, not an over- burden  For Key Indicators, see Chapter 8, Table 8.2

8 Sample Quality Indicators  From IRES Table 9.2, linked to QA Framework  Relevance: user feedback on satisfaction, utility of products and data  Accuracy: response rate, weighted response rate, number and size of revisions  Timeliness: time lag between reference period and release of data  Accessibility: number of hits, number of requests  Interpretability: amount of background info available  Coherence: validation of data from other sources

Quality assurance must be built into all stages of the survey process Survey Stages: 1.Specify needs 2.Design 3.Build 4.Collect 5.Process 6.Analyze 7.Disseminate 8.Archive 9.Evaluate 9 Quality Assurance Framework

1. Specify Needs Activities:  Determine needs: define objectives, uses, users  Identify concepts, variables  Identify data sources and availability  Prepare business case Quality Assurance  Consult with users and key stakeholders  Clearly state objectives, concepts  Establish quality targets  Check sources for quality, comparability, timeliness  Gather input and support from respondents 10

2. Design Activities:  Determine outputs  Define concepts, variables  Design data collection methodology  Determine frame & sampling strategy  Design production processes Quality Assurance  Consult users on outputs  Select, test & maintain frame  Design & test questionnaire and instructions  Use established standards  Develop processes for error detection  Develop & test imputation 11

3. Build Activities:  Build collection instrument  Build processing system  Design workflows  Finalize production systems Quality Assurance  Focus test questionnaire with respondents  Test systems for functionality  Test workflows; train staff  Document  Develop quality measures 12

4. Collect Activities:  Select sample  Set up collection  Run collection  Finalize collection Quality Assurance  Maintain frame  Train collection staff  Use technology with built in edits  Implement verification procedures  Monitor response rates, error rates, follow-up rates, reasons for non- response 13

5. Process Activities:  Integrate data from all sources  Classify and code data  Review, validate and edit  Impute for missing or problematic data  Create and apply weights  Derive variables Quality Assurance  Monitor edits  Implement follow-ups  Focus of most important respondents  Analyze and correct outliers 14

6. Analyze Activities:  Transform data to outputs  Validate data  Scrutinize and explain data  Apply disclosure controls  Finalize outputs Quality Assurance  Track all indicators  Calculate quality indicators  Compare data with previous cycles  Do coherence analysis  Validate against expectations and subject matter intelligence  Document all findings 15

7. Disseminate Activities:  Load data into output systems  Release products  Link to meta data  Provide quality indicators  Provide user support Quality Assurance  Format & review outputs  Verify that tools do not introduce errors  Verify disclosure control  Ensure all meta data and quality idicators are available  Provide contact names for user support 16

8. Archive Activities:  Create rules and procedures for archiving and disposal  Maintain catalogues, formats, systems Quality Assurance  Periodic testing of processes and systems  Ensure meta data are attached 17

9. Evaluate Activities:  Conduct post mortem reviews to assess performance, identify issues  Take corrective actions or make new investments, as required Quality Assurance  Consult with clients about needs, concerns  Monitor key quality indicators  Periodic data quality reviews  Perform ongoing coherence analysis  Compare with best practices elsewhere 18

19 Meta Data  Important for assessing “fitness for use” and ensuring interpretability  Required at every step of the survey process  Critical for enabling comparisons with other data  Should include results of data quality reviews  Figure 8.4: generic set of meta data requirements

20 Future of Meta Data  Should become a driver of survey design  Can be used proactively to prescribe definitions, concepts, variables and standards  Can support the harmonization of international surveys and data  Efforts are underway to create an integrated approach for producing and recording meta data

21 Thank you! Andy Kohut, Director Manufacturing and Energy Division Statistics Canada Section B-8, 11 th Floor, Jean Talon Building Ottawa, Ontario Canada K1A 0T6 Telephone: