Giovanna Brancato, Marina Signore Istat Work Session on Statistical Metadata (METIS) Metadata and Quality Indicators Reuse for Quality reporting Geneva, March 2010

Outline Work Session on Statistical Metadata (Metis) Geneva, March Istat information System on Survey Documentation (SIDI/SIQual) 2.Istat Quality Reporting for external users 3.Metadata and Quality documentation reuse for Istat quality reporting 4.Metadata and quality reporting reuse for ESMS Standards

Quality documentation and dissemination at Istat: SIDI and SIQual Work Session on Statistical Metadata (Metis) Process metadata Quality metadata Quality indicators SIQual navigation system (intranet/internet) Geneva, March 2010 SIDISIDI Statistical Yearbook Process report Quality report Programming Management Systems Quality pilots

Geneva, March 2010 Key concept Statistical production process: process aimed at producing statistical information  primary survey: data collection from reporting units (households, businesses, institutions) including administrative sources surveys 225 documented primary surveys 156 active, 53 ceased, 16 suspended  secondary study: processing of statistical data already available from previous primary surveys or secondary studies 101 documented secondary studies 81 active, 14 ceased, 6 suspended SIDI metadata Work Session on Statistical Metadata (Metis)

Geneva, March 2010 SIDI navigation environment Intranet: metadata and standard quality indicators functionalities for queries on organisational units (direction/service/unit) Internet: metadata SIQualSIQual Work Session on Statistical Metadata (Metis)

Geneva, March 2010 SIDI Standard Quality Indicators Work Session on Statistical Metadata (Metis) Group% of active processes with indicators Coverage and Unit nonresponse Coding Editing and Imputation Revision policy * 7.50 Timeliness and punctuality *92.24 Comparability *60.99 Coherence btw provisional and final *17.78 Coherence with other sources12.30 Resources and Costs65.98 Completeness status * Including Primary Surveys and Secondary Studies

Geneva, March 2010 Tools  Short methodological notes (SMNs)  Process reports (PR)  Quality reports (QR) Istat quality reporting for external users Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Use Compact reports included in the Statistical Yearbook Structure General conceptual metadata: observed issues, analysis unit Methodological process: reporting unit, periodicity; EU regulation, sampling design, data collection modes Quality metadata: synthesis of the main activities carried out to prevent, monitor and evaluate nonsampling errors (unit nonresponse and measurement error), validation techniques Dissemination of provisional and final results: Timeliness, geographical and sector level of aggregation Languages Italian (English) Istat short methodological note (SMN) Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Example SIDI documentation on observed issues: Attendance at museums and art exhibitions Chronic diseases Cinema attendance Friendship relations Games and other leisure activities Health conditions Housing conditions … SIDI metadata reuse in the SMN Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Example SIDI documentation on the activities to prevent unit non response: - Survey presentation letter signed by Istat President - Guarantees on statistical confidentiality - Interviewer identification badge - Description of survey objectives by interviewers - Telephone contacts to make an appointment for the interview - Establishing a toll free line or telephone number for further explanations SIDI metadata reuse in the SMN Work Session on Statistical Metadata (Metis) Compacted description in the short methodological note: “Activities for preventing unit nonresponse”

Geneva, March 2010 Examples SIDI documentation on Timeliness: SIDI quality indicators reuse in SMN Work Session on Statistical Metadata (Metis) average over a period single value

Geneva, March 2010 Use Standard process documentation to be included in publications on survey results as methodological annex. General structure - Introduction - Objectives - Survey design - Data capturing - Data pre-treatment - Processing - Documentation Istat process report (PR) Work Session on Statistical Metadata (Metis)

Geneva, March 2010 The report is pre-filled in with the information drawn from SIDI SIDI metadata reuse in PR Work Session on Statistical Metadata (Metis) An editor window allows to adjust and modify the text to turn it in a fluent language

Geneva, March 2010 Use It is a short report, including a subset of quality indicators, to be disseminated together with press releases on Istat web site Structure It is organised into the following sections: - Relevance - Accuracy - Timeliness and punctuality - Clarity and accessibility - Coherence and comparability Istat quality report (QR) Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Qualtiy indicators - Sampling error (confidence interval or coefficient of variation) - Weighted/Unweighted Response rate [respondent units / (eligible units and unresolved units)] - Net imputation Rate [imputations from blank to nonblank values/imputable values] - Modification Rate [changes from nonblank to nonblank values/imputable values] - Mean Revision (size and direction of the revisions) - Absolute mean revision (average size of revisions) - Timeliness (of provisional/final results, of data sources) - Comparability (length of the series) - Coherence between provisional and final estimates (difference or relative difference btw provisional and final estimates) Indicators included in the quality report (QR) Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Implementation phase Development of the product quality report integrated to SIDI management environment Limited reuse of metadata Reuse of standard quality indicators following the short methodological approach Need of developing new quality indicators and revising some existing ones SIDI metadata and quality indicators reuse in QR Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Definition Statistical metadata to be used by Member States for documenting statistical data to Eurostat  metadata  quality indicators (ongoing work)  harmonisation within Eurostat, OECD, IMF Uses  Comparisons and evaluations  Metadata and quality information for the users ESMS requirements Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Organization: Institutional mandate, confidentiality legislation, release policy, quality management Data set and variables: classification system, sector coverage, statistical concepts and definitions, statistical unit and population, reference area, time coverage, base period, unit of measure, reference period Dissemination frequency and formats Eurostat quality components: relevance, accuracy, … Statistical processing Standard quality indicators [Simulation: compilation of ESMS template with metadata and quality indicators for a survey ~ 60%] Nature of ESMS metadata Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Type of data and frequency of data collection Organisational data: univocally defined throughout the institution Metadata: stable over time Quality indicators: survey edition Standard quality indicators Harmonisation: among different Member States, among different EU regulations  Eurostat Handbook/Standards for Quality Reports (2009) Priority setting: identification of a minimum set Use: additional indicators upon request for expert users Aggregation: survey edition, annual averages (for monthly surveys) Dissemination: summary measures, ranges of variations, graphical representation Issues on ESMS Work Session on Statistical Metadata (Metis)

Metadata and quality indicators reuse for quality reporting Thank you for your attention ! Geneva, March 2010

Metadata documentation standards Thesaura: lists of standard items to be used to document process activities and quality control actions Content: Topics of the survey, analysis units, questionnaire Process: Reporting unit (sources of the secondary study), survey design, data collection, data transformation, data processing Quality: Activities carried out to prevent, monitor and evaluate survey errors Metadata qualitative descriptions: free notes supporting standard metadata items SIDI main features Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Use Standard process documentation to be included in publications on survey results as methodological annex. General structure - Introduction (name, period, description) - Objectives (information needs and periodicity, issues) - Survey design (population, sampling, reporting unit and frame, questionnaire design and testing, pilot survey) - Data capturing (data collection mode, field operations, sensitive questions, unit nonresponse monitoring, mandatory participation, response error) - Data pre-treatment (coding, manual revision, data entry, editing and imputation) - Processing (estimation, validation, use of generalised software) - Documentation (list of relevant documents) Istat process report (PR) Work Session on Statistical Metadata (Metis)

Geneva, March 2010 Use It is a short report, including a subset of quality indicators, to be disseminated together with press releases on Istat web site Structure It is organised into the following sections: - Relevance (regulations and use of the data) - Accuracy (methodological context, sampling error, nonsampling error, revision policy) - Timeliness and punctuality (production of preliminary results, timeliness) - Clarity and accessibility (links to data and documentation) - Coherence and comparability (national or international regulations and manuals, over time data comparability) Istat quality report (QR) Work Session on Statistical Metadata (Metis)