European Conference on Quality in Official Statistics (Q2010) 4-6 May 2010, Helsinki, Finland Brancato G., Carbini R., Murgia M., Simeoni G. Istat, Italian National Statistical Institute Presenter: Giovanna Brancato Consulting the users on quality documentation supply

Consulting the users on metadata and quality documentation supply Consulting the users on quality documentation supply Background Istat project, financed in the framework of Eurostat grants on quality management and evaluation Objectives To evaluate users’ needs and satisfaction with regard to some metadata and quality documentation tools, developed at Istat and coherent with Eurostat metadata standards. The tools are: a) SIQual (Information System on Quality) b) Process report c) Quality report Methods a): Web survey to quality managers from European Statistical Agencies b) and c): National users’ consultations by means of focus groups Istat team work Communication sector: A.M. Tononi Survey managers:A. Mancini, S. Montecolle Data capturing and web surveys: R. Balestrino, M. Degortes, M. Landriscina, A. Nunnari

Istat Metadata and Quality documentation tools (1) Consulting the users on quality documentation supply SIQual Reference metadata: content, process characteristics, methodologies On line data: publications, databanks Documentation: regulations, instructions, manuals questionnaires Short descriptions, standardised language Italian and English Standard quality indicators (for internal use only)

Istat Metadata and Quality documentation tools (2) Consulting the users on quality documentation supply Process report 3-5 page long Process description 1) Introduction 2) Informative needs 3) Survey design 4) Data collection 5) Data treatment 6) Estimation and validation 7) Documentation Highly standardised and integrated to the quality documentation system

Istat Metadata and Quality documentation tools (3) Consulting the users on quality documentation supply Quality report 2-3 page long Product quality-oriented Following Eurostat quality dimensions: Relevance, Accuracy and reliability, Timeliness, Accessibility and clarity, Comparability and Coherence Incorporating a minimum set of standard quality indicators (coefficient of variation, unit response rate, net imputation rate, modification rate, mean and mean absolute revision index, timeliness, length of comparable time series) Provided with a Reading guide useful for quality definitions and quality indicators’ formulas

Eurostat quality documentation standards Consulting the users on quality documentation supply Quality definition Quality dimensions Standard quality indicators and reports (EHQR*) Euro SDMX Metadata Structure (ESMS) * ESS Standards and Handbook for Quality Reports

Euro SDMX Metadata Structure, ESMS (1) Consulting the users on quality documentation supply ESMS Metadata items 1. Contact information12. Quality management 2. Metadata update13. Relevance 3. Statistical presentation14. Accuracy and reliability 4. Unit of measure15. Timeliness and punctuality 5. Reference period16. Comparability 6. Institutional mandate17. Coherence 7. Confidentiality18. Cost and burden 8. Release policy19. Data revision 9. Frequency of dissemination20. Statistical processing 10. Dissemination format21. Comment 11. Accessibility of documentation

Euro SDMX Metadata Structure, ESMS (2) Consulting the users on quality documentation supply ESMS Quality Indicators Completeness and User satisfaction Rate of available statistics Users satisfaction index Time since most recent satisfaction survey SamplingCoefficient of variation Non sampling error and revision Rate of overcoverage Edit failure rate Unit response rate Item response rate for a given var. Imputation rate for a given variable Number of mistakes Mean average revision Relative mean average revision

Euro SDMX Metadata Structure, ESMS (3) Consulting the users on quality documentation supply ESMS Quality Indicators (cont.) Timeliness and punctualityTimeliness of provisional & final Punctuality AccessibilityNumber of subscriptions/purchase per key paper report Number of access to databases Rate of completeness of metadata ComparabilityLength of comparable time series Asymmetries for mirror flows Cost and burdenAnnual operational costs Annual respondent burden

Istat metadata mapping with ESMS Consulting the users on quality documentation supply 1. Contact information12. Quality management 2. Metadata update13. Relevance 3. Statistical presentation14. Accuracy and reliability 4. Unit of measure15. Timeliness and punctuality 5. Reference period16. Comparability 6. Institutional mandate17. Coherence 7. Confidentiality18. Cost and burden 8. Release policy19. Data revision 9. Frequency of dissemination20. Statistical processing 10. Dissemination format21. Comment 11. Accessibility of documentation SIQual - METADATA Process report Quality report

Istat quality indicators mapping with ESMS Consulting the users on quality documentation supply Quality Indicators Completeness and User satisfaction Rate of available statistics Users satisfaction index Time since most recent satisfaction survey SamplingCoefficient of variation* Non sampling error and revision Rate of overcoverage Edit failure rate Unit response rate Item response rate for a given var. Imputation rate for a given variable Number of mistakes Mean average revision Relative mean average revision *Under study for a next implementation

Istat quality indicators mapping with ESMS Consulting the users on quality documentation supply Quality Indicators (cont.) Timeliness and punctualityTimeliness of provisional & final Punctuality AccessibilityNumber of subscriptions/purchase per key paper report Number of access to databases Rate of completeness of metadata ComparabilityLength of comparable time series Asymmetries for mirror flows Cost and burdenAnnual operational costs Annual respondent burden* *Under study for a next implementation

Evaluation methods for SIQual Consulting the users on quality documentation supply WEB SURVEY TO: Users/producers: quality managers members of Eurostat Working Group on Quality QUESTIONNAIRE SIQual from a general point of view: - usefulness - redundancy - lack of information - layout - friendliness Specific content areas. Clarity, easiness, and comprehensibility of: - homepage - 3 search criteria pages - single process description - help points and glossary SOFTWARE XsiMpLe, a prototype of software for computer assisted data collection, developed at Istat and based on open source technologies

Evaluation methods for the process and quality reports Consulting the users on quality documentation supply FOCUS GROUPS - Theory Discussion among a small group (6-10) of target population members. Objectives: to explore some pre-identified critical issues Methods: - guided by a moderator - lasting a couple of hours - several focus groups for different sub-populations. Critical issues: role of the moderator (create relaxed atmosphere, maintain the discussion within the framework of interest, …).

Evaluation methods for the process and quality reports Consulting the users on quality documentation supply APPLICATION - Selection of two Istat important statistical processes: Monthly survey on industrial production index Multipurpose survey: aspects of daily life - Identification of the 6 users for each process to invite to the focus groups: 1° focus group participants: two economic journalists, four economic analysts. 2° focus group participants: one social sector journalist, three academic representatives, two analysts from research agencies - Sending of the material in advance - Organisation of 1 focus group for each process, lasting from 11:00 a.m. to 5:00 p.m. - Audio-recording of the meetings - Conditions different from the standard application field

Preliminary results Consulting the users on quality documentation supply WEB SURVEY ON SIQual Launched on mid-April  First results available in June FIRST FOCUS GROUP General satisfaction with the integrated tools of documentation General appreciation for the consultation method PROCESS REPORT Introduce the use of flow charts for process description Improve communication ability QUALITY REPORT Users uncomfortable with the Relevance concept Need for incorporating some definitions available in the “Reading guide” into the report

Consulting the users on quality documentation supply Future work Analyse web survey data and conclude the analyses for the focus groups data Identify the most important comments from the users and make them to converge to concrete improvement changes on Istat documentation tools Provide feedback on metadata items and quality indicators defined at Eurostat level Fine tune the tools Launch the implementation strategy

Thank you