Statistics Portugal Quality Management System ESTP Training Course “Quality Management and survey Quality Measurement” Rome, 24 – 27 September 2013 Maria João Zilhão Unit Director Planning, Control and Quality Unit Statistics Portugal

680 employees (+ 500 interviewers) Lisboa, Porto, Coimbra, Évora, Faro National Statistical System (SEN) – National Strategy 2013 - 2017 European Statistical System (ESS) – European Statistical Programme 2013 - 2017

Mission statement The mission of Statistics Portugal is to produce and disseminate, in an effective, efficient and independent manner, high-quality official statistics relevant for the whole society.

Vision statement Statistics Portugal is acknowledged at an internal and external level as a reference statistical institution: as a producer and provider of high-quality official statistical information; as an independent and reliable organisation; as an entity stimulating statistical literacy in society; as a committed and efficient entity in international cooperation.

Values Statistics Portugal’s activities and its staff, in accordance with the European Statistics Code of Practice, are subject to the following Values: Professionalism, ethics and observance of confidentiality; Technical independence, objectivity and impartiality; Commitment to Quality; Customer-driven orientation and capacity to anticipate future customer needs; Efficacy and efficiency in Action; Respect for primary data providers; Creativity and innovation in terms of procedures, products and services; High motivation and strong focus on the acquisition of new skills.

The European Statistics Code of Practice – Implementation Implementation of the Code of Practice at Statistics Portugal Promotion of the Code of Practice at National Statistical System Internal Promotion (since june 2005) Self-assessment exercise (dez. 2005) Peer review (January 2008) Annual monitoring by Eurostat – Action Plans Future Peer Review (2014/15) Sessions to promote throughout the National Statistical System at the level of the Statistical Council Implementation by other National Statistical Producers – Entities with delegated functions by Statistics Portugal

The European Statistics Code of Practice for us … To ensure: Trustworthiness and credibility of the Statistical Auhorities Working methods and its results Establish the conduct of the organization How important are the Operational aspects? Systematization of a code Basic guidelines of a Statistical organization Important information to users Clear message of integrity Deployment into in-house documents / Policies / Procedures

Statistics Portugal example Documentation Mission, Vision and Values of the Organization Levels of documentation ESS Code of Practice Policies Internal level Guidelines / process description Operational procedures and tasks

Departments Departments Departments Planing and Quality Control Unit Statistics Portugal Quality Management System Documentation System Departments ISO 19011 ISO 10013 Quality Charter/ Policies Departments Process Documentaion Departments ISO 10002 Sugestions / Complaints Assessment of User/clients Satisfaction Internal Auditing External Auditing Quality reporting/ Methodological documentation (at survey level) Performance indicators/ management Planing and Quality Control Unit P D C A

Quality Charter at Statistics Portugal – “Carta da Qualidade” Third Edition (2009). Statistics Portugal Mission Statement, Vision Statement and Values. Public commitment with respect to quality in statistics, towards data suppliers and users/clients (statistics results accessibility and services available to the public, revisions and dissemination principles, suggestions complaints, satisfaction assessment). Fully aligned with the European Statistics Code of Practice.

Quality Management System - Framework ISO Norms 9000- … implementation of Quality Management Systems); (19011 - ..Auditing…); (10013 - …Documentation systems); … EFQM Excellence Model (European Foundation for Quality Management) CAF – Common Assessment Framework European Statistics Code of Practice

Documenting processes ISO 10013 Documenting processes Documentation System Procedures handbooks Documentation on survey processes Process design Definition of different phases of the process Definition of responsibilities Definition of minimum documentation Handbooks are produced by pluridisciplinary teams

Statistical Production Handbook Procedures Phases I. Conception II. Operation III. Dissemination IV. Evaluation Processes Processes Process Process II.1. Planing and preparation of data collection I.1. Viability Study III.1. Dissemination IV.1. Evaluation I.2. Methodological Study II.2. Data collection II.3. Data treatment and analysis Audits I.3. Technical approval Note: Each Process is composed by sub-processes and tasks

References for auditing ISO Norms: 9000 - …Implementation of Quality Management Systems 19011 - .. Auditing… 10013 - …Documentation systems

Top Management support Preconditions for auditing Top Management support A Clear communication and involvement of the staff Documentation System Proper preparation! The results of auditing must be consequent

Internal Quality Audits at Statistics Portugal (I) Objectives P D C A To verify the degree of procedures implementation (handbooks procedures) To Identify improvement opportunities To check the internal articulation of surveys To Improve survey documentation

Statistical Production Handbook Procedures Internal Quality Audits at Statistics Portugal (II) Statistical Production Handbook Procedures I. Conception II. Operation III. Dissemination IV. Evaluation II.1. Planing and preparation of data collection I.1. Viability Study III.1. Dissemination IV.1. Evaluation I.2. Methodological Study II.2. Data collection II.3. Data treatment and analysis I.3. Technical approval «

Internal Quality Audits at Statistics Portugal – Preparation (III) Training of 14 Auditors (Statisticians) (ISO 19011) Discussion Top Management Support Exchange of Information / Strong Communication about auditing, through Top Management, and the Quality Unit Internal Procedure on auditing (Clear rules!)

Internal Quality Audits at Statistics Portugal –(IV) Principal rules Top Management defines the Quality/Statistical Audits’ annual plan No one should audit its own activity One Audit team needs 2 or 3 auditors Auditors produce Audit Plans and reports according to Procedures Improvement actions must be carried out by the audited departments Improvement actions must be published and followed by the audit teams

External Quality Audits at Statistics Portugal Auditors are Peers (from other statistical Offices) Basic rules are the same as for internal auditing Two visits (+/- one week each) Contacts between auditors and audited teams in the mid period Auditors work recommendations with audited teams Results are presented to the board Action plan is made by audited teams and presented to the board

Benefits for Quality Improvement Some NSI’s have experienced Quality Auditing! The identification of improvement opportunities in the audited process The implementation of corrective / preventive actions The redefinition of procedures Auditing involves people in the quality program (auditors and audited teams) Good Recommendations - A look from the outside! Identification of good practices that can be spread in the organization Self-assessment and auditing can be combined together A time to re-think and improve! A Powerfull Tool! BUT!!!! -Time consuming activity May face resistances May be dificult to start - Better results in the long run if systematically implemented

Statistics Portugal experience: External Customer Satisfaction Surveys Preparation of Matrix Quality dimensions were discussed and defined accordingly to the following structure: Attributes that compose “Quality perception, concerning statistical information: Data pertinence Credibility Actuality Data accessibility Metadata accessibility Coherence and Comparability Geographical disaggregation Synthetic data Data punctuality 1. Quality perception, concerning… 1.1. Statistical information 1.2. Products 1.3. Services 2. Value for money 3. Global image 4. Expectations 5. Loyalty/fidelity

External Customers « A Matrix / Table Based on the concepts related to these 5 dimensions the matrix includes questions accordingly to several target groups: We can use also complementary information from database of customers: Examples: Libraries Researchers Central Public Administration Media Schools Site Examples: Typology of information requested Customers typology Customers level of experience etc «

External Customers Questionnaire structure and scales questionnaires are built in an identical way, having the same structure for different target groups taking into account the different target groups that are to be inquired, questions may vary (although they are linked to the dimensions and its concepts)

External Customers Variables Response scale 31 qualitative variables divided into 4 groups: recognised quality of the statistical information (7 queries) - based on the Eurostat quality dimensions recognised quality of the product (6 queries) recognised quality of the service (7 queries) Statistics Portugal perceived image (11 queries) Response scale Based on an ordinal scale with values comprised between 1 and 6, where 1 means full dissatisfaction/disagreement and 6 means full satisfaction/agreement.

Data analysis extreme balance responses (EBR) The analysis of the results was displayed by using the “extreme balance responses” (EBR). This aimed to value more the extreme answers and not the middle scores which tend to represent a less expressive dissatisfaction/disagreement or satisfaction/agreement. The attributed ponderations were as follows: EBR = F1*(-1)+ F2*(-0,5)+ F3*(-0,25)+ F4*(0,25)+ F5*(0,5)+ F6*(1) where FI = Relative frequency of each observed value for each one of the categories I =(1,…,6). Data became to be referenced in a metric scale: between -1 and +1, where values next to -1 mean full dissatisfaction/disagreement and values near +1 mean full satisfaction/agreement.

External Customers Targets: Aims: Permanent: Libraries (Statistics Portugal and Universities) Website; Post-Service WebInq (Respondants) Targets: Occasional: Media (2006); Researchers (2007) Business Associations (2009); Schools (2009) Schools (2010) To assess quality perceived by external Statistics Portugal costumers Identification and implementation of improvement actions Aims: