1. Maria João Zilhão Planning and Quality Control Unit « High Level Seminar “Quality Matters in Statistics” 21-23 June, Athens, Greece Implementation of the European Statistics Code of Practice in Statistics Portugal «

2. The trustworthiness of a Statistical organization To guaranty : Working methods and its results Code of Practice Establish the conduct of the organization

3. « How important is the Code of practice Code of Practice European Statistics Code of Practice 15 principles: Institutional environment Statistical Processes Statistical Output

4. Objectivity in production processes To transmit …. impartiality Integrity Confidentiality Trust (…)

5. How important are the Operational aspects? Systematization of a code Basic guidelines of a Statistical organization Clarification to the users Clear message of integrity Deployment into in-house documents / Policies / Procedures « European Statistics Code of Practice

6. Statistics Portugal example (i) Levels of documentation Mission, Vision and Values of the Organization ESS Code of Practice Policies Operational procedures and tasks Internal level

7. Revision Policy Dissemination Policy 1.Key factors underlying a revision 2.Typology of revisions 3.Dimensions of revisions analisys 4.General and operational principles of the revisions policy 1. Principles for the dissemination policy of Official Statistics 2. Release of official statistics 3. Accessibility of official statistics 4. Monitoring and assessment of the dissemination policy Statistics Portugal example (ii)

8. Quality Management Unit Internal Audits Departments External Audits Documenting Processes Customer Satisfaction Surveys Departments Suggestions / Complaints Performance indicators Quality Management System at Statistics Portugal Quality Documentation System ISO 19011 ISO 10013 P D C A Quatily reports / Method. Doc.

9. Opportunity to reflect upon the principles, and Statistics Portugal Performance Systematize the related documentation and relevant information « European Statistics Code of Practice – Preparation of the Peer Review  Internal team – different areas  Involvement of external stakeholders (other producers; respondents; users);  Identification of good practices;  Faced as an external audit – improved in-house process.

10. References for auditing ISO Norms: 9000:2000…Implementation of Quality Management Systems 19011 -.. Auditing… 10013 -…Documentation systems

11. Top Management support Documentation System The results of auditing and self-assessments must be consequent A Clear communication and involvement of the staff Preconditions Proper preparation!

12. Process design Definition of different phases of the process Definition of responsibilities Definition of minimum documentation Handbooks are produced by pluridisciplinary teams Documentation System ISO 10013 Documenting processes Procedures handbooks Documentation on survey processes

13. Phases I. Conception IV. EvaluationIII. DisseminationII. Operation II.2. Data collection II.3. Data treatment and analysis II.1. Planing and preparation of data collection I.1. Viability Study I.2. Methodological Study I.3. Technical approval Processes III.1. Dissemination IV.1. Evaluation Process Statistical Production Handbook Procedures Note: Each Process is composed by sub-processes and tasks Audits

14. Internal Quality Audits at Statistics Portugal (I) P D C A To verify the degree of procedures implementation (handbooks procedures) To Identify improvement opportunities To check the internal articulation of surveys To Improve survey documentation Objectives

15. I. Conception IV. EvaluationIII. DisseminationII. Operation II.2. Data collection II.3. Data treatment and analysis II.1. Planing and preparation of data collection I.1. Viability Study I.2. Methodological Study I.3. Technical approval III.1. Dissemination IV.1. Evaluation Statistical Production Handbook Procedures « Internal Quality Audits at Statistics Portugal (II)

16. Internal Procedure on auditing (Clear rules!) Top Management Support Exchange of Information / Strong Communication about auditing, through Top Management, and the Quality Unit Discussion Training of 14 Auditors (Statisticians) (ISO 19011) Internal Quality Audits at Statistics Portugal – Preparation (III)

17. Internal Quality Audits at Statistics Portugal –(IV) Principal rules Top Management defines the Quality/Statistical Audits’ annual plan No one should audit its own activity One Audit team needs 2 or 3 auditors Auditors produce Audit Plans and reports according to Procedures Improvement actions must be carried out by the audited departments Improvement actions must be published and followed by the audit teams

18. External Quality Audits at Statistics Portugal  Auditors are Peers (from other statistical Offices)  Basic rules are the same as for internal auditing  Two visits (+/- one week each)  Contacts between auditors and audited teams in the mid period  Auditors work recommendations with audited teams  Results are presented to the board  Action plan is made by audited teams and presented to the board

19. Benefits for Quality Improvement The identification of improvement opportunities in the audited process The implementation of corrective / preventive actions The redefinition of procedures Auditing involves people in the quality program (auditors and audited teams) Good Recommendations - A look from the outside! Identification of good practices that can be spread in the organization Self-assessment and auditing can be combined together A time to re-think and improve! Some NSI’s have experienced Quality Auditing! BUT!!!! -Time consuming activity -May face resistances - May be dificult to start - Better results in the long run if systematically implemented A Powerfull Tool!

20. Statistics Portugal experience: External Customer Satisfaction Surveys

21. « External Customers Preparation of Matrix Quality dimensions were discussed and defined accordingly to the following structure: 1. Quality perception, concerning… 1.1. Statistical information 1.2. Products 1.3. Services 2. Value for money 3. Global image 4. Expectations 5. Loyalty/fidelity Data pertinence Credibility Actuality Data accessibility Metadata accessibility Coherence and Comparability Geographical disaggregation Synthetic data Data punctuality Attributes that compose “Quality perception, concerning statistical information:

22. « External Customers A Matrix / Table Based on the concepts related to these 5 dimensions the matrix includes questions accordingly to several target groups: Examples:  Libraries  Researchers  Central Public Administration  Media  Schools  Site We can use also complementary information from database of customers: Examples:  Typology of information requested  Customers typology  Customers level of experience etc

23. « External Customers questionnaires are built in an identical way, having the same structure for different target groups Questionnaire structure and scales taking into account the different target groups that are to be inquired, questions may vary (although they are linked to the dimensions and its concepts)

24. « « « 31 qualitative variables divided into 4 groups:  recognised quality of the statistical information (7 queries) - based on the Eurostat quality dimensions  recognised quality of the product (6 queries)  recognised quality of the service (7 queries)  Statistics Portugal perceived image (11 queries) Variables Response scale External Customers

25. The analysis of the results was displayed by using the “extreme balance responses” (EBR). This aimed to value more the extreme answers and not the middle scores which tend to represent a less expressive dissatisfaction/disagreement or satisfaction/agreement. The attributed ponderations were as follows: « Data analysis extreme balance responses (EBR) EBR = F1*(-1)+ F2*(-0,5)+ F3*(-0,25)+ F4*(0,25)+ F5*(0,5)+ F6*(1) where FI = Relative frequency of each observed value for each one of the categories I =(1,…,6). Data became to be referenced in a metric scale: between -1 and +1, where values next to -1 mean full dissatisfaction/disagreement and values near +1 mean full satisfaction/agreement.

26. « External Customers Targets: Occasional:  Media (2006)  Researchers (2007)  Business Associations (2009)  Schools (2009 and 2010) Aims:  To assess quality perceived by external Statistics Portugal costumers  Identification and implementation of improvement actions Permanent:  Libraries (Statistics Portugal and Universities)  Website  Study visits / Events  Post- Service  Webinq

27. « « THANK YOU! Maria João Zilhão Statistics Portugal Planing and Quality Control Unit mjoao.zilhao@ine.pt