1. Statistics Portugal Quality Management System
ESTP Training Course
“Quality Management and survey Quality Measurement”
Rome, 24 – 27 September 2013
Maria João Zilhão
Unit Director
Planning, Control and Quality Unit
Statistics Portugal

2. 680 employees (+ 500 interviewers)
Lisboa, Porto, Coimbra, Évora, Faro
National Statistical System (SEN) – National Strategy
European Statistical System (ESS) – European Statistical Programme

3. Mission statement
The mission of Statistics Portugal is to produce and disseminate, in an effective, efficient and independent manner, high-quality official statistics relevant for the whole society.

4. Vision statement
Statistics Portugal is acknowledged at an internal and external level as a reference statistical institution:
as a producer and provider of high-quality official statistical information;
as an independent and reliable organisation;
as an entity stimulating statistical literacy in society;
as a committed and efficient entity in international cooperation.

5. Values
Statistics Portugal’s activities and its staff, in accordance with the European Statistics Code of Practice, are subject to the following Values:
Professionalism, ethics and observance of confidentiality;
Technical independence, objectivity and impartiality;
Commitment to Quality;
Customer-driven orientation and capacity to anticipate future customer needs;
Efficacy and efficiency in Action;
Respect for primary data providers;
Creativity and innovation in terms of procedures, products and services;
High motivation and strong focus on the acquisition of new skills.

6. The European Statistics Code of Practice – Implementation
Implementation of the Code of Practice at Statistics Portugal
Promotion of the Code of Practice at National Statistical System
Internal Promotion (since june 2005)
Self-assessment exercise (dez. 2005)
Peer review (January 2008)
Annual monitoring by Eurostat – Action Plans
Future Peer Review (2014/15)
Sessions to promote throughout the National Statistical System at the level of the Statistical Council
Implementation by other National Statistical Producers – Entities with delegated functions by Statistics Portugal

7. The European Statistics Code of Practice for us …
To ensure:
Trustworthiness and credibility of the Statistical Auhorities
Working methods and its results
Establish the conduct of the organization
How important are the Operational aspects?
Systematization of a code
Basic guidelines of a Statistical organization
Important information to users
Clear message of integrity
Deployment into in-house documents / Policies / Procedures

8. Statistics Portugal example Documentation
Mission, Vision
and Values of the Organization
Levels of documentation
ESS Code of Practice
Policies
Internal
level
Guidelines / process description
Operational procedures and tasks

9. Departments Departments Departments Planing and Quality Control Unit
Statistics Portugal Quality Management System
Documentation System
Departments
ISO 19011
ISO 10013
Quality Charter/ Policies
Departments
Process Documentaion
Departments
ISO 10002
Sugestions / Complaints
Assessment of User/clients Satisfaction
Internal Auditing
External Auditing
Quality reporting/ Methodological documentation (at survey level)
Performance indicators/ management
Planing and Quality Control Unit P D C A

10. Quality Charter at Statistics Portugal – “Carta da Qualidade”
Third Edition (2009).
Statistics Portugal Mission Statement, Vision Statement and Values.
Public commitment with respect to quality in statistics, towards data suppliers and users/clients (statistics results accessibility and services available to the public, revisions and dissemination principles, suggestions complaints, satisfaction assessment).
Fully aligned with the European Statistics Code of Practice.

12. Quality Management System - Framework
ISO Norms … implementation of Quality Management Systems); ( Auditing…); ( …Documentation systems); …
EFQM Excellence Model (European Foundation for Quality Management)
CAF – Common Assessment Framework
European Statistics Code of Practice

13. Documenting processes
ISO 10013
Documenting processes
Documentation System
Procedures handbooks
Documentation on survey processes
Process design
Definition of different phases of the process
Definition of responsibilities
Definition of minimum documentation
Handbooks are produced by pluridisciplinary teams

14. Statistical Production Handbook Procedures
Phases
I. Conception
II. Operation
III. Dissemination
IV. Evaluation
Processes
Processes
Process
Process
II.1. Planing and preparation of data collection
I.1. Viability Study
III.1. Dissemination
IV.1. Evaluation
I.2. Methodological Study
II.2. Data collection
II.3. Data treatment and analysis
Audits
I.3. Technical approval
Note: Each Process is composed by sub-processes and tasks

15. References for auditing
ISO Norms:
…Implementation of Quality Management Systems
Auditing…
…Documentation systems

16. Top Management support
Preconditions for auditing
Top Management support
A Clear communication and involvement of the staff
Documentation System
Proper preparation!
The results of auditing must be consequent

17. Internal Quality Audits at Statistics Portugal (I)
Objectives P D C A
To verify the degree of procedures implementation (handbooks procedures)
To Identify improvement opportunities
To check the internal articulation of surveys
To Improve survey documentation

18. Statistical Production Handbook Procedures
Internal Quality Audits at Statistics Portugal (II)
Statistical Production Handbook Procedures
I. Conception
II. Operation
III. Dissemination
IV. Evaluation
II.1. Planing and preparation of data collection
I.1. Viability Study
III.1. Dissemination
IV.1. Evaluation
I.2. Methodological Study
II.2. Data collection
II.3. Data treatment and analysis
I.3. Technical approval

19. Internal Quality Audits at Statistics Portugal – Preparation (III)
Training of 14 Auditors
(Statisticians)
(ISO 19011)
Discussion
Top Management Support
Exchange of Information / Strong Communication
about auditing, through Top Management, and the Quality Unit
Internal
Procedure on auditing
(Clear rules!)

20. Internal Quality Audits at Statistics Portugal –(IV)
Principal rules
Top Management defines the Quality/Statistical Audits’ annual plan
No one should audit its own activity
One Audit team needs 2 or 3 auditors
Auditors produce Audit Plans and reports according to Procedures
Improvement actions must be carried out by the audited departments
Improvement actions must be published and followed by the audit teams

21. External Quality Audits at Statistics Portugal
Auditors are Peers (from other statistical Offices)
Basic rules are the same as for internal auditing
Two visits (+/- one week each)
Contacts between auditors and audited teams in the mid period
Auditors work recommendations with audited teams
Results are presented to the board
Action plan is made by audited teams and presented to the board

22. Benefits for Quality Improvement
Some NSI’s have experienced Quality Auditing!
The identification of improvement opportunities in the audited process
The implementation of corrective / preventive actions
The redefinition of procedures
Auditing involves people in the quality program (auditors and audited teams)
Good Recommendations - A look from the outside!
Identification of good practices that can be spread in the organization
Self-assessment and auditing can be combined together
A time to re-think and improve!
A Powerfull Tool!
BUT!!!!
-Time consuming activity
May face resistances
May be dificult to start
- Better results in the long run if systematically implemented

23. Statistics Portugal experience: External Customer Satisfaction Surveys
Preparation of Matrix
Quality dimensions were discussed and defined accordingly to the following structure:
Attributes that compose
“Quality perception, concerning statistical information:
Data pertinence
Credibility
Actuality
Data accessibility
Metadata accessibility
Coherence and Comparability
Geographical disaggregation
Synthetic data
Data punctuality
1. Quality perception, concerning…
1.1. Statistical information
1.2. Products
1.3. Services
2. Value for money
3. Global image
4. Expectations
5. Loyalty/fidelity

24. External Customers « A Matrix / Table
Based on the concepts related to these 5 dimensions the matrix includes questions accordingly to several target groups:
We can use also complementary information from database of customers:
Examples:
Libraries
Researchers
Central Public Administration
Media
Schools
Site
Examples:
Typology of information requested
Customers typology
Customers level of experience
etc

25. External Customers Questionnaire structure and scales
questionnaires are built in an identical way, having the same structure for different target groups
taking into account the different target groups that are to be inquired, questions may vary (although they are linked to the dimensions and its concepts)

26. External Customers Variables Response scale
31 qualitative variables divided into 4 groups:
recognised quality of the statistical information (7 queries) - based on the Eurostat quality dimensions
recognised quality of the product (6 queries)
recognised quality of the service (7 queries)
Statistics Portugal perceived image (11 queries)
Response scale
Based on an ordinal scale with values comprised between 1 and 6, where 1 means full dissatisfaction/disagreement and 6 means full satisfaction/agreement.

27. Data analysis extreme balance responses (EBR)
The analysis of the results was displayed by using the “extreme balance responses” (EBR). This aimed to value more the extreme answers and not the middle scores which tend to represent a less expressive dissatisfaction/disagreement or satisfaction/agreement.
The attributed ponderations were as follows:
EBR = F1*(-1)+ F2*(-0,5)+ F3*(-0,25)+ F4*(0,25)+ F5*(0,5)+ F6*(1)
where FI = Relative frequency of each observed value for each one of the categories I =(1,…,6).
Data became to be referenced in a metric scale:
between -1 and +1, where values next to -1 mean full dissatisfaction/disagreement and values near +1 mean full satisfaction/agreement.

28. External Customers Targets: Aims: Permanent:
Libraries (Statistics Portugal and Universities)
Website; Post-Service
WebInq (Respondants)
Targets:
Occasional:
Media (2006); Researchers (2007)
Business Associations (2009); Schools (2009)
Schools (2010)
To assess quality perceived by external Statistics Portugal costumers
Identification and implementation of improvement actions
Aims: