1. MEDSTAT IV GSBPM how to use and implement
Training Course
« GSBPM »
Rome, 25 – 27 July 2017
GSBPM how to use and implement
Marina Signore
Director of Research, Data collection directorate, Istat
FACILITY FOR EUROMED DIALOGUE AND EXCHANGE OF BEST PRACTICES
MEDSTAT IV
Europeaid/136489/DH/SER/MULTI – Lot 1
NOM Prénom
Fonction

2. How to use and implement the GSBPM
TABLE O CONTENT
How to use the GSBPM
The revision of the GSBPM
QIs for the GSBPM
Implementing GSBPM: the Modernisation Maturity Model and the Roadmap
Use of the GSBPM in Istat

3.
Why do we need the GSBPM?

4. How to use the GSBPM?

5. How to use the GSBPM?
Flexible model: can be adapted to the needs of your organisation
Personalisation can be introduced but trade-off with comparability
As a documentation tool, to rationalise resources, to see gaps and overlapping,……
To integrate other standards
Your views?

6. The revision of the GSBPM
Started April 2017
General topics first: overarching processes, relationship to GAMSO, geospatial components
Specific topics afterwards: changing in the wording, in descriptions of sub-processes
Discussion forum:

7. How to use and implement the GSBPM
TABLE O CONTENT
How to use the GSBPM?
The revision of the GSBPM
QIs for the GSBPM
Implementing GSBPM: the Modernisation Maturity Model and the Roadmap
Use of the GSBPM in Istat

8. QIs for the GSBPM
Version June 2016 contains quality indicators for direct surveys
The extensions to administrative data and mixed sources is almost completed. Next version will be released by the end of 2017
Indicators were prepared only for direct surveys but extensions to administrative data and mixed sources are being undertaken
e.action?pageId=

9. QIs for the GSBPM Quality management overarching process
Quality indicators were developed for each phase (1 to 8) and sub-processes

10. Types of indicators Quantitative versus qualitative indicators
Generic versus specific indicators
Setting targets?
Relevant
Meaningful
Reliable

11. Characteristics of QIs for GSBPM
Generic indicators
Coherence with existing framework (UN, ESS Q&P indicators,…)
Quantitative indicators whenever possible
Qualitative indicators
yes/no
low/medium/large
Personalisation of the indicators left to NSIs (targets, categories, formulas,….)

12. Examples of QIs At sub-process level for each phase of GSBMP
For the overarching process of Quality management

13. Structure of the QIs
Quality Dimension
Indicator
Notes
……….
……………..
………………….
The NQAF dimensions were taken as reference for relating QIs to the corresponding quality dimension but mapping to the ES CoP was indicated only in case of discrepancies.

14. 1 Quality Dimension Indicator Notes Relevance
To what extent have stakeholders been identified and included in discussions about statistical needs?
To what extent has relevant supporting documentation been gathered?

15. 2 Quality Dimension Indicator Notes Soundness of implementation
When has the data collection technique last been revised/improved?
For outputs produced on a regular basis.
Corresponds to the appropriate statistical procedures principle in the ES Code of Practice

16. 3 Soundness of implementation
Quality Dimension
Indicator
Notes
Soundness of implementation
Has the questionnaire been tested using appropriate methods (e.g. questionnaire pre-test, pilot in real situation, in depth - interviews, focus groups, interviewer support,…)?
Corresponds to the appropriate statistical procedures principle in the ES Code of Practice

17. 4 Timeliness and punctuality
Quality Dimension
Indicator
Notes
Timeliness and punctuality
Delay between expected and actual sign-off of collection systems (including data transmission, security, collection management systems, and QC systems)
Delay between expected and actual sign-off of collection materials (questionnaire, training materials, etc)

18. 5 Accuracy and reliability Imputation rate
Quality Dimension
Indicator
Notes
Accuracy and reliability
Imputation rate
The indicator is expressed as the ratio of the number of replaced values to the total number of values for a given variable.
ESS QPI - A7. Imputation - rate

19. 6 Accessibility and clarity Metadata completeness - rate
Quality Dimension
Indicator
Notes
Accessibility and clarity
Metadata completeness - rate
The rate of completeness of metadata is the ratio of the number of metadata elements provided to the total number of metadata elements applicable. 
ESS QPI - AC3. Metadata completeness – rate

20. 7 Accessibility and clarity
Quality Dimension
Indicator
Notes
Accessibility and clarity
Date of last update of the content of the metadata.
- The date of the latest dissemination of the metadata should be specified.
- The date on which the metadata element was inserted or modified in the database should be specified.

21. 8
Quality Dimension
Indicator
Notes
Cost effectiveness
Percentage of GSBPM phases and sub-processes for which there were no gaps between planned and attained costs

22. QM
Quality Dimension
Indicator
Notes
Quality commitment
For what proportion of GSBPM sub-processes are standardised corporate solutions used?
Extent to which quality indicators, metadata and paradata are compliant to standards

23. Developing Qis for GSBPM
Potential users and stakeholders were consulted on the GSBPM QIs via
a group work session during the Standards-Based Modernisation Workshop,
an open consultation on the UNECE website.

24. Some challenges Categorizing the indicators such as: key indicators;
user-oriented versus producer oriented;
qualitative indicators versus quantitative indicators;
process versus product indicators.
is left to the users to categorize the indicators for their own use.

25. Coherence with Other Frameworks
There exist already many different frameworks for assessing and monitoring quality of products and processes
UN National Quality Assurance Framework (NQAF)
European Statistics Code of Practice
ESS Single Integrated Metadata Structure (SIMS)
ESS Quality and Performance Indicators (ESS QPI, 2014)
National quality assurance frameworks (e.g. Statistics Canada Quality Guidelines)

26. Comments and questions?
How to use them?
QIs for the GSBPM provide:
A standard framework and a common terminology
A process-oriented approach to Quality Management
Can be tailored by NSIs according to their needs
Comments and questions?

27. How to use and implement the GSBPM
TABLE O CONTENT
How to use the GSBPM
The revision of the GSBPM
QIs for the GSBPM
Implementing GSBPM: the Modernisation Maturity Model and the Roadmap
Use of the GSBPM in Istat

28. The Modernisation Maturity Model and the Roadmap
TABLE O CONTENT
How does a statistical organisation get started, or make further progress with modernisation? The first step is to know the starting point.
The Modernisation Maturity Model has been developed as a self-assessment tool for this purpose. It can be used to assess organisational maturity against different criteria and dimensions.
The next step is to understand how to progress to higher levels of maturity. The Roadmap provides a set of tools to help with this.

29. The Modernisation Maturity Model
TABLE O CONTENT
Introduction to the Modernisation Maturity Model and its Roadmap
Modernisation Maturity Model (MMM)
MMM Roadmap Tools

30. The Modernisation Maturity Model
TABLE O CONTENT
Description of Maturity Levels
from 1 to 5
from «initial awareness» to «mature implementation»
Description of Dimensions
Business; methods; information; applications and technology
Self- assessment criteria for each standard
GSBPM, GSIM, GAMSO and CSPA

31. The self-assessment template
TABLE O CONTENT
GSBPM Maturity Assessment
The current version of GSBPM is version 5.0.
Version Assessed:
Target Maturity to be achieved in           years.
Dimension
Current Maturity
 (1-5)
Target Maturity
Key Steps/
Requirements
Business
Methods
Information
Applications

32. The Roadmap
TABLE O CONTENT
The roadmap presents an iterative way to go through each maturity level.
Each instance of the roadmap can be repeated until the next maturity level has been reached.
The roadmap indicates those activities and supporting instruments which are applicable at each stage.

33. How to use and implement the GSBPM
TABLE O CONTENT
How to use the GSBPM
The revision of the GSBPM
QIs for the GSBPM
Implementing GSBPM: the Modernisation Maturity Model and the Roadmap
Use of the GSBPM in Istat

34. Use of the GSBPM in Istat
Istat modernisation process: key aspects
A recent use of the GSBPM in the context of Istat modernisation process

35. Istat new organisation
Approximate number of employees: 2,300 persons
Organisation chart:
Since April 2016
Governing bodies and Permanent Committees
Departments/Directorates:
Production
Corporate support services
Administrative support
Strategy and planning

36. Istat new organisation

37. Recent application of the GSBPM
Mapping of activities of the recently created Directorate for Data Collection
The Data Collection Directorate represents
Big change in the production process but also a big cultural change for a lot of people
Need to better understand who does what: responsibilities, work flows, procedures,…
The GSBPM provides an international framework for rationalising the activities and for supporting the transition

38. GSBPMP Input Istat Business Architecture (similar to GAMSO) and GSBPM
The role and activities of the Data Collection Directorate
Interviews with heads of division in the Data Collection Directorate
GSBPMP

39. The phases and sub-processes mapped
Planning
Innovation
Execution
Current activity

40. What has been done For each sub-process Definition of tasks
Exploitation of the activities (within the task) for Data Collection Directorate - “who does what”
Assigning responsibilities for each task among different directorates
Highlighting cross-cutting activities (for more than one process)

41. 2.3 Design Collection - Tasks
An example of tasks
2.3 Design Collection - Tasks
Definition of the collection method
Design of collection instruments, questions and response templates
Design of any formal agreements relating to data supply
Confirmation of legal basis for data collection
Design of process-specific provider management systems
Re-use of known methods and collection systems
…….

42. Example: 2.3 Design collection

43. Example: 2.3 Design collection

44. Example from Statistics Denmark

45. Example: 3.1 Build Collection Instrument

46. The questionnaire: 3 operational procedures
CAWI
CATI/
CAPI
PAPI

47. The mapping to GSBPM
Presented to top and middle management in a seminar (December 2016)
Sent to all Directorates for comments
Revised on the basis of the comments received (March 2017)
Version 1.0 adopted by Istat and published on Istat intranet (July 2017)

48. Lessons learnt
Good exercise to clarify tasks and responsibilities within the Data Collection Directorate and between Directorates
Way to identify potential gaps, lack of procedures,…
Way to support the formalisation of procedures
Could be extended to other areas?
Can be reviewed in future