MEDSTAT IV GSBPM how to use and implement Training Course « GSBPM » Rome, 25 – 27 July 2017 GSBPM how to use and implement Marina Signore Director of Research, Data collection directorate, Istat FACILITY FOR EUROMED DIALOGUE AND EXCHANGE OF BEST PRACTICES MEDSTAT IV Europeaid/136489/DH/SER/MULTI – Lot 1 NOM Prénom Fonction

How to use and implement the GSBPM TABLE O CONTENT How to use the GSBPM The revision of the GSBPM QIs for the GSBPM Implementing GSBPM: the Modernisation Maturity Model and the Roadmap Use of the GSBPM in Istat

http://www1.unece.org/stat/platform/display/GSBPM/GSBPM+v5.0 Why do we need the GSBPM?

How to use the GSBPM?

How to use the GSBPM? Flexible model: can be adapted to the needs of your organisation Personalisation can be introduced but trade-off with comparability As a documentation tool, to rationalise resources, to see gaps and overlapping,…… To integrate other standards Your views?

The revision of the GSBPM Started April 2017 General topics first: overarching processes, relationship to GAMSO, geospatial components Specific topics afterwards: changing in the wording, in descriptions of sub-processes Discussion forum: https://statswiki.unece.org/display/GSBPM/GSBPM+discussion+forum

How to use and implement the GSBPM TABLE O CONTENT How to use the GSBPM? The revision of the GSBPM QIs for the GSBPM Implementing GSBPM: the Modernisation Maturity Model and the Roadmap Use of the GSBPM in Istat

QIs for the GSBPM Version 1.0 - June 2016 contains quality indicators for direct surveys The extensions to administrative data and mixed sources is almost completed. Next version will be released by the end of 2017 Indicators were prepared only for direct surveys but extensions to administrative data and mixed sources are being undertaken http://www1.unece.org/stat/platform/pages/viewpag e.action?pageId=123142969

QIs for the GSBPM Quality management overarching process Quality indicators were developed for each phase (1 to 8) and sub-processes

Types of indicators Quantitative versus qualitative indicators Generic versus specific indicators Setting targets? Relevant Meaningful Reliable

Characteristics of QIs for GSBPM Generic indicators Coherence with existing framework (UN, ESS Q&P indicators,…) Quantitative indicators whenever possible Qualitative indicators yes/no low/medium/large Personalisation of the indicators left to NSIs (targets, categories, formulas,….)

Examples of QIs At sub-process level for each phase of GSBMP For the overarching process of Quality management

Structure of the QIs Quality Dimension Indicator Notes   ………. …………….. …………………. The NQAF dimensions were taken as reference for relating QIs to the corresponding quality dimension but mapping to the ES CoP was indicated only in case of discrepancies.

1 Quality Dimension Indicator Notes Relevance To what extent have stakeholders been identified and included in discussions about statistical needs?   To what extent has relevant supporting documentation been gathered?

2 Quality Dimension Indicator Notes Soundness of implementation When has the data collection technique last been revised/improved? For outputs produced on a regular basis. Corresponds to the appropriate statistical procedures principle in the ES Code of Practice

3 Soundness of implementation Quality Dimension Indicator Notes Soundness of implementation Has the questionnaire been tested using appropriate methods (e.g. questionnaire pre-test, pilot in real situation, in depth - interviews, focus groups, interviewer support,…)? Corresponds to the appropriate statistical procedures principle in the ES Code of Practice

4 Timeliness and punctuality Quality Dimension Indicator Notes Timeliness and punctuality Delay between expected and actual sign-off of collection systems (including data transmission, security, collection management systems, and QC systems) Delay between expected and actual sign-off of collection materials (questionnaire, training materials, etc)

5 Accuracy and reliability Imputation rate Quality Dimension Indicator Notes Accuracy and reliability Imputation rate The indicator is expressed as the ratio of the number of replaced values to the total number of values for a given variable. ESS QPI - A7. Imputation - rate

6 Accessibility and clarity Metadata completeness - rate Quality Dimension Indicator Notes Accessibility and clarity Metadata completeness - rate The rate of completeness of metadata is the ratio of the number of metadata elements provided to the total number of metadata elements applicable.  ESS QPI - AC3. Metadata completeness – rate

7 Accessibility and clarity Quality Dimension Indicator Notes Accessibility and clarity Date of last update of the content of the metadata. - The date of the latest dissemination of the metadata should be specified. - The date on which the metadata element was inserted or modified in the database should be specified.  

8 Quality Dimension Indicator Notes Cost effectiveness Percentage of GSBPM phases and sub-processes for which there were no gaps between planned and attained costs  

QM Quality Dimension Indicator Notes Quality commitment For what proportion of GSBPM sub-processes are standardised corporate solutions used? Extent to which quality indicators, metadata and paradata are compliant to standards  

Developing Qis for GSBPM Potential users and stakeholders were consulted on the GSBPM QIs via a group work session during the Standards-Based Modernisation Workshop, an open consultation on the UNECE website.

Some challenges Categorizing the indicators such as: key indicators; user-oriented versus producer oriented; qualitative indicators versus quantitative indicators; process versus product indicators. is left to the users to categorize the indicators for their own use.

Coherence with Other Frameworks There exist already many different frameworks for assessing and monitoring quality of products and processes UN National Quality Assurance Framework (NQAF) European Statistics Code of Practice ESS Single Integrated Metadata Structure (SIMS) ESS Quality and Performance Indicators (ESS QPI, 2014) National quality assurance frameworks (e.g. Statistics Canada Quality Guidelines)

Comments and questions? How to use them? QIs for the GSBPM provide: A standard framework and a common terminology A process-oriented approach to Quality Management Can be tailored by NSIs according to their needs Comments and questions?

How to use and implement the GSBPM TABLE O CONTENT How to use the GSBPM The revision of the GSBPM QIs for the GSBPM Implementing GSBPM: the Modernisation Maturity Model and the Roadmap Use of the GSBPM in Istat

The Modernisation Maturity Model and the Roadmap TABLE O CONTENT https://statswiki.unece.org/display/RMIMS/Modernisation+Maturity+Model+and+the+Roadmap+for+Implementing+Modernstats+Standards How does a statistical organisation get started, or make further progress with modernisation? The first step is to know the starting point. The Modernisation Maturity Model has been developed as a self-assessment tool for this purpose. It can be used to assess organisational maturity against different criteria and dimensions. The next step is to understand how to progress to higher levels of maturity. The Roadmap provides a set of tools to help with this.

The Modernisation Maturity Model TABLE O CONTENT Introduction to the Modernisation Maturity Model and its Roadmap Modernisation Maturity Model (MMM) MMM Roadmap Tools

The Modernisation Maturity Model TABLE O CONTENT Description of Maturity Levels from 1 to 5 from «initial awareness» to «mature implementation» Description of Dimensions Business; methods; information; applications and technology Self- assessment criteria for each standard GSBPM, GSIM, GAMSO and CSPA

The self-assessment template TABLE O CONTENT GSBPM Maturity Assessment The current version of GSBPM is version 5.0. Version Assessed: Target Maturity to be achieved in           years.   Dimension Current Maturity  (1-5) Target Maturity Key Steps/ Requirements Business Methods Information Applications

The Roadmap TABLE O CONTENT The roadmap presents an iterative way to go through each maturity level. Each instance of the roadmap can be repeated until the next maturity level has been reached. The roadmap indicates those activities and supporting instruments which are applicable at each stage.

How to use and implement the GSBPM TABLE O CONTENT How to use the GSBPM The revision of the GSBPM QIs for the GSBPM Implementing GSBPM: the Modernisation Maturity Model and the Roadmap Use of the GSBPM in Istat

Use of the GSBPM in Istat Istat modernisation process: key aspects A recent use of the GSBPM in the context of Istat modernisation process

Istat new organisation Approximate number of employees: 2,300 persons Organisation chart: Since April 2016 Governing bodies and Permanent Committees Departments/Directorates: Production Corporate support services Administrative support Strategy and planning  

Istat new organisation

Recent application of the GSBPM Mapping of activities of the recently created Directorate for Data Collection The Data Collection Directorate represents Big change in the production process but also a big cultural change for a lot of people Need to better understand who does what: responsibilities, work flows, procedures,… The GSBPM provides an international framework for rationalising the activities and for supporting the transition

GSBPMP Input Istat Business Architecture (similar to GAMSO) and GSBPM The role and activities of the Data Collection Directorate Interviews with heads of division in the Data Collection Directorate GSBPMP

The phases and sub-processes mapped Planning Innovation Execution Current activity

What has been done For each sub-process Definition of tasks Exploitation of the activities (within the task) for Data Collection Directorate - “who does what” Assigning responsibilities for each task among different directorates Highlighting cross-cutting activities (for more than one process)

2.3 Design Collection - Tasks An example of tasks 2.3 Design Collection - Tasks Definition of the collection method Design of collection instruments, questions and response templates Design of any formal agreements relating to data supply Confirmation of legal basis for data collection Design of process-specific provider management systems Re-use of known methods and collection systems …….

Example: 2.3 Design collection

Example: 2.3 Design collection

Example from Statistics Denmark

Example: 3.1 Build Collection Instrument

The questionnaire: 3 operational procedures CAWI CATI/ CAPI PAPI

The mapping to GSBPM Presented to top and middle management in a seminar (December 2016) Sent to all Directorates for comments Revised on the basis of the comments received (March 2017) Version 1.0 adopted by Istat and published on Istat intranet (July 2017)

Lessons learnt Good exercise to clarify tasks and responsibilities within the Data Collection Directorate and between Directorates Way to identify potential gaps, lack of procedures,… Way to support the formalisation of procedures Could be extended to other areas? Can be reviewed in future