1. breakout session of the measurement subgroup, CD4
breakout session of the measurement subgroup, CD4.0 TT A review of Statistical capacity assessments
François Fonteneau (PARIS21)

2. Outline
Purpose and methodology
Results
Possible next steps

3. Purpose
Check what Assessments actually measure – and what they mean by capacity.
Measure on NSO and other respondents (on-going).
the response burden
Propose an analytical question bank (next).

4. Method Step 1: Select 14 assessments tools on statistical capacity
1974 questions and indicators total
Step 2: Code each question/indicator to one or two dimensions from CD 4.0 Framework, using OAR
Resulting 2353 relations*
Step 3: Quantitative and qualitative analysis of questions and assessments
We have included fourteen assessments on statistical capacity in our Open Assessment Repository (OAR): the Self-Assessment Guidance Questionnaire from UNECA (SAGQ), the Snapshot (Eurostat), the Country Assessment of Agricultural Statistical Systems in Africa from AFDB (ASSA), the Tool for Assessing Statistical Capacity from US Census Bureau (TASC), the Light Self - Assessment Questionnaire on the implementation of the European Statistics Code of Practice from the European Commission and OECD (Light SAQ), the Statistical Capacity Indicators from the World Bank (SCI), the Generic National Quality Assessment Framework from UNSD (Generic NQAF, the Global Assessment of the National Statistical System from Eurostat and EFTA (GANSS), the Data Quality Assessment Framework for National Accounts Statistics from IMF(DQAF for National Accounts), the Assessing the National Health Information System. An Assessment Tool from HMN (HIS) , African Statistical Development Indicators from UNECA (StatDI), , the Environment Statistics Self-Assessment Tool from UNSD (ESSAT), the Pan-African Statistics Programme: Peer Reviews of NSIS/NSSS in African countries from Eurostat and AUSTAT (PAS), the extra modules added by IDB to the Tool for Assessing Statistical Capacity (TASC v.IDB).

5. Method
US Census Bureau / IADB Tool for Assessing Statistical Capacity (TASC)
UNECA Self-Assessment Guidance Questionnaire from (SAGQ)
UNECA African Statistical Development Indicators from UNECA (StatDI)
Eurostat Snapshot
Eurostat / EFTA Global Assessment of the National Statistical System
GS – FAO/AfDB Country Assessment of Agricultural Statistical Systems in Africa
EC/OECD Light Self - Assessment Questionnaire on the implementation of the European Statistics Code of Practice
Eurostat /AUSTAT Pan-African Statistics Programme: Peer Reviews of NSIS/NSSS in African countries (PAS)
World Bank Statistical Capacity Indicators from the World Bank (SCI)
HMN Assessing the National Health Information System
UNSD Environment Statistics Self-Assessment Tool (ESSAT)
UNSD Generic National Quality Assessment Framework (NQAF)
IMF Data Quality Assessment Framework for National Accounts Statistics (DQAF)
13-14 different assessments.
-different objectives
------measure capacity; measure capacity needs; measure data output vis a vis standards
- Different survey modes: survey; self assessment; peer reviews, etc

6. Results
Assessments focus mainly on organizational skills and knowledge
Power and politics and incentives are marginally assessed
Although most capacity development programmes target individuals, their capacities are marginally assessed
Categories ranked by relevance
We looked at the 14 most commonly used assessments of statistical capacity (including WB Statistical Capacity Indicators) and mapped their questions/indicators to our Capacity Development 4.0 framework. Here are the preliminary conclusions.
36%
12% 2%

7. Altogether, the top five dimensions represent more than half of the questions/indicators.
Methods, practices and quality controls and standards and regulations : build up 35% of the questions
The following three (transparency, laws and reference frameworks and strategic planning) make up another 19%.
The remaining 46 dimensions account for 46% of the sample.
WHAT DOES IT MEAN? - careful in interpretation

8. Crowded: ‘Methods, practices and quality control’
Purpose: Map these questions :
to GSBPM phases
to data quality sub-dimensions
Results: uneven distribution of questions
‘Methods, practices and quality control’ is the most covered dimension within the category organizational Skills & Knowledge and in all the assessments.
The terms of the dimensions ‘Methods, practices and quality control’ broadly refer to the series of steps and processes that statistical organisations follow to decide what to produce; design and implement data collection, process, analysis and dissemination of statistical products; and conduct evaluations of past implementations to improve future ones.
125 questions were selected from seven assessments
CROWDED OVERALL – BUT SUB-DIMENSIONS ORPHANS?

9. Orphan: ’Individual Resources: education and work experience’
We TENTATIVELY proposed questions:
Is the NSO able to attract enough candidates from the national/regional labour market to meet its staffing needs?
What is the minimum educational attainment in statistics or related, required to qualify for a statistician / managerial position in the NSO?
Does the NSO have a policy requiring future hires to have experience working with statistical software in a similar environment?
Does the NSO have a young professionals programme? …
Education refers to “organised and sustained communication designed” to transmit a body of knowledge and bring about learning and development in different subjects or in a specific field (OECD 2001; Merriam Webster, 2017). The educational background, attainment and field of specialisation of graduates, especially at the university level, are important for the NSO to recruit new staff at the relevant positions.
Work experience refers to the experience of working, skills and familiarity with a field of knowledge gained by actual practice in a specific field or occupation.
Balancing the work experience of NSO employees is relevant to ensure successful knowledge transfer from other institutions, such as academia, as well as within the organisation. Striking the right balance between junior and senior staff ensures that knowledge is accumulated in an efficient way.
PB: WHAT IS RELEVANT AND REQUIRED TO BE MEASURED => LINK WITH OPERATIONALIZATION

10. Next steps Further analysis: Further assessments? … which ones?
Further analysis by type of assessments?
Further “crowded category” work?
Further “orphan category” work?
Towards a stat Cap question bank ?
- Further document response burden?
Education refers to “organised and sustained communication designed” to transmit a body of knowledge and bring about learning and development in different subjects or in a specific field (OECD 2001; Merriam Webster, 2017). The educational background, attainment and field of specialisation of graduates, especially at the university level, are important for the NSO to recruit new staff at the relevant positions.
Work experience refers to the experience of working, skills and familiarity with a field of knowledge gained by actual practice in a specific field or occupation.
Balancing the work experience of NSO employees is relevant to ensure successful knowledge transfer from other institutions, such as academia, as well as within the organisation. Striking the right balance between junior and senior staff ensures that knowledge is accumulated in an efficient way.

11. Thank you Francois.Fonteneau@oecd.org www.paris21.org
Education refers to “organised and sustained communication designed” to transmit a body of knowledge and bring about learning and development in different subjects or in a specific field (OECD 2001; Merriam Webster, 2017). The educational background, attainment and field of specialisation of graduates, especially at the university level, are important for the NSO to recruit new staff at the relevant positions.
Work experience refers to the experience of working, skills and familiarity with a field of knowledge gained by actual practice in a specific field or occupation.
Balancing the work experience of NSO employees is relevant to ensure successful knowledge transfer from other institutions, such as academia, as well as within the organisation. Striking the right balance between junior and senior staff ensures that knowledge is accumulated in an efficient way.