breakout session of the measurement subgroup, CD4 breakout session of the measurement subgroup, CD4.0 TT A review of Statistical capacity assessments François Fonteneau (PARIS21)

Outline Purpose and methodology Results Possible next steps

Purpose Check what Assessments actually measure – and what they mean by capacity. Measure on NSO and other respondents (on-going). the response burden Propose an analytical question bank (next).

Method Step 1: Select 14 assessments tools on statistical capacity 1974 questions and indicators total Step 2: Code each question/indicator to one or two dimensions from CD 4.0 Framework, using OAR Resulting 2353 relations* Step 3: Quantitative and qualitative analysis of questions and assessments We have included fourteen assessments on statistical capacity in our Open Assessment Repository (OAR): the Self-Assessment Guidance Questionnaire from UNECA (SAGQ), the Snapshot (Eurostat), the Country Assessment of Agricultural Statistical Systems in Africa from AFDB (ASSA), the Tool for Assessing Statistical Capacity from US Census Bureau (TASC), the Light Self - Assessment Questionnaire on the implementation of the European Statistics Code of Practice from the European Commission and OECD (Light SAQ), the Statistical Capacity Indicators from the World Bank (SCI), the Generic National Quality Assessment Framework from UNSD (Generic NQAF, the Global Assessment of the National Statistical System from Eurostat and EFTA (GANSS), the Data Quality Assessment Framework for National Accounts Statistics from IMF(DQAF for National Accounts), the Assessing the National Health Information System. An Assessment Tool from HMN (HIS) , African Statistical Development Indicators from UNECA (StatDI), , the Environment Statistics Self-Assessment Tool from UNSD (ESSAT), the Pan-African Statistics Programme: Peer Reviews of NSIS/NSSS in African countries from Eurostat and AUSTAT (PAS), the extra modules added by IDB to the Tool for Assessing Statistical Capacity (TASC v.IDB).

Method US Census Bureau / IADB Tool for Assessing Statistical Capacity (TASC) UNECA Self-Assessment Guidance Questionnaire from (SAGQ) UNECA African Statistical Development Indicators from UNECA (StatDI) Eurostat Snapshot Eurostat / EFTA Global Assessment of the National Statistical System GS – FAO/AfDB Country Assessment of Agricultural Statistical Systems in Africa EC/OECD Light Self - Assessment Questionnaire on the implementation of the European Statistics Code of Practice Eurostat /AUSTAT Pan-African Statistics Programme: Peer Reviews of NSIS/NSSS in African countries (PAS) World Bank Statistical Capacity Indicators from the World Bank (SCI) HMN Assessing the National Health Information System UNSD Environment Statistics Self-Assessment Tool (ESSAT) UNSD Generic National Quality Assessment Framework (NQAF) IMF Data Quality Assessment Framework for National Accounts Statistics (DQAF) 13-14 different assessments. -different objectives ------measure capacity; measure capacity needs; measure data output vis a vis standards - Different survey modes: survey; self assessment; peer reviews, etc

Results Assessments focus mainly on organizational skills and knowledge Power and politics and incentives are marginally assessed Although most capacity development programmes target individuals, their capacities are marginally assessed Categories ranked by relevance We looked at the 14 most commonly used assessments of statistical capacity (including WB Statistical Capacity Indicators) and mapped their questions/indicators to our Capacity Development 4.0 framework. Here are the preliminary conclusions. 36% 12% 2%

Altogether, the top five dimensions represent more than half of the questions/indicators. Methods, practices and quality controls and standards and regulations : build up 35% of the questions The following three (transparency, laws and reference frameworks and strategic planning) make up another 19%. The remaining 46 dimensions account for 46% of the sample. WHAT DOES IT MEAN? - careful in interpretation

Crowded: ‘Methods, practices and quality control’ Purpose: Map these questions : to GSBPM phases to data quality sub-dimensions Results: uneven distribution of questions ‘Methods, practices and quality control’ is the most covered dimension within the category organizational Skills & Knowledge and in all the assessments. The terms of the dimensions ‘Methods, practices and quality control’ broadly refer to the series of steps and processes that statistical organisations follow to decide what to produce; design and implement data collection, process, analysis and dissemination of statistical products; and conduct evaluations of past implementations to improve future ones. 125 questions were selected from seven assessments CROWDED OVERALL – BUT SUB-DIMENSIONS ORPHANS?

Orphan: ’Individual Resources: education and work experience’ We TENTATIVELY proposed questions: Is the NSO able to attract enough candidates from the national/regional labour market to meet its staffing needs? What is the minimum educational attainment in statistics or related, required to qualify for a statistician / managerial position in the NSO? Does the NSO have a policy requiring future hires to have experience working with statistical software in a similar environment? Does the NSO have a young professionals programme? … Education refers to “organised and sustained communication designed” to transmit a body of knowledge and bring about learning and development in different subjects or in a specific field (OECD 2001; Merriam Webster, 2017). The educational background, attainment and field of specialisation of graduates, especially at the university level, are important for the NSO to recruit new staff at the relevant positions. Work experience refers to the experience of working, skills and familiarity with a field of knowledge gained by actual practice in a specific field or occupation. Balancing the work experience of NSO employees is relevant to ensure successful knowledge transfer from other institutions, such as academia, as well as within the organisation. Striking the right balance between junior and senior staff ensures that knowledge is accumulated in an efficient way. PB: WHAT IS RELEVANT AND REQUIRED TO BE MEASURED => LINK WITH OPERATIONALIZATION

Next steps Further analysis: Further assessments? … which ones? Further analysis by type of assessments? Further “crowded category” work? Further “orphan category” work? Towards a stat Cap question bank ? - Further document response burden? Education refers to “organised and sustained communication designed” to transmit a body of knowledge and bring about learning and development in different subjects or in a specific field (OECD 2001; Merriam Webster, 2017). The educational background, attainment and field of specialisation of graduates, especially at the university level, are important for the NSO to recruit new staff at the relevant positions. Work experience refers to the experience of working, skills and familiarity with a field of knowledge gained by actual practice in a specific field or occupation. Balancing the work experience of NSO employees is relevant to ensure successful knowledge transfer from other institutions, such as academia, as well as within the organisation. Striking the right balance between junior and senior staff ensures that knowledge is accumulated in an efficient way.

Thank you Francois.Fonteneau@oecd.org www.paris21.org Education refers to “organised and sustained communication designed” to transmit a body of knowledge and bring about learning and development in different subjects or in a specific field (OECD 2001; Merriam Webster, 2017). The educational background, attainment and field of specialisation of graduates, especially at the university level, are important for the NSO to recruit new staff at the relevant positions. Work experience refers to the experience of working, skills and familiarity with a field of knowledge gained by actual practice in a specific field or occupation. Balancing the work experience of NSO employees is relevant to ensure successful knowledge transfer from other institutions, such as academia, as well as within the organisation. Striking the right balance between junior and senior staff ensures that knowledge is accumulated in an efficient way.