Effective National Statistical System Assessments: Heading Towards Open Frameworks Results of NSS Assessment Tools Study Mary Strode – Independent Consultant March 2017 – 49th UN Statistical Commission

Questions on NSS Assessments: raised by Study What are we measuring? Why are we measuring it? What are we going to do with the results? How can we develop a more harmonised approach to assessing National Statistical Systems in the light of new developments in data?

Early tools for NSS assessment SCI/Statistical Performance Indicator Data Quality Assessment Framework DQAF Indicators of Statistical Capacity World Bank – Time Series – 10 years – 146 countries IMF Data Quality Tool PARIS21 Benchmarks Levels of DQAF

There are many more tools to chose from… 15 + and rising TASC US Bureau STEP PARIS21 ASDI UNECA Member Peer Reviews OECD /EU IdCA FAO SNAPSHOT EU GLOBAL ASSESSMENT ESS/EFTA ODIN Open Data Watch Data4SDGs Toolbox Partnership for SDGs

What are they measuring? They measure a variety of issues… some several in one tool Data quality Outputs, indicators, openness, timeliness and periodicity Adherence to standards and principles Prerequisites of quality (laws, resources, structure etc.) Compliance with Codes of Practice Monitor Progress of statistical improvement Advocacy – for statistical support Statistical strategies Project design SDG coverage Do they measure statistical capacity? Why do we want to measure it? Impact of TA, design appropriate TA, ability to support Development Goals, trust in the resulting statistics

Four Types of Tool Identified 1. Tools used national authorities to inform statistical national planning and strategic processes 2. Tools for partners who want to ‘invest’ in statistics to inform project design and to discuss with governments ( includes data quality) 3. Tools to assess statistical performance internationally – Global public goods 4. Assess compliance with codes of practice (e.g. European Statistics Code Of Practice)

Tools classified by type P21 Peer Review NSDS tools IdCA Snapshot TASC SCI/SPI STEP ODIN ASDI OECD Peer Review Global Assessment 1. National Planning 2. Project Design 3. Global Monitoring 4. Compliance

What do we mean by capacity? Statistics in the Development Context CAPACITY DEFINITIONS “ability of people, organizations and society as a whole to manage their affairs successfully”. OECD. An essential ingredient in the UNDP capacity development approach is transformation… it must bring about transformation that is generated and sustained over time from within. Transformation of this kind goes beyond performing tasks; If something does not lead to change that is generated, guided and sustained by those whom it is meant to benefit, then it cannot be said to have enhanced capacity, even if it has served a valid development purpose. STATISTICAL CAPACITY Statistical Capacity is a nation's ability to collect, analyze, and disseminate high-quality data about its population and economy. WBI PARIS21 Capacity 4.0… to manage the Data Revolution Surprisingly little guidance is offered on this topic. UN Inspectors evaluating capacity in 2016 provided a model based on the standard UN Generic Capacity Model

UNDP – GENERIC CAPACITY MODEL Adapted by ISWE for Statistics (2016) National regulatory environment Home user demand Freedom of press Rule of law Strength of economy Statistical Laws Codes of Practice Funding Resources Management Business processes Dissemination Quality processes Training Attitude Skills

Tools use different capacity dimensions STEP (GAMSO Structure) SNAPSHOT (ESS CoP) TASC (DQAF & SCI) 1. Vision Strategy for statistics (in national development) 1. Institutional Capacity  2. Planning and Managing 2. Governance & Leadership of NSS 2. Organisation of the NSS (Legal, institutional and strategic foundations) 3. Mapping  4. Questionnaire Content and Testing 3. Manage Strategic Collaboration 4. Capability Management 3. Resources 5. Sampling 5. Corporate Support – Finances, Human Resources, IT, Consumer Management, Buildings, Suppliers etc 4. Quality determinants 6. Field Operations  7. Data Processing 6. GBSPM (dissemination) 5. Relations with users 8. Data Analysis and Evaluation  7. Output indicators Key Indicators 9. Data Dissemination Revised version

Do we have a Theory of Change that works behind our tools? They are all GOOD - but all DIFFERENT Statistical development is related to development – economic, policy, social, regulatory – strength of institutions. SUPPORTING USE Are statistics unique in development - having most of the demand external to the country? We know surprising little about use of statistics in developing countries – why would governments invest in statistics if they don’t use the products? ISWE (UN 2016) spoke of the disconnect between use and production of statistics. In countries with weak demand – we focus on needs of users (dismally small) rather than supporting it.

What do we need to know in order to build capacity to meet future needs? AUGMENTATION OF CAPACITY Capacity is strongly enhanced by donor funding and activity – for TA, field costs etc. – outputs valuable - sustainable capacity? How much more augmentation is viable for SDGs, Big Data etc.? Could this overload fragile systems? HOW WILL COUNTRIES MANAGE NEW DATA SOURCES Should NSOs have stronger curation and quality role? – less data collection? Transferring institutional processes from developed countries may be misplaced. How transferable are these practices? Isomorphic mimicry How is this to be managed in the developing world

How will results of assessments contribute to the changed context for statistics? To what end do we need to develop this capacity? What will be its purpose? More Data? Is this overloading NSS? More Capacity? Can this be done sustainably? Whose capacities need to be developed? NSS – Production? Curation? Quality? Interpretation? Users – how to use this data Other producers – to augment NSS capacity What kinds of capacities need to be developed to achieve the broader development objectives? Technical Productive Demand Regulatory/Governance (e.g. laws for managing Dig Data) Are our tools giving us information to answer these questions? How can we move forward in a harmonised manner? UNDP

Moving Forward We have different frameworks in use by the tools. Different questions – often several hundred per tool – each with different formats and with implicit solutions to capacity problems. Burden and transaction costs on countries significant. Results are often not shared – inefficient & costly How does the measurement support an agreed ‘theory of change’ to support the capacity changes we need. Can this be agreed, globally or regionally?

Do we need a task team?