1. Transparency of Process: Measuring Statistical Quality in the Sustainable Development Goals
Jennifer Park
UNECE
28 June 2018

2. Statistical quality as transparency in process
What makes this model different?
Why are the SDGs a useful case study?
What does the model tell us about quality for the SDGs?
What strategies does the model recommend for future achievement?

3. 1. Why think of transparency of process as a way to assess quality?
Incorporates a dynamic understanding of NSS and ISS (not static nodes in a system but flows between and across systems)
Reflects progress as incremental
Emphasizes value of official statistics as relationships of trust
Orients assessment of quality beyond calculation to communication

4. How is progress measured? Arrows, not Boxes
High-level Political Forum
(July each year)
National
Thematic
Global
(Statistics)
UN SG Global progress report on SDGs
(UN Secretariat)
Financing for Development Forum
Global SDG indicator database (UNSD)
National voluntary reports
(national governments)
Regional reviews:
key findings and lessons
Statistical Annex
Thematic reviews by major groups, international organizations
Major groups and other stakeholders
NGOs
Research
Businesses
Local authorities
Custodian Agencies/International Organizations Databases
National Statistical Offices and Ministries
National SDG strategies
National SDG indicators
Global SDG indicators

5. How do these features help us improve quality?
Improves understanding of (expanding) statistical systems and the unique roles of various (and emerging) actors
Conceptualizes progress as achievable but constant
“Success is not final. Failure is not fatal. It is the courage to continue that counts.” – Winston Churchill
Extends thinking on the value of official statistics by viewing trust as dynamic and relationship-specific
Allows quality to be achieved even when policy decisions differ

6. 2. Why are the SDGs a useful case study?
The overall scope and ambition of the project
Global statistics for 244 global indicators over 15 years
The maturity of statistical indicators for some conceptual areas
61 (26%) indicators designated as Tier 2 (not yet routinely used)
84 (36%) Tier 3 (requiring conceptual work before use); 5 (2%) multiple tiers
The number and range of actors involved
193 countries, 50 custodian agencies, countless stakeholders
The high profile of these statistics to (local), national, and international policy making
Voluntary national reviews and Secretary-General’s annual report

7. SDG Indicator Development Process
5. IAEG-SDG proposes REVISIONS (substantive changes) to indicators UNSC for approval.
Stakeholders
UNSC
4. Custodian agencies, working with stakeholders, propose revisions or refinements to iAEG-SDG
1. NSOs in UNSC create IAEG-SDG to create initial indicator framework
IAEG-SDG
2. NSO staff in iAEG-SDG, with stakeholders and agencies, develop initial indicator list, tier system, and designate custodian agencies and their roles
Custodian agency
3. Tiering designation guides initial work (e.g., conceptualization, piloting, etc.)

9. Progress in SDG measurement--UNECE countries
National coordination
NSOs, other ministries, policy makers, stakeholders
~ 30% have prepared and 19% are preparing national road maps (or workplan)
National assessment of available statistics
Including data available to produce statistics
~ 60% have completed and another 15% are preparing their assessments
National plans to provide and communicate statistics for SDGs
~ 40% use and another 13% plan to use platforms to share SDG statistics
~ 60% have reported national progress through voluntary national reviews
National plans to address data gaps
~ 75% report statistics for 50% or more of tier 1 indicators; about 30% for tier 2

10. Tiers 1 and 2 Indicator Statistics Availability by UNECE Region and Subregion (February 2018)
 6. Building Capacity for SDG Statistics
Recommendation: Consider identifying regional/ subregional priorities for capacity development
Tool: Evaluation of progress, HLG Survey results; GIST
Indicator: Percent of indicators for which countries can report 50% or more of tier 1 indicators, and tier 2 indicators, respectively, by region and subregion. (By definition, we would expect at least 50% of ECE countries to be able to report 50% or more of the statistics under tier 1.)
Data Source: Analysis of IAEG census of tier 1 and 2 indicator statistics availabilty for region and subregion
No information: 10.4 percent. Note, ECE stats pertain to the whole region; they do not show non-EECA and the non SEE countries, so differences among regions are sharper. [I did not receive the underlying data as non-EECA, non-SEE.]
No information: Tier 1: 10.4%; Tier 2: 47.2%
February 25, 2019

11. SDG Indicator Statistics Availability by Region and Subregion ( >=50%) (February 2018)
Orange is ECA (see prior note). Yellow is EECCA, and grey is SEE.
In this chart, Tiers 1 and 2 are calculated together, so it is to be expected that percent countries that report 50% or more for SDG 13 indicators, for example, is much lower than 50%.
It seems that percent of SDG indicators for which there 50% or more countries who can report it seems lower among non EECA and non SEE compared to others. Iin part, this can be due to small sample sizes, However, this could also be based on a difference in interpretation of what it means for data to be available? Statistics to be available?
February 25, 2019

12. Progress in SDG measurement--globally
International coordination
NSOs, custodian agencies, stakeholders, UNSC (IAEG and HLG), HLPF
Increased efforts to map statistical policy making
Global assessment and management of available statistics
IAEG-SDG efforts documenting data and metadata; UNSD global database
Data flows and communicating statistics for SDGs
IAEG-SDG and CCSSA guidance, UNSC data flow discussions
Secretary-General’s annual report; discussion at HLPF
Developing statistical capacity
HLG-PCCB CAP, financing plans, more focused coordination with custodian agencies

14. 3. What have we achieved? Assessing quality through transparency of process
Improved mapping of NSS and ISS as (connected) networks
Methodological work to be done is achievable--incrementally
Improved understanding of data flows among actors
Effective connection with data users—the public and policy makers
Universal need for developing statistical capacity at institutional level

15. What does a process model teach us?
Wider conceptualization of ISS relationships better reflects reality
Recognizes equities and comparative advantage
Methodological work cannot be done quickly enough for policymakers
“Can’t get blood from a stone.” – Jen’s Grandma Balinsky
Underscores differing expectations in roles regarding data flows
“Good fences make good neighbors.” – Robert Frost
Value both national and globally harmonized statistics for global indicators
Different does not necessarily mean wrong. Welcome outside review.
Global statistical capacity is developed at the national level
Effective development is country-led, demand driven and focuses on practical tools to address national priorities.

16. 4. What strategies do the process model suggest for the SDGs—and beyond?
Build communication loops with non-statisticians to meet user needs
Most ISS actors are not (necessarily) statisticians
Manage expectations and herald achievements of incremental progress
Data flows must not only provide accurate data; flows must be efficient and accountable
Only possible through relationships of trust (and transparency)
Promote the utility of both national and globally harmonized statistics
Develop statistical capacity through practical tools; develop demand for statistical capacity by linking statistics to national priorities

17. How does this understanding help us better achieve quality?
Understanding user needs in ISS supports relevance and utility
Protects the scientific process to improve conceptualization and measurement
Quality through accuracy is not enough. Timeliness and assessment of objectivity are essential.
Moves discussion and effort away from choosing one “right” statistic to identifying the statistic (and data) most appropriate for a given use
Centers statistical capacity development in national needs

18. Thank you for your kind attention
Comments or questions are most welcome at
All materials are available on the UNECE public wiki

19. Transparency of Process: Measuring Statistical Quality in the Sustainable Development Goals
Jennifer Park
UNECE
28 June 2018