2. Monitoring the MDGs A catalogue of procedures and an assessment of statistical capacity Part 1: International study

3. Overview Objective of the study The method used The main findings

4. The catalogue For each of the 48 MDG indicators the catalogue specifies: 1.Agency responsible for reporting international statistics 2.Definition of the indicator 3.Availability and timeliness of data 4.Original data sources 5.Construction of the indicator and/or process of reporting 6.Checks performed on the original sources 7.Comparison with other international data sources 8.Notes on websites holding the indicator data This exercise has illustrated various shortcomings of the current dataset, and it also highlighted possible areas for immediate improvement.

5. Example from the indicator catalogue

6. Main shortcomings: 1.Definitional Issues 2.Data availability 3.Issues that arise from modelling exercises 4.Timeliness 5.Comparability & Differences in Calculation Methods

7. Key Issues to Address: Appropriate use of available data from household surveys Use by lead organisations of data reporting questionnaires Use of international population data Management of common methodologies and definitions Data management practices

8. Appropriate use of household surveys: 1/3 of indicators depend on data from household surveys Monitoring Goals 1, 4, 5, 6, and 7 depend almost exclusively on household surveys However an expansion of HH Survey roles could result because: –There are surveys escaping the screening system Examples – Integrated Household Survey in Malawi; PIHS in Pakistan (p.12) –There is scope for using HH Surveys to overcome problems of old or missing data (p.13) Examples – Education Indicators; Literacy Indicators …but…large management & coordination problems to address consistency and timeliness issues.

9. Lead agency use of questionnaires: A number of lead agencies use questionnaires sent to governments for data reporting on the following indicators (p.15): –Education indicators –Water and sanitation indicators –Immunization coverage indicators Quality of these data does not always satisfy the required standards because of the: –Inability to control the process –Absence of direct contact

10. Use of international population data: International population data are used in two ways: –For regional and global estimates (weights) –In the construction of country indicators The combining different data sources in the construction of single country estimates generates inconsistencies in the following indicators: –Net enrolment ratio –Literacy rates –Water and sanitation –…..an example….

11. The NER in Albania is: 100% (according to international sources) 81% (according to national sources) What explains this difference? The population aged 5-9 is: 320 thousands (according to international sources) 400 thousands (according to national sources) 80%*400=320!

12. Management of common methodologies and definitions: Lack of clarity in the definition of some indicators: –For example in the HIV/AIDS indicators (footnote 5, p.9) Clear guidelines, but lack of coordination in the implementation: –For example in the definition of literacy (p. 18) Adjustment of methodologies and definitions to the variety of country realities: –For example: immunization in Vietnam

13. Proportion of births attended by skilled health personnel CountryWHOUNICEFOriginal Source Paraguay58.1 (1998)70.9 (1998)ENSMI (1999) China89.3 (1995)70 (1999) 1) Western Pacific Regional Office 2) Chinese Health Statistical Digest

14. Data Management issues: A number of problems/difficulties have been noticed in the way in which data is presented by lead agencies: Absence of detailed explanatory notes on (p.18): –Sources of data on which estimates are based –Explanation of any dramatic changes in the data time series Use of year-ranges for data presentation (p.19) Coordination in updating web pages, and absence of information on when the last update took place

15. Further issues to explore: MANAGEMENT –Issue of accountability and definition of responsibilities between the various international agencies and countries DATA COLLECTION –Further review of the scope for improving data from agency questionnaires –An assessment of the general quality of databases of household surveys and hh surveys more generally METHODS –Systematic listing of inconsistencies in methodologies –A review of inconsistencies in population data –How current methodologies for education and literacy indicators could integrate information from household surveys ANALYSIS –Review case for broadening scope of re-analysis of raw data