Evaluating the quality of vital statistics UN Expert Group Meeting, June 2011 Dr AJ Lanyon Australian Bureau of Statistics 30 June 2011

2 Outline Chapter V: some comments Utility of an assessment framework and guidance tool Example: WHO/UQ HIS-Hub tool Country Developmental Pathways Conclusion?

3 Chapter V: comments Evaluating quality can be challenging and complex Chap V currently provides comprehensive overview of different types of approaches and methods (direct and indirect) Should, in general, be carried forward

4 Chapter V: comments However: –elements of the Chapter are very technical and dense –could benefit from reference to an assessment framework that provides guidance about why, when and how to assess different elements of the quality of vital statistics –could consider a recommendation on how to manage revisions of data and subsequently vital statistics

5 Example framework Work by WHO / UQ HIS-Hub across past 5 years has demonstrated value in improvement of CR and VS systems of: –overarching system assessment framework –assessment guidance tool. Help to: –Identify strengths / areas for improvement –Develop feasible action plans for CR and VS system improvement.

6 Context Tool developed in light of slow progress in a number of countries in strengthening their CRS Framed to empower countries to develop action-based plan with the support of key stakeholders in their CR and VS systems

Framework focus? Reviewing the system underpinning the production, quality and use of birth, death and cause-of-death information Not just focussed on the product or the output i.e. vital statistics Did not include foetal deaths as many countries not able to collect relevant data – could be expanded to 7

Evidence that it works.. Countries who have successfully used the framework now have: –Strong understanding of strengths/weaknesses of their systems –Complete understanding of key stakeholders necessary to support effective system functioning –Action plan in place to achieve functional and sustainable system and quality VS –Intra-agency co-ordinating committee moving forward 8

What’s in the framework? Inputs – L egal framework for CR and VS systems; Registration infrastructure and resources. Processes – Registration practices, coverage and completeness; Death certification and cause-of-death; ICD mortality coding practices. Outputs – Data access, use and quality checks. 9

What’s in the guidance tool? Comprises series of questions for each sub-component Helps investigate how CR and VS systems have necessary inputs and processes in place to produce quality outputs i.e. vital statistics Assists in better description of: –any issues or problems, and –possible strategies to address. 10

Review approach Promotes establishment of an inter- agency review team Required to gather relevant (specified) evidence Basic assumption: –Buy-in of key stakeholders to process and ownership of final action plan –Collaboration = KEY TO SUCCESS 11

Component E1 Data quality and plausibility checks –Levels of fertility and mortality –Causes of death Asks questions like: –Are fertility indicators routinely calculated? –If so, which ones? –What data sources are used as the denominators to calculate rates? –Describe plausibility and consistency checks carried out (gives examples) 12

13 Developmental pathways Rapid assessment version of the tool has been applied in 26 Asia Pacific countries Results help to differentiate between strength of countries CR and VS systems Identified 3 broad groups of countries: –dysfunctional to weak systems: –Functional but inadequate; –Satisfactory (minor adjustments possible).

14 Developmental pathways Analysis of 26 assessments Cross-cutting issues: –Coder qualification and training alongside quality of coding –Routine procedures to check quality of fertility and mortality data produced –Certification practices –Lack of collaboration across government departments.

15 Developmental pathways Analysis also re-affirmed different countries: –different levels of system maturity –different strengths, weaknesses, opportunities and threats, and –different developmental priorities and pathways. Cross-cutting issues could be addressed through regional programmes.

Conclusion Can’t replace technical sophistication required to evaluate quality of VS Can provide a framework to support countries to work through (what can be) a complex and challenging process to understand quality strengths and develop strategies to address issues. 16