1. Performance Monitoring Public Health Intelligence training course day 5 Liz Rolfe Knowledge & Intelligence Team (South West)

2. Learning Objectives At the end of the course, participants will be able to... explain what is meant by performance monitoring identify the drivers, purpose and applications of performance monitoring illustrate how performance monitoring is undertaken in practice 2Performance Monitoring

3. Performance monitoring Executives worth their salt will be asking... are we achieving our objectives? are we meeting our targets? what should our priorities be? what’s our strategy for....? what changes do we need to make to local services? where and how should we target our resources? are our initiatives and service changes having the intended effect? 3Performance Monitoring

4. Performance monitoring But just in case they don’t there is legislation and performance management... LAs have a statutory duty to undertake JSNA and provide a joint health and wellbeing strategy LA Health & Wellbeing Boards are a statutory committee The Public Health Outcomes Framework sets out the key indicators the Department of Health expects local authorities to work towards. Regulations on the exercise of local authority public health functions include child height & weight measurement, health checks, open access sexual health services, provision of public health advice service, and health protection arrangements. 4Performance Monitoring

5. Drivers and purpose Demand for accountability of public services; “Clinical governance”. Quantifying and qualifying the results of surveillance. Encourage best practice; identify opportunities for development and shared learning. Clarify an organisation’s objectives and priorities. Measuring progress against targets. 5Performance Monitoring

6. Assessing Performance 6Performance Monitoring

7. Monitoring and evaluating play an important role Definition? Measuring and comparing observed data with a benchmark or baseline Comparisons can be over time (e.g. one year to the next) between areas (e.g. local authorities) against a target (e.g. 98%) against an expected value 7Performance Monitoring

8. Choosing an indicator – The good indicators guide A poorly designed indicator with reliable data, or a well designed indicator with unreliable and/or untimely data, has very little value (and is sometimes dangerous). 8Performance Monitoring

9. The good indicators guide – 10 key questions 1.What is being measured? 2.Why is it being measured? 3.How is the indicator defined? 4.Who does it measure? 5.When does it measure it? 6.Will it measure absolute numbers or proportions? 7.Where do the data actually come from? 8.How accurate and complete will the data be? 9.Are there any caveats/warnings/problems? 10.Are particular tests needed (e.g. standardisation, significance tests or SPC) to test the meaning of the data and the variation they show? 9Performance Monitoring

10. The good indicators guide Selection of indicators and the data that feed into them will often involve a trade off between what is convenient (and possible) and what is ideal. The right indicator at the right time and in the right context can stimulate huge change where it might have no effect in other contexts and at other times. Change hearts and minds Try not just to present the data – look for the story it is telling. 10Performance Monitoring

11. Does this tell you enough? 11Performance Monitoring Sources: Office for National Statistics and Teenage Pregnancy Unit

12. Comparators give context 12Performance Monitoring Sources: Office for National Statistics and Teenage Pregnancy Unit

13. Ways of evaluating performance Spine chart Tartan rug Benchmarking Statistical Process Control o Control and run charts o Funnel plots 13Performance Monitoring

14. Ways of evaluating performance 14Performance Monitoring *Has two dimensions and so can be displayed differently e.g. Many indicators, one organisation and overtime; or one indicator and many organisations over time.

15. Spine Chart 15Performance Monitoring

16. Tartan rug 16Performance Monitoring

17. Benchmarking “an externally-agreed comparator to compare performance between similar organisations and systems”, Good Indicator Guide, APHO Contextualise performance Encourages best practice Shared learning and areas for development 17Performance Monitoring

18. Benchmarking 18Performance Monitoring Which local authority would you say had an issue with perinatal mortality? Can you think of any issues regarding presenting data in a league table such as this?

19. Statistical Process Control ‘common cause variation’ the normal, everyday, inevitable (and usually unimportant), variation which is intrinsic and natural to any system ‘special cause variation’ the more important variation which is indicative of something special happening and which calls for a fuller understanding and often action SPC distinguishes between common cause and special cause variation 19Performance Monitoring

20. Types of control chart Conventional control charts (and run charts) The indicator of interest is plotted on the y-axis, against time or the unit of analysis on the x-axis. Control charts can be plotted with small numbers of data points although their power is increased with more data.. 20Performance Monitoring Control charts are plots of the data with lines indicating the target value (mean, median) and control limits superimposed.

21. Types of control chart Funnel plots A type of chart where the indicator of interest is plotted against the denominator or sample size. This gives it the characteristic funnel shape 21Performance Monitoring

22. Using control charts and SPC methods Control charts can help us to present and interpret our information more intelligently. They can be used: To detect unusual or outlying patterns, e.g. poor performance, outbreaks or unusual patterns of disease In health profiling and assessing levels of performance To decide whether or not targets are being met In assessing health inequalities (i.e. funnel plots) 22Performance Monitoring

23. Run and Control Charts Run chart – displays any data over time against its median Control chart – looks similar to a run chart but plots data against its mean and the appropriate control limits Good for monitoring trends over time Good for monitoring data against a specific target o A&E 4 hour waits o Childhood obesity 23Performance Monitoring

24. Run Chart 24Performance Monitoring

25. Control chart showing A&E 4 hour waits: 25Performance Monitoring Consistently not meeting the target Source: Good indicator guide Absolute target

26. Control chart A&E 4 hour waits: change in target 26Performance Monitoring Green line = target Red lines = CIs

27. Funnel plot - Perinatal Mortality 27Performance Monitoring

28. Breast cancer mortality 28Performance Monitoring APHO technical briefing 2: Statistical process control methods in public health intelligence

29. Lung cancer mortality 29Performance Monitoring APHO technical briefing 2: Statistical process control methods in public health intelligence

30. Summary Good monitoring and benchmarking requires use of comparators Good data visualisation is helpful Statistical process control allows you to distinguish between common cause variation and special cause variation, which might indicate a problem or an exceptional performance within the system. 30Performance Monitoring

31. Further information The Good Indicator Guide: how to use and choose performance indicators http://www.apho.org.uk/resource/item.aspx?RID=44584 http://www.apho.org.uk/resource/item.aspx?RID=44584 APHO Technical Briefings: http://www.apho.org.uk/resource/view.aspx?RID=39306 No. 2: Statistical process control methods in public health intelligence No. 3: Commonly used public health statistics & their confidence intervals No. 4: Target setting in a multi-agency environment Health Profiles: http://www.healthprofiles.info 31Performance Monitoring

32. Further information APHO tools: Funnel plots for count data, proportions and percentages, rates (including directly standardised rates) and indirectly standardised ratios Commonly used public health statistics and their confidence intervals Both available from http://www.apho.org.uk/default.aspx?RID=39403http://www.apho.org.uk/default.aspx?RID=39403 DSR batch calculator http://www.erpho.org.uk/statistical_tools.aspxhttp://www.erpho.org.uk/statistical_tools.aspx Spine chart creator, including new spine chart based on SPC limits http://www.wmpho.org.uk/tools/ http://www.wmpho.org.uk/tools/ 32Performance Monitoring

33. Learning Objectives At the end of the course, participants will be able to... explain what is meant by performance monitoring identify the drivers, purpose and applications of performance monitoring illustrate how performance monitoring is undertaken in practice 33Performance Monitoring