1. www.ideasontario.ca Data is your Friend Collecting, Charting, Analyzing, and Interpreting Data to Support Quality Improvement Michael Campitelli and Ruth Croxford QI Epidemiologists, Institute for Clinical Evaluative Sciences (ICES) Doug Mitchell Director Decision Support, Guelph General Hospital Susan Taylor Director, QI Program Delivery, Health Quality Ontario

2. www.ideasontario.ca 2 Relationships with commercial interests: –Grants/Research Support: None –Speakers Bureau/Honoraria: None –Consulting Fees: None –Other: None Faculty: Ruth Croxford

3. www.ideasontario.ca 3 Relationships with commercial interests: –Grants/Research Support: None –Speakers Bureau/Honoraria: None –Consulting Fees: None –Other: None Faculty: Michael Campitelli

4. www.ideasontario.ca 4 Relationships with commercial interests: –Grants/Research Support: None –Speakers Bureau/Honoraria: None –Consulting Fees: None –Other: None Faculty: Susan Taylor

5. www.ideasontario.ca 5 Relationships with commercial interests: –Grants/Research Support: None –Speakers Bureau/Honoraria: None –Consulting Fees: None –Other: None Faculty: Doug Mitchell

6. www.ideasontario.ca 6 Outline Review (45-50 minutes) Bar charts and Pareto charts Scatter plots Run charts and SPC charts Statistical testing between groups Break (10 minutes) Case Studies specific to the health care sector you identify with the most (45-50 minutes) Primary care Long-Term care Acute care

7. www.ideasontario.ca 7 GUELPH GENERAL HOSPITAL – HIP AND KNEE REPLACEMENT IMPROVEMENT INITIATIVE

8. www.ideasontario.ca 8 Abbreviated, anonymized version of the data

9. www.ideasontario.ca 9 HC Data Guide, p 65 (fig 2.28) Tools to learn from variation in data

10. www.ideasontario.ca 10 Bar Charts

11. www.ideasontario.ca 11 Bar Charts

12. www.ideasontario.ca 12 Bar Charts (fictional data)

13. www.ideasontario.ca 13 Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital. Aaron Levenstein, Professor of Business, Baruch College

14. www.ideasontario.ca 14 Pareto Charts

15. www.ideasontario.ca 15 Pareto Charts (fictional data)

16. www.ideasontario.ca 16

17. www.ideasontario.ca 17 Scatter Plots

18. www.ideasontario.ca 18 Scatter Plots

19. www.ideasontario.ca 19 Scatter Plots

20. www.ideasontario.ca 20 Histograms

21. www.ideasontario.ca 21 Bikini #1

22. www.ideasontario.ca 22 Histograms Surgeon % within target MeanMedian A33%176163 B24%154155 C66%6871

23. www.ideasontario.ca 23 HC Data Guide, p 65 (fig 2.28) Tools to learn from variation in data

24. www.ideasontario.ca 24 Run Charts

25. www.ideasontario.ca 25 Run Charts

26. www.ideasontario.ca 26 Run Charts Four (probability-based) rules to identify non-random signals of change in a run chart (Health Care Data Guide, pgs 76 – 85) A trend –Five or more consecutive points all going up or all going down. A shift –Six or more consecutive points either all above or all below the median Too many or too few runs (crossings of the median) –Depends on the number of points on the graph - requires a table An astronomical data point

27. www.ideasontario.ca 27 http://www.qihub.scot.nhs.uk/knowledge-centre/quality-improvement-tools/run- chart.aspx

28. www.ideasontario.ca 28 Run Charts

29. www.ideasontario.ca 29 The Run Chart as a Bikini

30. www.ideasontario.ca 30 Corresponding Shewhart Chart

31. www.ideasontario.ca 31 Statistical Process Control (SPC) Charts

32. www.ideasontario.ca 32 Shewhart Chart Selection Guide HC Data Guide p. 151 (fig 5.1)

33. www.ideasontario.ca 33 Types of Measures – Continuous Variables

34. www.ideasontario.ca 34 Types of Measures – Count Data Requires two columns of data for each time period: the count and the number of “opportunities” The event being counted can occur more than once per “opportunity” Rate of flash sterilizations (flash sterilizations per 100 surgeries)

35. www.ideasontario.ca 35 Types of Measures – Classification Data Requires two columns of data for each time period: the total number of people or events that were observed, and the number of “non-conforming” events. Percent (percent of patients seen within the target time)

36. www.ideasontario.ca 36 Learning from a Shewhart Chart Rules for detecting special cause variation. Annotation Setting and re-setting the baseline

37. www.ideasontario.ca 37 SPC Chart Rules

38. www.ideasontario.ca 38 Baseline Data

39. www.ideasontario.ca 39 First PDSA

40. www.ideasontario.ca 40 New Baseline

41. www.ideasontario.ca 41 PDSA 2

42. www.ideasontario.ca 42 Final Graph

43. www.ideasontario.ca 43 Rare Events (fictitious data)

44. www.ideasontario.ca 44 Rare Events (fictitious data)

45. www.ideasontario.ca 45 Data for Judgement vs. Data for Improvement Measurements towards a target may hide or discourage authentic and sustainable improvement Targets for accountability may focus on what is easily measured rather than what has value (process rather than outcome)

46. www.ideasontario.ca 46 Data for Judgement vs. Data for Improvement (fictitious data)

47. www.ideasontario.ca 47 Data for Judgement vs. Data for Improvement (fictitious data)

48. www.ideasontario.ca 48 Statistical Testing

49. www.ideasontario.ca 49 Statistical testing Statistical testing is a common form of analysis used in clinical research and epidemiological studies Tests the hypothesis that the average/proportion/rate of some outcome in one group of patients is equal to the average/proportion/rate in another group of patients Statistical tests produce a P-value, which represents the likelihood that the observed difference in the outcome between the two groups is due to chance Studies often set the significance level at 0.05, meaning if there is less than 5% chance the observed results are due to chance, we deem the results `statistically significant`

50. www.ideasontario.ca 50 Statistical testing versus QI analysis While used heavily in clinical research and epidemiology, statistical testing is not the analytic method of choice (e.g., the `Gold Standard`) for quality improvement QI involves conducting sequential tests of change over time to some existing process; therefore, it is logical that tracking outcome and process measures over time in an SPC chart would be the preferred method of analysis Performing statistical tests, rather than tracking measures over time, may cause us to claim improvement when none has occurred, or miss improvement when some has occurred.

51. www.ideasontario.ca 51 A medical unit has 40 COPD discharges per month. On July 1, they implement a self-management training program prior to discharge for all patients. There is a statistically significant decrease (p=0.026) in COPD readmission rates after the implementation.

52. www.ideasontario.ca 52 Here are the readmission rates plotted by month. There is an apparent decrease happening throughout the year, perhaps due to other quality improvement initiative. Difficult to tie decrease to the July 1 initiative

53. www.ideasontario.ca 53 Is statistical testing forbidden in QI… All QI projects should strive to track measures over time and use annotated run and SPC charts for analysis of their data Having said that, sometimes it is not feasible to collect data in any other fashion (e.g., satisfaction surveys which are burdensome and time-consuming to complete), and you are stuck with having to do a pre- post comparison The following website has multiple online calculators to help you perform basic statistical tests between 2 groups for averages (means), proportions, and rates: http://www.socscistatistics.com/tests/

54. www.ideasontario.ca 54 Thank You michael.campitelli@ices.on.ca ruth.croxford@ices.on.ca

55. www.ideasontario.ca 55 Delivered in partnership and collaboration with: Funding provided by the Government of Ontario