HSAG Performance Improvement Training Statistical Testing Presented by Donald Grostic, MS Health Services Advisory Group, Inc. February 14, 2008

Agenda I.Background  Why do statistical testing?  When is statistical testing used?  Which PIP Activities and Evaluation Elements does statistical testing apply? II.What statistical test do I use?  Chi-square test of inference  How is the retention rate numerator and denominator used in a chi-square test?  Website recommendation for Chi-square test  How do I interpret the results of the Chi-square test? III.Question and answer session

Background Why do statistical Testing? –The CMS Protocols require that statistical testing be used to prove that any improvement in rates is real. –Without statistical testing, a PIP would not meet the CMS Protocols.

Background (continued) When is statistical testing used? –After collecting retention rate baseline data (four quarters of data equaling a year end roll up), completing a causal barrier analysis, implemented interventions and completing a first re-measurement (four more quarters of data equaling the second year end roll up). –At this point (after eight quarters of data has been collected and rolled up into two year end rates), statistical testing should be completed to compare the annual roll up baseline retention rate to the annual roll up first re-measurement rate. –Statistical testing would also need to be performed between the annual re-measurement 1 and re-measurement 2.

Background (continued) Which PIP Activities and Evaluation Elements does statistical testing apply? –Activity VIII Evaluation Element 7: Identifies statistical differences between initial measurement and re- measurement. –Activity IX Evaluation Element 4: There is statistical evidence that observed improvement is true improvement. –Activity X Evaluation Element 1: Repeated measurements over comparable time periods demonstrate sustained improvement, or that a decline in improvement is not statistically significant.

Chi-square Test What statistical test should I use? –Retention rate is calculated using a numerator and denominator. When a rate is calculated, the chi-square test is used. –For example if the annual roll up for the baseline year had 20 clients retained out of 42 total clients, the retention rate would be 20/42 = 0.47x100 = 47.6 %. –If in the annual roll up first re-measurement had 60 clients retained out of 78 total clients, the retention rate would be 60/78 = x100 = 76.9%.

Chi-square Test (continued) Hypothetical Retention Rate data showing chi-square test Time Frame (X) Retained Client (Y) Baseline R1 Total Yes (numerator) No Total (denominator) How is the retention rate numerator and denominator used in a chi-square test?

Chi-square Test (continued) Website recommendation for a chi-square test. – Time Frame (X) Retained Client (Y) Baseline R1 Total Yes (numerator) No Total (denominator) Note: Only enter numbers with arrows

Chi-square Test (continued) Chi-square value = P-value =

Chi-square Test (continued) How do I interpret the results of the statistical test? –If the p-value is less than 0.05, then the change in retention rates is statistically significant. –Since the p-value is equal to which is less than 0.05, the conclusions is that there is a statistically significant improvement in the annual roll up retention rates between the baseline (47.6%) and first re-measurement (76.9%).

Questions ?