1. Measuring Changes in Service in an Established Telemedicine Program Elizabeth A. Krupinski, PhD Arizona Telemedicine Program University of Arizona

2. Rationale When a telemedicine program first begins, it is often sufficient to simply report frequencies of numbers of patients, types of consults, sub- specialties offered etc. Frequencies are very useful and can be used as a straight-forward measure of program success - the higher the numbers, the greater the success.

3. For Example w From its inception in the 2nd Quarter (Qtr) of 1997 through the 1st Qtr of 2000 the Arizona Telemedicine Program (ATP) conducted 1610 telemedicine consults w 60% were Store-Forward (SF) and 40% were Real-Time (RT) interactions w 75% were initial and 25% were follow-up consultations w These frequencies give a general idea of what the ATP has done over the past 3 years

4. Rationale However, as a program becomes more established it is possible to examine these simple frequencies in more sophisticated ways. For example, looking at changes over time can reveal whether consult volume has stabilized or has it continued to grow, whether certain sub- specialties are being used more consistently than others, etc.?

5. Graphing w Graphing is an easy way to illustrate data as a function of time w It is necessary to choose a meaningful unit of time w We typically use yearly quarters Provides enough data for statistical analyses Parallels the seasons, which might affect volume Easily tracks the Fiscal Year schedule

6. For Example Color coding each year helps make trends stand out * represent points of significant change - see ANOVA panel * *

7. Basic Statistical Analyses w There is a wide variety of statistical techniques that can be used to analyze frequency data w Basic Summary Statistics can describe Central Tendency & Dispersion These are useful because they give an overall picture of the data But, because they summarize the data the time variable is lost

8. For Example w An examination of the consults provided by the ATP on a monthly basis reveals: –Mean # consults = 46.00 –Standard Deviation = 21.75 –Median = 54.00 –Inter Quartile Range = 35.00 –Minimum = 7 consults –Maximum = 87 consults –Skew = -0.342 –Kurtosis = -1.023

9. Advanced Statistical Analyses w To examine the data and include the time variable two other statistical techniques are useful: Correlation & Regression Techniques Analysis of Variance (ANOVA) Techniques w More complicated techniques also exist, but the two above can reveal a significant amount of information about changes in data

10. Regression Plot 2nd Qtr 3rd Qtr 1st Qtr 97 98 00 Quarterly Breakdown

11. Regression Statistics w Regression Equation: Y = 12.91 + 4.97 X The value of 4.97 indicates the type (+ or -) & degree of slope of the regression line w Correlation Coefficient: r = + 0.782 Values closer to + 1 indicate a strong linear relationship w Conclusion: The ATP has continued to see a linear increase in the number of consults with each quarter

12. Data Dispersion & Regression w sest y = 13.76 (average dispersion of data around the regression line) w Heteroscedasticity: dispersion around the regression line is not constant Dispersion in the middle of the plot is much greater than at either end New sites were being added to the network so there was a period of significant change and adjustment in the number of consults

13. # Consults by Site Site & Date of First Consult * 1st Qtr 2000 only

14. ANOVA Analysis w Initial omnibus F-test compares between and within variances to determine if there is an overall difference among the quarters w Post-hoc Fisher’s Protected Least Squares Difference Tests determine exactly which quarters differ w Allows for identification of specific points of difference or change

15. For Example* w F = 7.926, df = 11,23 p < 0.0001 w 2nd Qtr 1998 represents the first significant increase in ATP case volume w 1st Qtr 1999 represents another significant point of increase in ATP case volume w Otherwise case volume has been fairly consistent since 3rd Qtr 1998 * see quarterly volume bar graph for illustration

16. # Medical Sub-Specialties w Analyzing the number of sub-specialty consults provided is another valuable measure of program success w Does a program start out offering lots of services then narrow down to a few, does it continue to offer a variety of services over time or is there another pattern of services provided?

17. ATP Top 10 Sub-Specialties

18. # Sub-Specialties Analysis w Basic Statistics Mean # sub-specialties / month = 11.86 Standard Deviation = 3.56 Median = 13.00 Inter Quartile Range = 3.75 Minimum = 5 Maximum = 20 Skew = -0.28 Kurtosis = -0.77

19. Regression Plot Quarterly Breakdown 2nd Qtr 3rd Qtr 1st Qtr 97 98 00

20. # Sub-Specialties / Quarter * * * significant increase

21. Sub-Specialty Analyses w Regression: Y = 8.021 + 0.576 X r = 0.554 sest y = 3.05 The fairly flat slope and moderate r-value suggest a constant relation between quarters & # of sub-specialties w ANOVA: F = 3.758, df = 11,23 p = 0.0036 Post-hoc tests: Significant increase in # of sub-specialties in 1st Qtr 1998 & 1st Qtr 1999

22. Sub-Specialty Conclusions w The ATP has provided consults in 53 different sub-specialty services w Although dermatology and psychiatry are the services provided most often, the ATP has consistently provided consults in about 13 different* sub-specialties each month since the program’s inception *The individual sub-specialties vary each month

23. Summary w The ATP has had a significant and consistent increase in teleconsult volume since the program began w Numbers from the 1st Qtr 2000 suggest the trend will continue w The ATP has maintained its goal of being a multi-specialty telemedicine provider

24. This work was supported by 1) US Dept. Agriculture, Rural Utilities Service Distance Learning and Telemedicine Grant 2) US Dept. Commerce, National Telecommunications and Information Administration TIIAP Grant 3) Office of Rural Health Policy, HRSA Dept. Health & Human Services Rural Telemedicine Grant Program 4) The State of Arizona