1. Progress on use of MAST in Renewing Health Kristian Kidholm Odense University Hospital, Denmark

2. Content 1.Introduction to MAST 2.Empirical test of MAST in Renewing Health 3.Problems faced and solutions: 1.How to ensure that studies are well designed? 2.Agreement on use of outcome measures? 3.How to ensure data quality? 4.Agreement on reporting of results? 5.The overall result – how to identify European added value? 4.Use of MAST in other projects

3. 3 If the purposes of an assessment of telemedicine applications are: –To describe effectiveness and contribution to quality of care AND –To produce a basis for decision making Then the relevant assessment is: A multidisciplinary process that summarizes and evaluates information about the medical, social, economic and ethical issues related to the use of telemedicine in a systematic, unbiased, robust manner. MAST – definition of assessment

4. 4 If the purposes of an assessment of telemedicine applications are: –To describe effectiveness and contribution to quality of care AND –To produce a basis for decision making Then the relevant assessment is: A multidisciplinary process that summarizes and evaluates information about the medical, social, economic and ethical issues related to the use of telemedicine in a systematic, unbiased, robust manner. MAST – definition of assessment Based on HTA (EUnetHTA) Based on scientific methods and studies

5. 5 Preceding assessment: Are you the right one to do the assessment right now? Eg. Legal issues, reimbursement, maturity, number of patients Multidisciplinary assessment (domains): 1. Health problem and characteristics of the application 2. Safety 3. Clinical effectiveness 4. Patient perspectives 5. Economic aspects 6. Organisational aspects 7. Socio-cultural, ethical and legal aspects Transferability assessment: Cross-border Scalability Generalizability Elements in MAST STEP 1: STEP 2: STEP 3:

6. 6 The design should reflect the aim: Estimate effectiveness and contribution to quality Use highest possible level of evidence (Davies and Newman, 2011) Safety, clinical, economic and patient outcomes: - Cluster RCT - Pragmatic RCT If RCT is not feasible: Use other designs: –Quasi-experimental comparison of different hospitals/units –Uncontrolled “before and after” studies Step 2: Design and methods for data collection Risk of systematic differences Follow guidelines for data collection, analysis and reporting

7. Strengths and weakness Weaknesses of MAST: Time consuming Focused on outcomes Only relevant in assessment of matured telemedicine applications. Strengths of MAST: Based on the requests and comments from stakeholders Multidisciplinary and comprehensive Based on scientific studies and criteria for quality Based on HTA (EUnetHTA): Familiar to stakeholders in EU, hospitals..

8. The result of MAST for decision makers? 1. Problem, Application 2. Safety 3. Clinical 4. Patient 5. Economic 6. Organi- zational 7. Socio- cultural DescribeEvidence? Outcome? Evidence? Outcome? Evidence? Outcome? Evidence? Outcome? Evidence? Outcome? Evidence? Outcome? 12345671234567 1234567

9. Empirical test of MAST in Renewing Health EC project: Renewing Health Objective: –Large scale, real life implementation of telemedicine services –Assessment of outcomes based on MAST Budget: 14 mill EURO Pilots: –20 pilots in 9 European regions (I, DK, S, N, ES, GR, D, A, FIN) –Patients: 7.900 patients with COPD, diabetes, CHF

10. Example: Assessment of the COPD suitcase in Denmark Safety Clinical effectiveness DESIGN: RCT, Patient perspectives similar for intervention and control group (n = 266) Economic aspects Organisational aspects Mortality FEV1, SAT, MRC, BMI SF-36 Exercise Investments Number of consultations Number of telephone calls Number of readmissions, bed days Number of outpatient visits Number of home nurse visits Use of emergency ward Changes in revenue (DRG) WSD acceptability questionnaire Qualitative interviews Interview with nurses on task shifts, use of time, satisfaction etc. Transferability assessment: Comparison of DK, E, GR

11. Individual pilots in Renewing Health Diabetes patients Cluster 1:Medium term monitoringSNSFA Cluster 2:Long term monitoringDGR Cluster 3:Ulcer monitoringDK COPD patients Cluster 4:Short term, after dischargeGRESDK Cluster 5:Long term monitoringIAD Heart patients Cluster 6: Medium term monitoringSSF Cluster 7:Remote monitoring CHFIGR Cluster 8: Remote monitoring ICDI(DK) Cluster 9: Remote monitoring ACTI

12. Problems and solutions 1.How to ensure that studies are well designed? Almost all studies are pragmatic RCT’s Common scientific protocols (CONSORT, MAST): 1.Oobjectives and the trial type 2.Planned sample sizes 3.Trial start and end dates 4.Eligibility criteria 5.Enrolment modalities 6.Description of the randomisation methodology 7.Demographic and clinical baseline characteristics 8.Interventions 9.Primary and secondary outcomes 10.Evaluation time points 11.Economical evaluation 12.Evaluation of perception of health care professionals 13.Evaluation of patient satisfaction 14.Additional evaluations 15.Statistical analysis

13. Problems and solutions 2. Agreement on use of outcome measures? Similar primary outcomes within each cluster Minimum dataset in all studies Demographic data: Based on WHO project STEPS Clinical effectiveness: Health related quality of life – SF36 Patient perception: WSD patient acceptability questionnaire

14. Problems and solutions 2. Agreement on use of outcome measures? Similar primary outcomes within each cluster Minimum dataset in all studies Economic aspects: Investments in the telemedicine application Running costs of delivering telemedicine and comparator: Each patient's use of health care service: Number of admissions Number of bed days Number of GP visits Number of visits to emergency department Reimbursement of the telemedicine service (business case)

15. Problems and solutions 2. Agreement on use of outcome measures? Similar primary outcomes within each cluster Minimum dataset in all studies Organizational aspects: Effects on work processes: –Workflow and task shifting –Training –Communication Effects on structural outcomes: –Description/number of units collaborating –Changes in organisation –Changes in geographical spread Cultural outcomes: –Staff attitudes towards the application Common list of questions

16. Problems and solutions 3. How to ensure data quality? 1. Similar coding of common variables 2. Collection of CRF from all pilots 3. Assistance in development of Epidata Databases 4. Monthly monitoring of data collection in each pilot Based on GCP Number of patients Completeness

17. Problems and solutions 3. How to ensure data quality?

18. Problems and solutions 4. Agreement on reporting of results? Guide for analysis of results within each domain: 1. Health problem and characteristics of the application 2. Safety (adverse effects) 3. Clinical effectiveness 4. Patient perspectives 5. Economic aspects 6. Organisational aspects 7. Socio-cultural, ethical and legal aspects Based on STARE-HI + CONSORT Based on validation studies by WSD Based on guide by Drummond et al 2005 Based on guide for organizational studies

19. Problems and solutions 5. Comparison of results – how to identify European added value? Meta-analysis of results from pilots within a cluster Study Outcome RH No 1 RH No 2 RH No 3 Other No 4 No. Of patients10020060032 Clinical outcome- 2%+ 1%+5 %*-1% SF-36 dimension X (0-100)+ 5+ 10 *+ 8 *-2 Patient acceptability (0-100)-10+ 6+ 8* Readmissions 0,1-1,2- 2,3*0,3 Costs per patient (€)220015008502300 Interpretation: - Some show positive effect, others negative - Some statistically significant, others not - Some studies are large, others not

20. Problems and solutions 5. Comparison of results – how to identify European added value? Meta-analysis of results from pilots within a cluster A statistical analysis of results from independent studies, which aims to produce a single estimate of a treatment effect Account can be taken of heterogeneity of the studies: Inclusion of characteristics of patients (meta level or individual patient-data)

21. Problems and solutions 5. Comparison of results – how to identify European added value? Meta-analysis of results from pilots within a cluster Reasons for meta-analysis: Obtain more precise estimate of the effect of an intervention Increase the statistical power (ability to find stat. significant effects) Increase generelisability Increase possibilities for subgroup analysis Used by Cochrane Collaboration, NICE etc.

22. Use of MAST in other projects Jose Asua Batarrita, INAHTA Board Director, at WoHIT 2011. Approach used in “...a number of telemedicine projects related to home telecare of frail elderly patients with heart failure and COPD, teledermatology, teleoncology and teleophthalmology “

23. Use of MAST in other projects MAST is recommended as the assessment model in the national telemedicine strategy by the Association of Danish Regions Renewing Health is also collaborating with inCASA:

24. www.renewinghealth.eu Questions? Kristian.kidholm@ouh.regionsyddanmark.dk