1. QUASI-EXPERIMENTAL STUDY DESIGNS IN EVALUATING MEDICINES USE INTERVENTIONS 1 Lloyd Matowe 2 Craig Ramsay 1 Faculty of Pharmacy, Kuwait University 2 HSRU, University of Aberdeen

2. Problem Statement: Reliable and valid information on the effectiveness and cost-effectiveness of various interventions for improving use of medicines is required if policy makers are to make decisions based on these interventions. Rigorous research can help provide such information. The randomized controlled trial (RCT) is seen as the gold standard methodology for the evaluation of health care interventions. However, there are many interventions for which it is impossible or impractical to use RCTs. In such cases quasi-experimental designs are often used. Objective: To evaluate the use of quasi-experimental designs in assessing interventions to improve use of medicines. Design: A rigorous systematic review using the Cochrane Effective Practice and Organization of Care (EPOC) strategy to assess interventions to change medicines use. Outcome Measures: Appropriateness of design and analysis compared to the EPOC criteria. Results: Fifteen interrupted time series studies and twelve controlled before and after studies were identified. 16 studies were reanalyzed. More than 50% of the studies were inappropriately analysed. Conclusion: Quasi-experiments appear superficially simple, but they are often analyzed inappropriately and poorly reported. To improve the quality of design and analysis, we suggest the use of quality criteria that we developed. ABSTRACT

3. BACKGROUND Reliable and valid information on the effectiveness and cost-effectiveness of various interventions for improving medication use is required if policies are to be formulated based on these. The RCT is seen as the gold standard methodology for evaluating interventions. However there are many situations where it is impractical to carry out a randomised controlled trial. Quasi-experiments are often used as an alternative. There is suspicion that these are often conducted and analysed inappropriately.

4. OBJECTIVE The objectives of this study is to evaluate the use of quasi-experimental designs in medicines use interventions studies.

5. METHODS - 1 A rigorous systematic review Subset of a large study on the effectiveness of various guideline implementation strategies Search strategies –Cochrane Effective Practice and Organisation of care –Medline –EMBASE –HEALTHSTAR –SIGLE –Bibliographies of previously published reviews

6. METHODS - 2 Included studies: Controlled before and after and interrupted time series analysis Participants: Health care professionals Interventions: policies/ guidelines to improve medication use Outcomes- Objective measures of provider behaviour and/ or patient outcome Time series regression used to reanalyse time series data where reanalysis was necessary

7. RESULTS - 1 Search strategy identified 150 000 hits for the main implementation project 5 000 hits identified as potentially relevant and were evaluated 285 studies included 44 studies were on medicines use 17 were controlled clinical trials or randomised controlled trials 27 studies were quasi-experiments –15 Interrupted Time Series studies –12 Controlled before and after studies

8. RESULTS - 2 24 of 27 studies were multiple interventions Studies were in 5 main areas Area of interest# of studies Antibiotics6 Antipsychotics5 Pain relief medicines4 GIT drugs4 Hypertensives3 Others5

9. RESULTS - 3 Time series Overall methodological quality poor 8 of 15 studies analysed inappropriately Effect size different in 6 of the 8 studies on reanalysis Common errors included: -Failure to test for autocorrelation and trend -Analysis using non time series methods -Failure to ascertain intervention as independent from other interventions -Failure to report methodological weakness

10. RESULTS - 4 Controlled before and after studies Overall methodological quality poor 8 of 12 had unit of analysis errors Common errors included: -Failure to protect against contamination -Failure to measure record baseline data -Failure to perform power calculation -Failure to report methodological weakness

11. SUMMARY Quasi-experiments are often designed and analysed inappropriately Methodologies and design are often inadequately reported

12. RECOMMENDATIONS Time series analysis –We recommend use of time series regression methods over ARIMA or other methods for data analysis –Autocorrelation and trend should be tested for –Researchers should ascertain that the intended intervention was independent of other interventions –For both ITS and CBAs methodological weaknesses should be reported

13. RECOMMENDATIONS CBAS –Researches should protect against contamination –Baseline characteristics should be measured –Power calculation should be performed