Download presentation
Presentation is loading. Please wait.
Published byAlejandro Anderson Modified over 11 years ago
2
Manual Evaluation: Are we doing what we think we are doing?
Associate Professor Dee Mangin Director Primary Care Research Unit Department of Public Health and General Practice Christchruch School of Medicine, University of Otago You will be wondering what a family doctor from New Zealand is doing… and at the end I am here from the other side of the world because the issue we are trying to address here is global OR Pharmaceutical promotion is the greatest threat globally to rational use of medicines and its influence is increasing Now more than ever we need to to include in the medical and pharmacy school curricula teaching to assist students to understanding the range of promotional strategies and to develop strategies to minimise inappropriate prescribing as a result And up until now there has been neither curriculum time nor resource material for medical and pharmacy schools 20 May 2009 WHA
3
Rationale for testing the manual in a pilot study
Overview Rationale for testing the manual in a pilot study Protocol for testing Timelines What I would like to describe for you in the next 15 minutes is outline the planned pilot study testing of this WHO / HAI teaching manual – covering the reasons why it is essential that we do pilot it and the way we plan to do it. We would really welcome interest from any of you who are associated with medical and pharmacy student teaching who are interested in being involved as a pilot study site
4
Background WHO/HAI have developed a comprehensive teaching package comprising a manual and accompanying resource material as well as training for teachers implementing and this So we have this very comprehensive manual and teaching resource package that Barbara has spoken about
5
Background It is planned to evaluate this in a pilot study to look at its effectiveness and at areas for improvement in the final manual WHY DO THIS STUDY? Why not just send it out? Why do we need to do a pilot study?
6
Background Many educational resources and techniques are implemented without ever having any assessment of their effectiveness Many – dare I say most educational ideas and resources are implemented without piloting – without ever really knowing whether they are effective And also Without knowing whether there is some possibly counterproductive effect Counterproductive? Some of you will think How could providing information be counterproductive In this instance we are implementing a manual in an environment where most physicians and pharmacists acknowledge that people are influenced by promotion, but believe that they individually arent. This is called the illusion of unique invulnerability. It is possible that in this environment, an educational strategy might further increase the confidence of individuals in engaging with the pharmaceutical industry, believing they are even less vulnerable to such influence when armed with knowledge.
7
So we don’t know whether our efforts are effective
So, like this cohort of roman soldiers marching on Often we don’t know whether our efforts are effective
8
Our wish to think that they are can be misleading……
Overview Our wish to think that they are can be misleading…… Many strategies for medical education have little effect on prescribing when they are tested in studies And our desire to think they are can sometimes be misleading Many strategies for medical education have little effect on participants when assessed in a research framework Printed information Lectures …. Like this one Guidelines have limited effect
9
Evidence for prescribing education strategies that have been found effective
The effect of dissemination only strategies such as didactic lectures and written information are small at best and of unknown clinical significance Farmer et al. Cochrane Database of Systematic Reviews 2008;3 Facilitated teaching with evidence based resources and audit and feedback have been found to have a moderate effect Richards et al Family Practice 2000 A previous WHO educational resource (WHO Guide to Good Prescribing) using facilitated teaching for medical students on pharmacotherapy showed a significant effect on students skills in a randomised controlled trial Impact of a short course in pharmacotherapy for undergraduate medical students: An international randomised controlled study. de Vries T Henning R Hogerzeil H Bapna J et al The Lancet 1995 What we do know about some of the effecgtive strategies is that they generally have a small to moderate effect Facilitated teaching with evidence based resources similar to the WHO manual proposal, combined with audit and feedback have a moderate effect A previous WHO resource – the companion =volume for this manual – was implemented within a pilot study and found to have a significant effect on students prescribing skills in a test situation
10
Randomised controlled trial 10 volunteer sites
Study Design Randomised controlled trial 10 volunteer sites 5 intervention sites 5 control sites Quantitative and qualitative methods to assess change in students knowledge attitudes and skills So for these reasons to both test the effect and improve the content of this manual we are planning a randomised controlled trial of implementation Why an RCCT? There are a number of potential biases in designs other than RCTs Before and after designs with no comparison group are likely to show effects in the students that are dues to influences other than the manual While at medical and pharmacy school students are subject to a wide range of influences and in addition mature a great deal in their thinking in each year. Satisfaction surveys are likely to be subject to Social desirability bias – that is students may choose the answers they think they ‘ought’ to We are proposing to randomise by site rather then randomised students within a site as many of the changes are attitudinal ones. During medical and pharmacy training one of the most important influences on attitudes are peers and it would be impossible to avoid ‘contamination’ through discussion between students who did and did not receive teaching using the manual. We will use quantitative methods to look at shifts in quantifiable aspects of students knowledge attitudes and skills But there are a number of sources of bias in using just this kind of assessment – There is still a risk of Social desirability bias – that is students may choose the answers they think they ‘ought’ to in both arms For this reason we are planning an complementary qualitative analysis that will use before and after focus groups to understand the kinds of more subtle effects the implementation of the manual has had on student attitudes and behaviour. This part of the evaluation will also involve key informant interviews with the teachers to tease out aspects of the manual that were helpful and unhelpful – what caught students interest and what was counterproductive.
11
What is the effect on students
Questions What is the effect on students knowledge, attitudes and skills of students in understanding and responding to pharmaceutical promotion What are the strengths and weaknesses of the content of the manual and resources provided assessed across a variety of settings These are the two important questions to answer One about the effect on students of teaching using these resources The other is around the material provided – its strengths and weaknesses and how it might be improved
12
Data collection Before and after assessment of knowledge, attitudes and skills of students Questionnaire Focus groups Key informant interviews and content evaluation by teachers in the intervention group Students will have before and after questionnaires covering their knowledge attitudes and skills as well as before and after focus groups to explore more deeply their attitudes and planned engagement style with promotion in particular Teachers will provide feedback on the content of the manual and also participate in interviews to get their views on how well the manual worked overall
13
Two comparisons between these randomised intervention and control groups
The difference between the before and after manual assessments at the beginning and end of the year compared to the control sites The difference between their assessments at the end of the year and students at the end of the year in the previous year group Again compared to the differences at the control sites We will compare the difference in before and after assessments in the sites that had the manual and resources with those that carried on with their normal teaching curriculum. We will do a second independent comparison between the students at the end of the previous group, before the manual is used at the site , with the next group of students in at the same stage of training, who have receive d teaching using the manual and resources. These differences between consecutive years will also be compared between intervention and control sites. This will give us 2 independent ways of assessing what the effect of the manual has been on these students
14
Comparison 1 Change in scores at intervention sites compared to control sites intervention sites control sites manual teaching as usual Before Before compared to After After For those who like diagrams… difference difference
15
Comparison 2 Difference in scores between study groups at the end of the year and students at the end of the previous year intervention sites control sites Students from previous year tested for comparison Students from previous year tested for comparison Study start Study start Manual implemented Teaching as usual Students at end of study year Students at end of study year
16
Comparison 2 Difference in scores between study groups at the end of the year and students at the end of the previous year intervention sites control sites Study start Study start Manual implemented Teaching as usual Students from previous year tested for comparison Students from previous year tested for comparison Students at end of study year Students at end of study year
17
Medical or pharmacy students
Volunteer sites Medical or pharmacy students Site prepared to teach all manual chapters Site able to teach within one academic year (12 months) Ethics approval will be gained where required The volunteer sites we need are medical or pharmacy students who are taught in spanish english or russian. THe volunteer sites need to be able to teach all or most of the manual chapters and do so within one academic year in order that we can test and get the results in a way that is timely enough to allow revision before release of the final version. We will of course gain ethics approval in all sites that require this.
18
Volunteer sites Training, travel and accommodation will be provided for a teacher from each site The evaluation assessments and analysis will be organised and co-ordinated by the study team based at HAI These involve the before and after questionnaires and focus groups for students/interviews for teachers at each site What will the volunteer sites get Teacher will come to Amsterdam to receive 3 days of training in the manual the resources and the implementation The research team will organise and co-ordinate the before and after assessments at each site . That is the questionnaires and focus groups
19
Volunteer sites: Timeline
Plan to implement as academic years begin from later in 2009 through to 2010 Training for teachers from sites randomised to intervention in Amsterdam Sept 2009 ‘Before’ assessments as academic years start ‘After’ assessments completed late 2010 as years end Training for teachers from sites randomised to control in Amsterdam in Sept 2010 This is the timeline The draft manual is ready and we plan to implement it as a pilot later this year and through to early 2010 as academic years start in different countries. As sites volunteer we would complete the training and assessment of students prior to the manual teaching, and then aim to complete the after manual assessments late in 2010. At that point the ‘control ‘ site teachers would then receive the manual and training in its implementation in Amsterdam.
20
It is not new Promotion is not new but its influence is increasing
21
As promotion of pharmaceuticals becomes and increasingly important influence Countering the adverse effects of promotion is vitally important for the rational use of medicines in ensuring that patients receive the best that medical science has to offer.
22
We need to ensure medical and pharmacy graduates are well equipped to understand and deal with the effects of promotion on their prescribing decisions So our imperative to provide health professionals with the tools to deal with it also becomes more urgent Because In the end
23
Our common goal is the relief of suffering.
Inappropriate medication decisions made in response to misleading promotion can increase the burden of suffering rather than relive it
24
It is important that we not just have good ideas
But that we know whether our collective efforts are effective We not to just have good ideas about how we might do this , We also need to know that our collective efforts are effective
25
In order that disaster does not follow disaster
26
Volunteers! Pilot study Testing the manual in your setting
Expression of interest forms at the back of the room OR So this is my last word about the manual evaluation WE NEED YOU
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.