2. Manual Evaluation: Are we doing what we think we are doing?
Associate Professor Dee Mangin
Director Primary Care Research Unit
Department of Public Health and General Practice
Christchruch School of Medicine,
University of Otago
You will be wondering what a family doctor from New Zealand is doing… and at the end
I am here from the other side of the world because the issue we are trying to address here is global OR
Pharmaceutical promotion is the greatest threat globally to rational use of medicines and its influence is increasing
Now more than ever we need to to include in the medical and pharmacy school curricula teaching to assist students to understanding the range of promotional strategies and to develop strategies to minimise inappropriate prescribing as a result
And up until now there has been neither curriculum time nor resource material for medical and pharmacy schools
20 May 2009
WHA

3. Rationale for testing the manual in a pilot study
Overview
Rationale for testing the manual in a pilot study
Protocol for testing
Timelines
What I would like to describe for you in the next 15 minutes is outline the planned pilot study testing of this WHO / HAI teaching manual – covering the reasons why it is essential that we do pilot it and the way we plan to do it.
We would really welcome interest from any of you who are associated with medical and pharmacy student teaching who are interested in being involved as a pilot study site

4. Background
WHO/HAI have developed a comprehensive teaching package comprising a manual and accompanying resource material as well as training for teachers implementing and this
So we have this very comprehensive manual and teaching resource package that Barbara has spoken about

5. Background
It is planned to evaluate this in a pilot study to look at its effectiveness and at areas for improvement in the final manual
WHY DO THIS STUDY?
Why not just send it out?
Why do we need to do a pilot study?

6. Background
Many educational resources and techniques are implemented without ever having any assessment of their effectiveness
Many – dare I say most educational ideas and resources are implemented without piloting
– without ever really knowing whether they are effective
And also
Without knowing whether there is some possibly counterproductive effect
Counterproductive? Some of you will think
How could providing information be counterproductive
In this instance we are implementing a manual in an environment where most physicians and pharmacists acknowledge that people are influenced by promotion, but believe that they individually arent.
This is called the illusion of unique invulnerability.
It is possible that in this environment, an educational strategy might further increase the confidence of individuals in engaging with the pharmaceutical industry, believing they are even less vulnerable to such influence when armed with knowledge.

7. So we don’t know whether our efforts are effective
So, like this cohort of roman soldiers marching on
Often we don’t know whether our efforts are effective

8. Our wish to think that they are can be misleading……
Overview
Our wish to think that they are can be misleading……
Many strategies for medical education have little effect on prescribing when they are tested in studies
And our desire to think they are can sometimes be misleading
Many strategies for medical education have little effect on participants when assessed in a research framework
Printed information
Lectures …. Like this one
Guidelines have limited effect

9. Evidence for prescribing education strategies that have been found effective
The effect of dissemination only strategies such as didactic lectures and written information are small at best and of unknown clinical significance
Farmer et al. Cochrane Database of Systematic Reviews 2008;3
Facilitated teaching with evidence based resources and audit and feedback have been found to have a moderate effect
Richards et al Family Practice 2000
A previous WHO educational resource (WHO Guide to Good Prescribing) using facilitated teaching for medical students on pharmacotherapy showed a significant effect on students skills in a randomised controlled trial
Impact of a short course in pharmacotherapy for undergraduate medical students: An international randomised controlled study. de Vries T Henning R Hogerzeil H Bapna J et al The Lancet 1995
What we do know about some of the effecgtive strategies is that they generally have a small to moderate effect
Facilitated teaching with evidence based resources similar to the WHO manual proposal, combined with audit and feedback have a moderate effect
A previous WHO resource – the companion =volume for this manual – was implemented within a pilot study and found to have a significant effect on students prescribing skills in a test situation

10. Randomised controlled trial 10 volunteer sites
Study Design
Randomised controlled trial
10 volunteer sites
5 intervention sites 5 control sites
Quantitative and qualitative methods to assess change in students knowledge attitudes and skills
So for these reasons
to both test the effect and improve the content of this manual
we are planning a randomised controlled trial of implementation
Why an RCCT?
There are a number of potential biases in designs other than RCTs
Before and after designs with no comparison group are likely to show effects in the students that are dues to influences other than the manual
While at medical and pharmacy school students are subject to a wide range of influences and in addition mature a great deal in their thinking in each year.
Satisfaction surveys are likely to be subject to Social desirability bias – that is students may choose the answers they think they ‘ought’ to
We are proposing to randomise by site rather then randomised students within a site as many of the changes are attitudinal ones. During medical and pharmacy training one of the most important influences on attitudes are peers and it would be impossible to avoid ‘contamination’ through discussion between students who did and did not receive teaching using the manual.
We will use quantitative methods to look at shifts in quantifiable aspects of students knowledge attitudes and skills
But there are a number of sources of bias in using just this kind of assessment –
There is still a risk of Social desirability bias – that is students may choose the answers they think they ‘ought’ to in both arms
For this reason we are planning an complementary qualitative analysis that will use before and after focus groups to understand the kinds of more subtle effects the implementation of the manual has had on student attitudes and behaviour. This part of the evaluation will also involve key informant interviews with the teachers to tease out aspects of the manual that were helpful and unhelpful – what caught students interest and what was counterproductive.

11. What is the effect on students
Questions
What is the effect on students
knowledge, attitudes and skills of students in understanding and responding to pharmaceutical promotion
What are the strengths and weaknesses of the content of the manual and resources provided
assessed across a variety of settings
These are the two important questions to answer
One about the effect on students of teaching using these resources
The other is around the material provided – its strengths and weaknesses and how it might be improved

12. Data collection
Before and after assessment of knowledge, attitudes and skills of students
Questionnaire
Focus groups
Key informant interviews and content evaluation by teachers in the intervention group
Students will have before and after questionnaires covering their knowledge attitudes and skills as well as before and after focus groups to explore more deeply their attitudes and planned engagement style with promotion in particular
Teachers will provide feedback on the content of the manual and also participate in interviews to get their views on how well the manual worked overall

13. Two comparisons between these randomised intervention and control groups
The difference between the before and after manual assessments at the beginning and end of the year compared to the control sites
The difference between their assessments at the end of the year and students at the end of the year in the previous year group
Again compared to the differences at the control sites
We will compare the difference in before and after assessments in the sites that had the manual and resources with those that carried on with their normal teaching curriculum.
We will do a second independent comparison between the students at the end of the previous group, before the manual is used at the site , with the next group of students in at the same stage of training, who have receive d teaching using the manual and resources. These differences between consecutive years will also be compared between intervention and control sites.
This will give us 2 independent ways of assessing what the effect of the manual has been on these students

14. Comparison 1
Change in scores at intervention sites compared to control sites
intervention sites control sites
manual teaching as usual
Before Before
compared to
After After
For those who like diagrams…
difference
difference

15. Comparison 2
Difference in scores between study groups at the end of the year and students at the end of the previous year
intervention sites control sites
Students from previous year tested for comparison
Students from previous year tested for comparison
Study start
Study start
Manual implemented
Teaching as usual
Students at end of study year
Students at end of study year

16. Comparison 2
Difference in scores between study groups at the end of the year and students at the end of the previous year
intervention sites control sites
Study start
Study start
Manual implemented
Teaching as usual
Students from previous year tested for comparison
Students from previous year tested for comparison
Students at end of study year
Students at end of study year

17. Medical or pharmacy students
Volunteer sites
Medical or pharmacy students
Site prepared to teach all manual chapters
Site able to teach within one academic year (12 months)
Ethics approval will be gained where required
The volunteer sites we need are medical or pharmacy students who are taught in spanish english or russian.
THe volunteer sites need to be able to teach all or most of the manual chapters and do so within one academic year in order that we can test and get the results in a way that is timely enough to allow revision before release of the final version.
We will of course gain ethics approval in all sites that require this.

18. Volunteer sites
Training, travel and accommodation will be provided for a teacher from each site
The evaluation assessments and analysis will be organised and co-ordinated by the study team based at HAI
These involve the before and after questionnaires and focus groups for students/interviews for teachers at each site
What will the volunteer sites get
Teacher will come to Amsterdam to receive 3 days of training in the manual the resources and the implementation
The research team will organise and co-ordinate the before and after assessments at each site . That is the questionnaires and focus groups

19. Volunteer sites: Timeline
Plan to implement as academic years begin from later in 2009 through to 2010
Training for teachers from sites randomised to intervention in Amsterdam Sept 2009
‘Before’ assessments as academic years start
‘After’ assessments completed late 2010 as years end
Training for teachers from sites randomised to control in Amsterdam in Sept 2010
This is the timeline
The draft manual is ready and we plan to implement it as a pilot later this year and through to early 2010 as academic years start in different countries.
As sites volunteer we would complete the training and assessment of students prior to the manual teaching, and then aim to complete the after manual assessments late in 2010.
At that point the ‘control ‘ site teachers would then receive the manual and training in its implementation in Amsterdam.

20. It is not new
Promotion is not new but its influence is increasing

21. As promotion of pharmaceuticals becomes and increasingly important influence Countering the adverse effects of promotion is vitally important for the rational use of medicines in ensuring that patients receive the best that medical science has to offer.

22. We need to ensure medical and pharmacy graduates are well equipped to understand and deal with the effects of promotion on their prescribing decisions
So our imperative to provide health professionals with the tools to deal with it also becomes more urgent
Because
In the end

23. Our common goal is the relief of suffering.
Inappropriate medication decisions made in response to misleading promotion can increase the burden of suffering rather than relive it

24. It is important that we not just have good ideas
But that we know whether our collective efforts are effective
We not to just have good ideas about how we might do this ,
We also need to know that our collective efforts are effective

25. In order that disaster does not follow disaster

26. Volunteers! Pilot study Testing the manual in your setting
Expression of interest forms at the back
of the room OR
So this is my last word about the manual evaluation
WE NEED YOU