1. Evaluating a pilot educational initiative across a European network of primary care organisations: the IPCRG's Teach the Teacher Programme.
IPCRG has an important educational role. To this end we have developed an educational strategy which includes our flagship programme e-quality which is based on an extensive review of the literature into what kinds of educational interventions can contribute to clinical behavior change and improved health outcomes. So this is an important and complex area and one we are keen to learn about.
This presentation describes a new educational initiative that we piloted last year.
The ‘teach the teacher’ programme is based on a ‘cascade’ or ‘train the trainer’ model of education – whereby 1) a team of experts teach a group of colleagues who 2) develop in-country programmes and teach their colleagues to develop specialist knowledge and change clinical practice.
Abstract: Implementation Science / Service Development
Evaluating a pilot educational initiative across a European network of primary care organisations: the IPCRG's Teach the Teacher Programme.

Juliet MCDONNELL1, Siân Williams1, Ioanna Tsiligianni2
1IPCRG, 2Clinic of Social and Family Medicine University of Crete
Brief outline of context
IPCRG is a global network of organisations with an interest in assessing and treating chronic respiratory disease (CRD) in primary care. IPCRG has a key role in supporting improvement using evidence-based resources such as its SIMPLES structured asthma review and desktop helper on difficult to manage asthma. Evidence from the recent U-BIOPRED project, on severe asthma needs to be disseminated. IPCRG has developed an education strategy that proposes various approaches/programmes to build capacity for primary care development.
Brief outline of what change you planned to make
In January 2015, we launched a pilot ‘Teach the Teacher: Difficult to Manage Asthma' programme. We brought together 14 clinical colleagues from seven European countries,  to take part in a workshop in Rome. Our aim was to build their ability to teach other clinical colleagues the key messages concerning difficult to manage asthma; including how to use and adapt existing IPCRG resources for local use. We convened an Expert Faculty, including patient representation (EFA) and supported attendees to design and plan educational programmes to take place in their own country. 
Measurement of improvement
An evaluation immediately following the workshop in Rome,  was rated highly by all participants, with overall scores of 4 or 5 (where 5 was ‘excellent’).
Subsequently, all the participants developed proposals for programmes in their own countries – these seven events took place in autumn 2015 and involved over 230 health professionals - including specialist nurses, physiotherapists, General Practitioners (GP); early career GPs; and GPs with a special interest in Asthma/COPD.

Effects of change
We proposed an evaluation framework (Guskey 2002). This includes -
i) participant reactions, ii) learning, iii) organisational changes, iv) use of new knowledge and v) impact on practice and service users.
Early indicators suggest that programmes were well received by participants ('positive participant reaction') – other dimensions of evaluation require more time to report and the impact on participants clinical practice and services users may prove more challenging to demonstrate.
Lessons learnt
Findings will be presented that describe the challenges that teams encountered in assessing learning needs, setting up education programmes and in supporting clinical practice change. We reflect on the challenge of meaningful evaluation of practice change or impact. These insights are important in thinking about the design, implementation and evaluation of clinical educational programmes globally.
Message for others
We would like to acknowledge our collaborators,  the Teach the Teacher Faculty and the first cohort of Teachers.
Declaration of interest
The European programme U-BIOPRED funded the programme and included an update on the findings of U-BIOPRED on severe asthma.

2. Aims:
To disseminate UBIOPRED research evidence and evidence based tools such as IPCRG ‘SIMPLES’ and desktop helper.
To build a trusted faculty with the knowledge, influence, communication and teaching skills
to deliver effective workshops
that are relevant to local situation and
can be embedded in local primary care practice and pathways including hospital care.
But although the idea looks simple - this model is not quite as straight forward as you might think.
It is not just about the transmission of knowledge from one person to another.
In this presentation I would like to help you understand our approach a little better – to explain to you why we chose this model, what happened when we tried to pilot it and whether it was successful.
First I would like to say more about why we selected this approach -
This project was a partnership with U-BIOPRED. We saw it as a way to disseminate the U-BIOPRED findings and to make a connection with IPCRG evidence based tools concerning difficult to manage asthma . (IPCRG emphasizes the importance of differentiating between patients with difficult to manage asthma from this with severe asthma).
This approach is appropriate to the context in which we are working - i.e. a network of primary care and respiratory organisations.
The approach allows us to test this model to see if it works, we hope to adapt it to other areas of interest. So we are testing our education strategy.
The approach allows us to build capacity - a trusted faculty or ‘community of practice’- colleagues we can work with in the future on other projects
Very importantly - this approach builds on ‘in-country’ knowledge and networks- it allows for local adaptation which is important given the different health systems and variations across different countries.

3. Teaching and facilitating learning
Assessment of learning
Educational research and scholarship
Educational management and leadership
Designing and planning learning
It is also important to say that we emphasised both subject specific knowledge (U-BIOPRED /IPCRG) and also the skills of teaching and facilitating learning/ designing and planning learning as important features of the programme. We wanted our colleagues to think about what they were teaching but also the process of learning and change.
These skills are acknowledge by the Academy of Medical educators framework -
Academy of Medical Educators Framework -

4. So, just to clarify this was a pilot educational programme which involved a European network of primary care organisations
We invited colleagues from seven countries to take part – Portugal, Italy, Romania, Greece, Netherlands, UK, turkey, Ireland
We established an expert faculty – including patient representatives (EFA)
This course is supported by an unrestricted educational grant from U-BIOPRED
Any opinions expressed are those of the faculty, not U-BIOPRED

5. Teach the Teacher Faculty Event Develop proposals
Rome - Jan 2015
Teach the Teacher Faculty Event
Develop proposals
Agree proposals and funding
Pilot local events
Evaluate
Initially we met in Rome for a weekend - this is the project timeline – gives an overview of the process and what we were asking people to commit to.
Dec 2015
Jan 2015
June 2015

6. Evaluating professional development
Participant reactions
Participant learning
Organisational support and change
Participant use of new knowledge and skills
Impact on participants, practice and service users
Because this was a pilot - it was important for us to think about the impact of this project and our activities
We proposed a model for evaluation – this is based on a model for evaluating professional teacher development and not medical education
We chose it because it is a simple model and one that is easy to apply
It is based on Kirkpatrick – which although developed in a business context has often been used in the context of medical education.
Source: Evaluating Professional Development . T Guskey 2002
Source: Evaluating Professional Development . T Guskey 2002

7. When we said good- bye to everyone after a weekend in Rome - we didn’t know what would happen!

8. Results At programme level
Strong ‘positive participant reaction’ following the initial workshop
Excellent engagement throughout with our colleagues
Evidence of ‘participant learning’ and ‘use of new knowledge’ and ‘organisational change’
We received seven proposals and delivery of in-country programmes in 2015.
There was variation in the design and focus of in-country events
Over 230 health professionals including specialist nurses, physiotherapists, General Practitioners (GP); early career GPs; and GPs with a special interest in Asthma/COPD.
Evaluation reports, including evidence were provided by all participating countries.
At programme level we had
strong ‘positive participant reaction’ following the event,
excellent engagement throughout with our colleagues;
evidence of participant learning, use of new knowledge and organisational change in the form of proposals and seven in-country programmes delivered in There was variation in the design and focus of in-country events
Four countries opted for a focused educational event(s), one of these was multi professional, one focused on early career doctors, another met different audiences in two regional locations (Italy, Turkey, UK, Greece)
Three countries opted for a cascade approach – training an in-country faculty who will develop regional events (Netherlands, Portugal, Romania)
Evaluation reports, including evidence were provided by all participating countries. Financial incentives were likely a significant factor.

9. Evaluating professional development
Participant reactions
Participant learning
Organisational support and change
Participant use of new knowledge and skills
Impact on participants, practice and service users
This was the model we offered to our colleagues to help them think about evaluation.
Now I would like to share some of the lessons we learned from their experiences. It’s important to say that for many groups this is a work in progress.

10. Participant reactions
Importance of pre assessment
Delphi
Questionnaire/ s
Interviews
Informing workshop design
Post workshop questionnaire
100 % response rate (n= 15)
The workshop was rated highly by all participants, with overall scores of 4 or 5 (where 5 was ‘excellent’).
One of the first important insights in our work was the importance of local needs assessment – good educational principles – thinking about the audience and not assuming you know what it is they need to learn. This was done using various methods
Questionnaire, interviews, correspondence
This is important because it allows adaptation of the educational event which is sensitive to local context and allows local variation and needs to be highlighted (in Greece , Athens vs Heraklion).
It also begins to engage the audience and build their commitment to the programme at an early stage.
In the Netherlands our colleagues conducted a Delphi exercise with all of their stakeholders to adapt and agree learning material and approach ( See Honkoop,P et al 2016)
Using the proposed framework – the first level is participants reaction and most people reported this – generally it ahs a positive response (e.eg our event in Rome) .
It has limitations it doesn't tell us what happened next.

11. Participant learning Participant learning Participant reactions
In class evaluation
Pre and post workshop questionnaire
tests knowledge
Examples of evidence provided from in-country programmes which evaluated ‘participant learning’ – tend to be short term
Pre and post workshop questionnaire
tests knowledge

12. Organisational support and change
Participant reactions
Participant learning
Organisational support and change
Questionnaire about barriers
Qualitative responses n=7
Data – key themes
Logistics/geography
Different systems in pc
Incentives /finance
This level of evaluation changes here and we look at organisational variables which may help or hinder the intervention.
So there could be a really good programme which may be hindered by organisational factors.
We asked our colleagues in different countries to reflect on this and to identify factors which helped and hindered the success or otherwise of their programme.
Brief findings:

13. Participant use of new knowledge and skills
Participant reactions
Participant learning
Organisational support and change
Participant use of new knowledge and skills
7 proposals for in country training
Reviewed by expert faculty
Programme delivery in 2015 n=7
Programme design (evidence)
Evaluation
Examples of evidence provided from in-country programme which evaluated participants use of new skills and knowledge
Examples cited:
There is less evidence here and this could be for a few reasons
1) it may be too early to evaluate this
2) Alternatively, this area is more complex and resource intensive to collect

14. Impact on participants, practice and service users
Participant reactions
Participant learning
Organisational support and change
Participant use of new knowledge and skills
Impact on participants, practice and service users
Generally positive response to in-country events
More challenging area of evaluation
Has significant resource implications
Needs a longitudinal study
Examples of evidence provided from in-country programmes which evaluated impact on participants, practice and service users
Again, there is less evidence here and this could be for a few reasons
1) it may be too early to evaluate this
2) this area is more complex and resource intensive to collect

15. Evaluating in-country programmes
There was variation in the design and focus of in-country events
One programme focused on early career GPs, another in a specialist network of GPs with an interest in respiratory disease, one was multi-disciplinary.
Overall a ‘positive participant reaction’ was reported.
We know that teams encountered various challenges in assessing learning needs, setting up education programmes and in supporting clinical practice change.
Importance of needs assessment
We concluded that evidence for clinical practice change is harder to achieve without additional local resources and a longer-term strategy.
The evaluation reports help us understand local implementation.
In-country programmes too reported a ‘positive participant reaction’.
We know that over 230 health professionals took part in educational events in seven countries – this included specialist nurses, physiotherapists, General Practitioners (GP); early career GPs; and GPs with a special interest in Asthma/COPD.
There was variation in the design and focus of in-country events –
Four countries opted for a focused educational event(s), one of these was multi professional, one focused on early career doctors, another met different audiences in two regional locations (Italy, Turkey, UK, Greece
Three countries opted for a cascade approach – training an in-country faculty who will develop regional events (Netherlands, Portugal, Romania)
We know that teams encountered various challenges in assessing learning needs, setting up education programmes and in supporting clinical practice change.
We conclude that evidence for clinical practice change is harder to achieve without additional local resources and a longer-term strategy.

16. The question of ‘fidelity’
System for classification of modifications to evidence based programmes or interventions
Who makes the modification?
What was modified?
At what level?
What is the nature of the context modification?
What is the nature of the content modification
(Stirman et al 2013)
The project allowed for variation – we would argue it was important for success. However we are interested in how and why variation may have occurred
Development of a framework and coding system for modifications and adaptations of evidence-based interventions
Shannon Wiltsey Stirman, Christopher J Miller, Katherine Toder and Amber Calloway
Stirman et al. Implementation Science 2013, 8:65

17. Conclusion
We have increased capacity to teach in primary care settings in seven member countries
We can now work with established faculty on other programmes
We can explore the potential for working in different ways e.g. e-learning or virtual programmes which require a different skill set
Financial incentives were likely a significant factor in the success of the project.
The insights gained continue to inform our thinking about design, implementation and evaluation of clinical educational programmes globally.