from EBOT exam database Prof Dr Jorge Mineiro MD, PhD, FRCSEd EBOT Spring Meeting Lyon - France EBOT Exam experience & evolution where we stand – where to go from EBOT exam database Prof Dr Jorge Mineiro MD, PhD, FRCSEd Chairman of EBOT Examining Committee Lyon, 28th May 2016

EBOT Exam experience and evolution ◙ how we have started ◙ progress and evolution ◙ where we are now ◙ how much it costs ◙ the near future ◙ strategy for the future at the long run ◙ decisions to be made ◙ conclusions j.mineiro 16

EBOT Exam experience and evolution ◙ introduction ● the principles within EU Ensure ? Garantizarlos mejores cuidados medicos para el ciudadano europeo j.mineiro 16

EBOT Exam experience and evolution ◙ why should there be a certification ● the pillars for patient safety Developing and maintaining high standards in education and training is in the interest of patient safety and healthcare. This is particularly important in an environment of rapidly changing scientific knowledge, high patient expectations and increasingly sophisticated health care delivery systems. Good quality patient care requires each registered Medical practitioner to continuously participate in learning activities and the need to prove this life long commitment from specialized doctors – (recertification). De conocimento cientificii en constant cambio, espectativas elevadas dse los pacientesy un systema de salud cada dia mas sofisticado j.mineiro 16

EBOT Exam experience and evolution ◙ what is the purpose ● under EU regulations From the perspective of the European Union, the general purpose of EU Certification at specialist level is to create an internal European market for doctors who have completed recognized higher specialist training, in accordance with certain EU Regulations. This is in order to allow those doctors automatic(?) recognition of their qualifications in any EU Member State. Reconocida por las instancias de la comunidad europea el reconocimiento automotico j.mineiro 16

EBOT Exam experience and evolution ◙ what is the purpose ● under EU regulations Although there is automatic recognition arrangements enshrined within the European legislation to facilitate the free movement of European Nationals the truth is that this in not equivalent and is no guarantee that employers will accept the same administrative recognition ! no es garantia de acceptacion y de empleo en outros paises de la comunidad j.mineiro 16

EBOT Exam experience and evolution ◙ how can we standardize the quality of training in orthopaedics? ● most European countries had very similar training programs ■ similar program format BUT completely distinct in implementation ■ in most countries training programs are ruled governmental bodies ■ by 1998, Minimal Requirements for Orthopaedic Training had been approved within UEMS-Orth and divulged within all societies ■ the only way to converge on the quality of what we all do was to assess the end of training (outcome of residency program) Upon analysing the different training programs – there were quite a few differences ( countries with trainees part time , others full time countries with exam at end of 6y others yearly and other with no exam, the number of operations was discriminated in some and not mentioned in others even the content of training was different in some countries MS trauma and Orthopaedics separated,etc.... ) it was accepted that there was no way we could change the programs of training within each country and so concern about the quality of specialists that were coming out of the different residency programs in the same Europe despite the approval of MR decision was made to change strategy and assess the end of training comparing under the same exam the different training schemes. ■ BUT even so, no country was prepared to give up their final exam/national qualification, so double checking was the solution ■ EBOT Exam was created with the support of ALL members of the UEMS-Orth in 1999/2000 j.mineiro 16

EBOT Exam experience and evolution ◙ why take the certification ● at present, more and more exceptions to the rule - What is the relevance of an European Board Certification With the exception of few specialities (anaesthetics, urology,…), in a few countries, they are not part of a formal professional recognition of specialist doctors and have no legal status yet, either nationally or at European level. However they are consider an asset in specialist doctor’s CV or portfolio and in certain specialities it is already recognised as the national final qualification (at end of training), in others as European Qualification. Qual es la bentaja de lacertificacion europea ? Aunque notenga um satus … en el momento tiene bentajas porq es una mas valia importanteinel curriculum or portfolio de le medico a ajudado a promocionen la carrera In orthopaedics, EBOT Diploma has helped 68% of the fellows to get new appointments in Europe (2011 data) j.mineiro 16

EBOT Exam experience and evolution ◙ why take the certification ● if we look across different European Boards ● where does the EBOT Exam stands CESMA Council of European Specialities Medical Assessments - 27 European Boards with examinations (under UEMS) Paediatric Surgery, Neurosurgery, Nuclear Medicine, Pathology, Plastic & Reconstructive surgery, Urology, Vascular Surgery, Anaesthetics, Pneumology, General surgery and Orthopaedics & Traumatology … Council for European Specialists Medical Assessment - Criteria for application, format, price, frequency …. - Most boards allow last year residents to take part I - 7 European Countries have adopted the Anaesthetic Board Examination as their final exam at end residency (compulsory) ● EBOT Exam one of/the most prestigious advanced exam in CESMA j.mineiro 16

EBOT Exam experience and evolution ◙ are we double checking ? ● let’s look at our data from the EBOT PAST DATA Exam Year Venue Nº Candidates Pass Fail %     2001 Rhodes - Greece 11 8 3 73% 2002 Stockholm - Sweden 16 10 6 63% 2003 Amsterdam – Netherlands 12 9 3 75% 2004 Berlin - Germany 15 10 5 67% 2005 Madrid - Spain 10 5 5 50% Torino – Italy 18 12 6 67% Vienna - Austria 12 7 5 58% Geneva – Switzerland 13 6 7 46% Brussels – Belgium 31 15 16 48% **Lisbon – Portugal 35 25 10 72.5% **Paris – France 60 39 21 66.5% **Dublin – Ireland 78 58 20 74,3% **Barcelona - Spain 90 66 24 74% ** Vienna – Austria 86 64 22 75% **Rotterdam - Holland 105 70 35 67% **In 2010 the rules changed and the written paper was set separately + trainees

EBOT Exam experience and evolution ◙ we are increasing in number of candidates ● since 2010 we have been increasing 30 to 40% / Year 245 candidates - 84 spanish + So we'll have 161 candidates from all over Europe who then would eventually sit the English oral part. However, not all will pass. Let's assume it'll be 100/110 who pass plus then the resits for the oral from last year, we'll end up with about 130-150 candidates in Lisbon!!! Years since 2001

EBOT Exam experience and evolution ◙ assessment at the end of training ● is there a global format for assessment at the end of training? ■ by analysing these results we may go back and screen the different stages of the training process in each country with the same objective in mind – improve the quality of post graduate medical training ! – but there is still no standard method of assessment at the end of training in EU Meejorar la calidad Pero es interessante ver ■ from EBOT – countries in Europe that no candidates have ever passed the exam – one went back and restructured the Orthopaedic Training Scheme and final assessment, others sat back and put pressure on lowering the standards of the exam, others are concern about their performance at European level j.mineiro 14

EBOT Exam experience and evolution ◙ at the second decade of the exam ▲ what have we done to improve standards ? A. Exam preparation B. Final assessment j.mineiro 16

EBOT Exam experience and evolution ◙ at present ▲ we have addressed two different issues optimizing results: A.Exam preparation B.Final assessment ■ A. Exam preparation -Interim Exam + EBOT Writing Committee - 2011 -Examiners Courses prior to every final exam - 2001 -Review Courses (under review by EFORT) for candidates -European Educational Platform (EEP) (EFORT) 2015 European countries have approved a syllabus + European curriculum + relevance of a final assessment j.mineiro 16

EBOT Exam experience and evolution ◙ at present ● Interim Exam ● Happens regularly since 2011 - MCQ ● It is run always in the Spring (12th of April 2016) all around Europe on the same day at the same time • 305 registered for Interim, 262 candidates (14th of April 2015) have taken their departments (last year there were 149) Translations will loose quality of the exam – impossibleto grade dthe difficulty of each MCQ in the different languages Run in English – decision taken not to do different translations ■ free of charge for the next 3 years (sponsored by the European Board of Orthopaedics and Traumatology from 2013) j.mineiro 16

EBOT Exam experience and evolution ◙ at present ● EBOT Writing Committee ● Happens regularly since 2011 - Meet twice a year always in Lisbon • Spring meeting for preparation of new MCQ and Autumn meeting for selection and review MCQ Translations will loose quality of the exam – impossibleto grade dthe difficulty of each MCQ in the different languages ● Validation Committee planned, chaired Prof Enrique Barrena ■ database of over 1.700 MCQs with detailed information on each one that we are reviewing as we get more information j.mineiro 16

EBOT Exam experience and evolution ◙ at present ● Final EBOT Exam Happens regularly since 2001 in the first weekend in October It has 2 parts – Part I written(Spring) + Part II oral (October) Last year in Rotterdam – Holland (this year will be in Lisbon) Explicar q o exame de outubro é o oral Different locations is Europe with easy low cost access Run in English – discussion in 5 different stations j.mineiro 16

EBOT Exam experience and evolution ◙ at present ● Final Exam ▲ we have addressed two different issues optimizing results : A.Exam preparation B.Final assessment ● B. Final Assessment (EBOT Fellowship Examination) : -separated written part I (Pearson Vue examining centres) -invited all European speciality societies to join the Writing Committee -improved the set up of part II – examinations halls -optimize Writing Committee performance (three to four years appointment) + setting basis for a Validation Committee In order to improve the WC performance we are creating a Validation Committee j.mineiro 16

EBOT Exam experience and evolution ◙ at present ● Final Exam ▲ we have addressed two different issues optimizing results : A.Exam preparation B.Final assessment ● EBOT Writing Committee : -meets twice a year in Lisbon – Autumn meeting to review new questions and Spring to choose MCQ both exams -together with we developed very sophisticated program for MCQ -we are always in need for new MCQ !!!! In order to improve the WC performance we are creating a Validation Committee j.mineiro 16

Item Analysis Question Quality Point bi-serial DI-Index Question Difficulty P-value Distractor efficiencies All Best performers Worst performers Item analysis is displayed together with a question. The question difficulty or p-value is calculated together with the discrimination values which give insight into whether the question can separate the good from the poor performers. A histogram showing how many candidates answered each of the answer options is displayed which can give valuable insight in how a poor question can be improved.

Selecting Questions Features include: Select questions by filtering and sorting. Question details Exclude from previous exams Preview exam as candidate Export questions Questions are filtered from various parameters and selected for the the exam. A number of features ae provided to make this task efficient including both tabular and graphical feedback to ensure the questions selected balance the chosen subject areas.

Exam Metrics Mean Standard deviation Cut scores Features include: The exam metrics will give an exam-by-exam comparison of the most important statisitics to ensure that exam standards are maintained. Cut scores are also generated based on candidate performance. Mean Standard deviation Cut scores Features include: Standard error Kuder Richardson 20

EBOT Exam experience and evolution ● Final EBOT Exam ▲ we have addressed two different issues optimizing results : A.Exam preparation B.Final assessment ● B.Final Assessment – Oral Part II Exam in English : -it was run in Rotterdam and local organizer is Dennis Darte For the first time in the history of the exam we will have more than 100 candidates – this has always been our main objective BUT now that we are getting there !! It puts a tremendous burden on the organization and we do need a lot of help again from the national societies – WE DESPERATELY need more examiners in the different subjects ● 173 candidates registered for the written part I (146 succeeded – 26% trainees) – every year numbers > rapidly!! (last year we had 98) ● 20% will fail +10 will not turn up + 10% are last year residents BUT we will have ressits – we had 105 candidates in 2015 ! j.mineiro 16

EBOT Exam experience and evolution ● Final EBOT Exam For the first time we had over 100 candidates for part II exam Part II oral ( 3 and 4th of October 2015) - 105 candidates took the exam in Rotterdam – Holland Explicar q o exame de outubro é o oral Location – at University Center (67% passing rate this year) Pool of 50 “Certified” EBOT Examiners for Exam Rotterdam j.mineiro 16

EBOT Exam experience and evolution ● Final EBOT Exam For the first time we had over 150 candidates for part II exam Part II oral ( 1 and 2nd of October 2016) - 245 candidates registered for written exam 161 for the English and 84 for the Spanish (10-20% will not proceed) Explicar q o exame de outubro é o oral Location – Lisbon (Marriott Hotel ??) + Madrid (conv centre??) Pool of 75 “Certified” EBOT Examiners for Exam in Lisbon Pool 40 “Certified” EBOT Spanish Examiners for Exam Madrid j.mineiro 16 *all costs of the Spanish EBOT Exam are covered by SECOT

EBOT Exam experience and evolution ● Final EBOT Exam ▲ addressing the two different issues optimizing results : A.Exam preparation B.Final assessment ● B.Final Assessment (EBOT Fellowship Examination) : - create different exams centres in October, same day but different languages – this autmn english + german* + spanish ● in order to keep the high standards of the exam we are enquiring with the EBOT pool of examiners, a second language they would be comfortable examining – building pool examiners different languages j.mineiro 16

EBOT Exam experience and evolution ● Interim & Final Exam ▲ EBOT Examination – how will it be structured : - Interim Exam – European wide, run all around Europe in the different hospitals / centres (more countries on board) – 12th April - Written Part I final – European wide, run all around Europe different centres in each country – English – 9th June - Oral Part II final – European, run at one or two centres in Europe, in English and other languages by multinational faculty appointed by national societies – +Spanish – 1 & 2nd October j.mineiro 16

EBOT Exam experience and evolution

▲ what about costs ? ● Logistics/development + Final & Interim Exam EBOT Exam experience and evolution ◙ from 2015 EBOT Exam ● Logistics/development + Final & Interim Exam ▲ what about costs ? j.mineiro 16

EBOT Exam experience and evolution ◙ from 2015 EBOT Exam ● Final Exam ▲ we are addressing COSTS for optimizing results : ● written part I – 300€ + oral part II – 500€ for EU citizens 200€ for residents and 900€ for non-europeans Pearson Vue gets 225€ / candidate j.mineiro 16

EBOT Exam experience and evolution ◙ how can we reduce costs ● optimize the resources – from an European perspective Accept what is different + centralize what is in common - Written exam with Pearson Vue – CESMA negotiation - Oral exam registration and preparation for the event - Approved budget annually for secretariat costs Explicar q o exame de outubro é o oral - Interim Exam – CESMA negotiation ● National Societies / CESMA Examiners + MCQ Courses j.mineiro 16

EBOT Exam experience and evolution ◙ near future - plans for 2020? ● Final Exam ▲ Final EBOT Examination – how may it be structured : - Written Part I – European wide, run all around Europe different centres in each country - Clinical Part II – National, run in each country, by their own board in their own language with examiners appointed Nat Soc+EBOT - Oral Part III – European, run at different centres in Europe, in english and other languages by multinational faculty appointed by the national societies ● PRIORITY is multilingual oral part II j.mineiro 16

EBOT Exam experience and evolution ◙ looking into the future – at the long run ● how do i envision the future Combined venture at European + National level - Written exam all around EU at European level - Oral exam will be at the national level with input from EBOT Explicar q o exame de outubro é o oral - Costs will be shared by both “bodies” - Central Secretariat EFORT ??? / UEMS ??? / other ???? ● Develop other means of sponsoring education / assessment at the end of specialized training j.mineiro 16

EBOT Exam experience and evolution ◙ decisions to be made ● trend across EU is for European Qualification to succeed ● we are achieving in exam what we have always aimed for! ● we are getting closerrrrr to the national recognition of the EBOT Diploma - we are at the turning point ! Explicar q o exame de outubro é o oral ► What should be EBOT/UEMS-Ort strategy for the exam over the next 15 years ? Should we proceed and develop it further ? Should we stop what we are doing ? j.mineiro 16

EBOT Exam experience and evolution ◙ conclusion ● European Board Exams are gaining broader base of support Europe wide and throughout the world ● more and more standards of these exams are being accepted ● tendency within EU is for European qualifications to take relevance over the national ones ● need to support the partnership with EFORT for the assessment at the end of training (advantages over UEMS central office) Sustainability ?? ● we do need to work together on finding EU “sponsoring” for development of final assessment at the end of speciality training as a gold standard qualification across the EU j.mineiro 16

EBOT Exam experience and evolution ◙ introduction ● “bureaucracy” j.mineiro 16