Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry

Psychiatry Examination 80 EMIs 80 EMIs 1 hour 15 minutes 1 hour 15 minutes Standard set Standard set 2 OSCE stations 2 OSCE stations Each station 10 minutes Each station 10 minutes 3 tasks based on competencies 3 tasks based on competencies % given for each task % given for each task

May OSCE StationMaxMinAll A60.1(14.3)46.4(5.6)54.5(10.3) B 60.8 (6.6) 48.2(4.4)55.5(9.0) C66.1(8.5)49.0(9.7)56.2(9.8) D67.7(12.8)47.2(4.1)54.3(10.8)

Psychiatry OSCE Changes to the OSCE Changes to the OSCE Criterion marking scheme Criterion marking scheme Global clinical assessment Global clinical assessment Honours Honours Pass Pass Borderline Borderline Fail Fail 2 examiners were placed in each room 2 examiners were placed in each room Standard set pass mark for each task Standard set pass mark for each task

Task Develop CAL package: Train the examiners in the new marking scheme Train the examiners in the new marking scheme Standardise marking scheme Standardise marking scheme To allow for multiple sites to accommodate increasing numbers of students To allow for multiple sites to accommodate increasing numbers of students Improve inter-rater reliability Improve inter-rater reliability

Development of Package Scripted 10x 2-3 minute OSCE scenarios Scripted 10x 2-3 minute OSCE scenarios Borderline/Pass Borderline/Pass Pass/Honours Pass/Honours 2 hours filming in my office 2 hours filming in my office 2 lecturers 2 lecturers 1 bullied student 1 bullied student

Creation of Media wmv files converted to flv using Flash Macromedia wmv files converted to flv using Flash Macromedia Images created utilising Flash Macromedia Images created utilising Flash Macromedia MP3 creation using Audacity MP3 creation using Audacity

Xerte Template ‘ratings’ created by Julian Tenney Template ‘ratings’ created by Julian Tenney Media was imported into Xerte through the Wizard Media was imported into Xerte through the Wizard

The Package An introduction to the new marking scheme An introduction to the new marking scheme Example mark sheets Example mark sheets Video of example students Video of example students Rated on criteria Rated on criteria Global impression given Global impression given User then offered feedback User then offered feedback

Using the Wizard Once Wizard create – efficient method of presenting content Once Wizard create – efficient method of presenting content Define scenario Define scenario Define marking sheet, including number of criteria Define marking sheet, including number of criteria Define fail/borderline/pass/honours Define fail/borderline/pass/honours Import media Import media

Sample of Package

Distribution of Package Letters and s were sent to all examining consultants with following link Letters and s were sent to all examining consultants with following link Package ran for ~20 minutes Package ran for ~20 minutes

Result of Package Introduction 30% examiners accessed the package prior to the OSCE 30% examiners accessed the package prior to the OSCE Remainder underwent abridged version before the OSCE Remainder underwent abridged version before the OSCE

Analysis ANOVAs undertaken on awarded scores by the different examiners on the same station ANOVAs undertaken on awarded scores by the different examiners on the same station

December OSCE StationMaxMinAll A70.3(5.0)53.4(8.6)62.3(8.6) B72.6(9.1)52.1(8.85)63.7(11.5) C66.1(5.1)51.0(7.6)56.9(8.6) D66.1(56.7)56.6(5.0)60.7(7.4)

**p<0.001 *p<0.05 MayDecember OSCE **(10.3)62.3**(8.6) OSCE **(9.77)63.7**(11.5) OSCE **(9.02)56.9**(8.6) OSCE **(10.8)60.7(7.4)

Variation StationMayDec A B C D

Limitations Flash Player 7 not installed as standard on healthcare trusts’ computers Flash Player 7 not installed as standard on healthcare trusts’ computers Minor problems with Xerte - since resolved Minor problems with Xerte - since resolved Delay on slow broadband connections Delay on slow broadband connections ?Representative videos ?Representative videos Any improvement likely to be due to the new marking scheme Any improvement likely to be due to the new marking scheme

Xerte Once familiar: Easy to use Easy to use Multimedia packages can be create in minutes Multimedia packages can be create in minutes Easily distributed Easily distributed Much more complicated to create bespoke packages Much more complicated to create bespoke packages

Future Improve IT access in the healthcare trusts Improve IT access in the healthcare trusts Reduce bitrate Reduce bitrate Produce improved examples of borderline students Produce improved examples of borderline students Extend the package to attempt to standardise the Formative assessments Extend the package to attempt to standardise the Formative assessments ?Make available to students ?Make available to students