Exams and assessing competence

Part A/Paper 1 changes Part A/Paper 1 changesOSPHEARCP Assessing competence in HK training – the role of exams

Part A changes -What Paper I - ‘Knowledge Paper‘ – No change Paper II - ‘Skills Paper‘ Proposed change - 2b to become a short answer paper – 90 minutes, five sections (10 marks per section) – short questions with short answers (numerical or no more than a paragraph) - no choice – questions will not be published on the website

Part A – Why change? PMETB - PMETB - direction of travel for work that must be done to ensure that the MFPH is part of an assessment programme that enables the public, the profession and the candidates/trainees to have confidence in those decisions taken on the basis of examination results Changes in response to long term aspirations of Southgate review

When Needs PMETB approval before final confirmation (PMETB panel meeting January 2009) Commencement set for Jan 2010

The OSPHE “Objective Structured Public Health Examination”

Why the need? Validity of the General Viva Training issues with time spent on thesis Issues around the limited competencies assessed by thesis Issues around the narrow range of assessment available with thesis marking and viva

OSPHE –part B MFPH After Part A (Knowledge & Knows How) OSPHE (Shows How) ReliableValid Will encourage participation in service training opportunities during training as this will be the best form of preparation.

Milestones  Part A – 16m after starting Masters  Part B – 6-9m wte after Part A

What is assessed in the OSPHE? 1. Presenting communication skills in a typical Public Health setting (presenting to a person or audience). 2. Listening communication skills in a typical Public Health setting (listening and ascertaining key information). 3. Demonstrating ascertainment of key Public Health points from the material provided and using it appropriately and in relation to wider Public Health information sources. 4. Giving a balanced view/explaining appropriately key Public Health concepts in a Public Health setting. 5. Appropriately and sensitively handling uncertainty, the unexpected, conflict and/or responding appropriately to challenging questions. ALL the above are part of Curriculum

The OSPHE It is a test of 5 competencies – NOT performance at individual stations It is possible to not do very well at 2 out of the 6 stations – and pass Feedback is given as a score for each competency overall - not overall station scores.

FORMAT 12 Stations of 8 mins in each. 6 of them are briefing/preparation time for the following assessed stations Giving a short presentation Explaining something verbally to a health professional, lay person, representative, manager etc Media Interview role play Phone conversation Small meeting

OSPHE - format Candidates have 6 stations which equally and independently tests all five competencies Each station lasts 17 mins 8 mins of briefing/preparation time 1 min changeover 8 mins assessment time

Marking Examiner watching and marking during exam 5 grades Weightings Automated scoring

The OSPHE Taken around mid-point stage of training We do not require candidates to be familiar with NHS procedures or any particular culture. Question & Marking examples on Faculty website

Advice to post part A trainees “Spend as little time as possible at desk using PC” “Spend as much time as possible doing active service PH ; meetings, discussions with other professionals, managers, lay representatives of interest groups, discussions with journalists, radio interviews etc

As the Trainer Ensure their service work/projects are in keeping with lots of talking & interaction Every week before trainer-trainee meeting think of something which has happened over last week suitable for a mini-OSPHE Role play it with your trainee – then give them constructive feedback Try to be able to observe your trainee in meetings etc – and give them feedback

350 Euston Road

Waiting Area

Briefing Room

Exam Corridor

Prep Room

Exam Room

Assessment Exams Work based assessments Includes in-work and real-time assessments, directly observed procedures (DOPs), case-based discussions (CBDs), multi- source feedback (MSF)

Evidence Presented through portfolio In a variety of formats (including supervisor reports) Assessed at ARCP

ARCP - Annual Review of Competency Progression Deanery process Requires reports from Educational Supervisor & Academic Supervisor Based on documentation only –Sign-off sheets –Sample of workplace assessments –Sample of log-sheets –Other records (MSF, exam etc)

Documents Required for the Review by the ARCP Panel Educational Supervisors reports to include any identified problems with competencies Copies of all WBAs completed in that year Updated CV – to include a sheet indicating new achievements since last review such as: examinations, audits, presentations and posters, research, record of CPD for that year, details of placements for that year. Personal development plan for the year Results of MSF Printout of log book if relevant

Educational Supervisor Report

Academic Supervisor Report

Workplace Assessment Record Sheets

ARCP Panel Consider the evidence presented Consider progress against time-line Decide on outcome Provide feedback (Involves lay representation)

How the Panel Works Convened by the deanery - a minimum of 5 members Chaired by the chair of the STC, TPD or Associate Dean Process is an assessment of documentation and submitted evidence Trainee should not normally attend the panel meeting Panel does not plan remedial programme Other issues may be discussed by the panel but assessment of evidence and judgement must be kept separate

ARCP Outcomes 1 – Satisfactory progress 2 – Specific development needed 3 – Inadequate progress 4 – Left programme 5 – Incomplete evidence 6 – Completed training

Outcomes from the ARCP panel 1. Satisfactory progress – achieving progress and competences at expected rate. 2. Unsatisfactory progress – further development of specific competencies required but additional training time NOT required i.e. acceptable progress overall and not expected to delay training. TPD and ES will make sure the trainee and employer know what must be done.

Continued… Inadequate progress and additional training time required – panel identifies need for additional training time and recommends what is required and the level of supervision BUT the deanery determines the details depending on local circumstances and resources. The total overall extension should be no more than 1 year. An absolute maximum of 2 years during the total of training can only be granted by the Postgraduate Dean. Time can be made up of shorter periods of time divided over the course of training.

Continued… Released from programme with or without specified competencies. The panel can recommend a release from programme if insufficient and sustained lack of progress is seen despite additional training time. Any achieved competencies are recorded and documented. The trainee must give up their training number and advice on career options should be made available.

Continued…... 5.Incomplete evidence presented – additional training time may be required. The trainee supplies incomplete or insufficient information such that the panel cannot make a statement about progress. 6.Completion of training – all necessary competencies have been achieved

Comments on ARCP Just started Systematic and uniform for all specialities Moving to electronic portfolio

e-portfolio Currently being piloted with a group of trainees & trainers Planned to be up and running for all in January 2009 A Allows every functionality encompassed by paper based portfolio Gives interaction between educational supervisor and trainee Evidence can be stored either confidentially or shared Able to support any file format (pdf excel word ppt music files etc) MSF incorporated

Front page for a trainee

Front page as a trainer

View portfolio

Assessments

Diary

Activities e.g. presentations

Add evidence

Benefits e-portfolio Hold record of trainee work, trainers’ assessments Avoid time wasted on formatting Avoid time wasted on re-typing Allow trainers and trainees to keep track of their communications Summarise progress (for trainees, trainer and TPD) Summarise what still needs to be done

Assessing competence in HK training – the role of exams Part A/Part 1 Part B (OSPHE)/Part 2 ?Portfolio assessment

Possible scenarios Do nothing Replace Part 2 General Viva with OSPHE and keep thesis examination Replace Part 2 General Viva with OSPHE and replace thesis examination with formal and structured portfolio examination

Replacing general viva with OSPHE OSPHE structure validated, tried and tested Scenarios will need to be HK relevant Examiner training

Group Work – Assessing competence in HK training – the role of exams Part 1/A exam – is it appropriate for HK? What is the role of Part 2 general viva exam? What is being assessed and is the viva exam the best way to assess it? What are the advantages and disadvantages (problems) of the Part 2 general viva exam for HK? What are the benefits of replacing the general viva with the OSPHE? What are the disadvantages, problems and barriers to replacing the general viva with the OSPHE?

Groupwork What is the role of Part 2 thesis exam? What is being assessed and is this exam the best way to assess it? What are the advantages and disadvantages (problems) of the Part 2 thesis exam for HK? What are the benefits of replacing the thesis exam with a portfolio examination? What are the disadvantages, problems and barriers to replacing the thesis exam with a portfolio examination?

Group work – messages to take back to the FPH About Part A changes About the relevance or not of the OSPHE exam for HK – give reasons

Group session If we started from scratch how would we design a PG training programme in PH? 1. Recruitment 2. Structure Think about what you have now, what you want to keep and what you want to change and why.

Feedback from groups What did your ideal training programme consist of? From the ideas you have generated for an ideal training programme what do you think is achievable in the short term? long tem?