1. Exams and assessing competence

2. Part A/Paper 1 changes Part A/Paper 1 changesOSPHEARCP Assessing competence in HK training – the role of exams

3. Part A changes -What Paper I - ‘Knowledge Paper‘ – No change Paper II - ‘Skills Paper‘ Proposed change - 2b to become a short answer paper – 90 minutes, five sections (10 marks per section) – short questions with short answers (numerical or no more than a paragraph) - no choice – questions will not be published on the website

4. Part A – Why change? PMETB - PMETB - direction of travel for work that must be done to ensure that the MFPH is part of an assessment programme that enables the public, the profession and the candidates/trainees to have confidence in those decisions taken on the basis of examination results Changes in response to long term aspirations of Southgate review

5. When Needs PMETB approval before final confirmation (PMETB panel meeting January 2009) Commencement set for Jan 2010

6. The OSPHE “Objective Structured Public Health Examination”

7. Why the need? Validity of the General Viva Training issues with time spent on thesis Issues around the limited competencies assessed by thesis Issues around the narrow range of assessment available with thesis marking and viva

8. OSPHE –part B MFPH After Part A (Knowledge & Knows How) OSPHE (Shows How) ReliableValid Will encourage participation in service training opportunities during training as this will be the best form of preparation.

9. Milestones  Part A – 16m after starting Masters  Part B – 6-9m wte after Part A

10. What is assessed in the OSPHE? 1. Presenting communication skills in a typical Public Health setting (presenting to a person or audience). 2. Listening communication skills in a typical Public Health setting (listening and ascertaining key information). 3. Demonstrating ascertainment of key Public Health points from the material provided and using it appropriately and in relation to wider Public Health information sources. 4. Giving a balanced view/explaining appropriately key Public Health concepts in a Public Health setting. 5. Appropriately and sensitively handling uncertainty, the unexpected, conflict and/or responding appropriately to challenging questions. ALL the above are part of Curriculum

11. The OSPHE It is a test of 5 competencies – NOT performance at individual stations It is possible to not do very well at 2 out of the 6 stations – and pass Feedback is given as a score for each competency overall - not overall station scores.

12. FORMAT 12 Stations of 8 mins in each. 6 of them are briefing/preparation time for the following assessed stations Giving a short presentation Explaining something verbally to a health professional, lay person, representative, manager etc Media Interview role play Phone conversation Small meeting

13. OSPHE - format Candidates have 6 stations which equally and independently tests all five competencies Each station lasts 17 mins 8 mins of briefing/preparation time 1 min changeover 8 mins assessment time

14. Marking Examiner watching and marking during exam 5 grades Weightings Automated scoring

15. The OSPHE Taken around mid-point stage of training We do not require candidates to be familiar with NHS procedures or any particular culture. Question & Marking examples on Faculty website www.fph.org

16. Advice to post part A trainees “Spend as little time as possible at desk using PC” “Spend as much time as possible doing active service PH ; meetings, discussions with other professionals, managers, lay representatives of interest groups, discussions with journalists, radio interviews etc

17. As the Trainer Ensure their service work/projects are in keeping with lots of talking & interaction Every week before trainer-trainee meeting think of something which has happened over last week suitable for a mini-OSPHE Role play it with your trainee – then give them constructive feedback Try to be able to observe your trainee in meetings etc – and give them feedback

18. 350 Euston Road

19. Waiting Area

20. Briefing Room

21. Exam Corridor

22. Prep Room

23. Exam Room

24. Assessment Exams Work based assessments Includes in-work and real-time assessments, directly observed procedures (DOPs), case-based discussions (CBDs), multi- source feedback (MSF)

25. Evidence Presented through portfolio In a variety of formats (including supervisor reports) Assessed at ARCP

26. ARCP - Annual Review of Competency Progression Deanery process Requires reports from Educational Supervisor & Academic Supervisor Based on documentation only –Sign-off sheets –Sample of workplace assessments –Sample of log-sheets –Other records (MSF, exam etc)

27. Documents Required for the Review by the ARCP Panel Educational Supervisors reports to include any identified problems with competencies Copies of all WBAs completed in that year Updated CV – to include a sheet indicating new achievements since last review such as: examinations, audits, presentations and posters, research, record of CPD for that year, details of placements for that year. Personal development plan for the year Results of MSF Printout of log book if relevant

28. Educational Supervisor Report

29. Academic Supervisor Report

30. Workplace Assessment Record Sheets

31. ARCP Panel Consider the evidence presented Consider progress against time-line Decide on outcome Provide feedback (Involves lay representation)

32. How the Panel Works Convened by the deanery - a minimum of 5 members Chaired by the chair of the STC, TPD or Associate Dean Process is an assessment of documentation and submitted evidence Trainee should not normally attend the panel meeting Panel does not plan remedial programme Other issues may be discussed by the panel but assessment of evidence and judgement must be kept separate

33. ARCP Outcomes 1 – Satisfactory progress 2 – Specific development needed 3 – Inadequate progress 4 – Left programme 5 – Incomplete evidence 6 – Completed training

34. Outcomes from the ARCP panel 1. Satisfactory progress – achieving progress and competences at expected rate. 2. Unsatisfactory progress – further development of specific competencies required but additional training time NOT required i.e. acceptable progress overall and not expected to delay training. TPD and ES will make sure the trainee and employer know what must be done.

35. Continued…... 3. Inadequate progress and additional training time required – panel identifies need for additional training time and recommends what is required and the level of supervision BUT the deanery determines the details depending on local circumstances and resources. The total overall extension should be no more than 1 year. An absolute maximum of 2 years during the total of training can only be granted by the Postgraduate Dean. Time can be made up of shorter periods of time divided over the course of training.

36. Continued…... 4. Released from programme with or without specified competencies. The panel can recommend a release from programme if insufficient and sustained lack of progress is seen despite additional training time. Any achieved competencies are recorded and documented. The trainee must give up their training number and advice on career options should be made available.

37. Continued…... 5.Incomplete evidence presented – additional training time may be required. The trainee supplies incomplete or insufficient information such that the panel cannot make a statement about progress. 6.Completion of training – all necessary competencies have been achieved

38. Comments on ARCP Just started Systematic and uniform for all specialities Moving to electronic portfolio

39. e-portfolio Currently being piloted with a group of trainees & trainers Planned to be up and running for all in January 2009 A Allows every functionality encompassed by paper based portfolio Gives interaction between educational supervisor and trainee Evidence can be stored either confidentially or shared Able to support any file format (pdf excel word ppt music files etc) MSF incorporated

40. Front page for a trainee

41. Front page as a trainer

42. View portfolio

43. Assessments

44. Diary

45. Activities e.g. presentations

46. Add evidence

47. Benefits e-portfolio Hold record of trainee work, trainers’ assessments Avoid time wasted on formatting Avoid time wasted on re-typing Allow trainers and trainees to keep track of their communications Summarise progress (for trainees, trainer and TPD) Summarise what still needs to be done

48. Assessing competence in HK training – the role of exams Part A/Part 1 Part B (OSPHE)/Part 2 ?Portfolio assessment

49. Possible scenarios Do nothing Replace Part 2 General Viva with OSPHE and keep thesis examination Replace Part 2 General Viva with OSPHE and replace thesis examination with formal and structured portfolio examination

50. Replacing general viva with OSPHE OSPHE structure validated, tried and tested Scenarios will need to be HK relevant Examiner training

52. Group Work – Assessing competence in HK training – the role of exams Part 1/A exam – is it appropriate for HK? What is the role of Part 2 general viva exam? What is being assessed and is the viva exam the best way to assess it? What are the advantages and disadvantages (problems) of the Part 2 general viva exam for HK? What are the benefits of replacing the general viva with the OSPHE? What are the disadvantages, problems and barriers to replacing the general viva with the OSPHE?

53. Groupwork What is the role of Part 2 thesis exam? What is being assessed and is this exam the best way to assess it? What are the advantages and disadvantages (problems) of the Part 2 thesis exam for HK? What are the benefits of replacing the thesis exam with a portfolio examination? What are the disadvantages, problems and barriers to replacing the thesis exam with a portfolio examination?

54. Group work – messages to take back to the FPH About Part A changes About the relevance or not of the OSPHE exam for HK – give reasons

55. Group session If we started from scratch how would we design a PG training programme in PH? 1. Recruitment 2. Structure Think about what you have now, what you want to keep and what you want to change and why.

56. Feedback from groups What did your ideal training programme consist of? From the ideas you have generated for an ideal training programme what do you think is achievable in the short term? long tem?