Developing Satisfying and Effective Feedback for Medical Students David Hope, Avril Dewar, Kyle Gibson, Jamie Davies & Helen Cameron June 2014.

Slides:



Advertisements
Similar presentations
Demanding Questions and Difficult Answers Alan Maddocks Carol Newbold Loughborough University.
Advertisements

Completing the cycle: an investigation of structured reflection as a tool to encourage student engagement with feedback Jackie Pates Lancaster Environment.
Drafting an Improvement Plan Using NSS Data Catherine Rendell – Deputy Director Academic Quality Assurance and Enhancement, University of Hertfordshire.
Alastair Wright 4 th year MBBS. Comments taken from Year 4 feedback for terms 1 and 2 of academic year.
Academic assessment of work placement – made easy?
A Department of Geographical and Life Sciences b Learning and Teaching Enhancement Unit
Evaluating two feedback mechanisms for MCQ exams David Hope, Kyle Gibson & Helen Cameron August 2013.
Introduction to the Fundamentals of Nursing Practice – Assessment through OSCE
Personal Tutoring - What students want : First Year Students Experiences Background:  Transition:  Transition: many not prepared for managing the demands.
The importance of Online and Computer aided assessment in E-Learning E.C Makazhe – Computer Science.
Mathematics Support Centres: Who uses them & who doesn’t? Why and why not? 1.
Oxford Centre for Staff and Learning Development Student assessment: lightening the load while increasing the learning Dr Chris Rust Head, Oxford Centre.
Choosing Oxford. 2 Who am I? Dr Stephen Drape Access and Schools Liaison Officer for Computer Science (Also a Departmental Lecturer) 8 years at Oxford.
Improving Students’ understanding of Feedback
A mathematics support program for first year engineering students 1 School of Mathematics and Physics The University of Queensland 23 June 2014.
Evaluation of a Novel Mentorship Programme for Medical Students in Paediatrics Benita Morrissey Paul Winyard Caroline Fertleman April 2015.
General practitioners caring for people with dementia and their carers Dr Catherine Speechly 1, Ms Belinda Giles 1, Prof Charles- Bridges-Webb 1, Dr Yvonne.
The Evaluation of Training for IAPT therapists in Cumbria Professor Dave Dagnan Consultant Clinical Psychologist.
International Conference on Enhancement and Innovation in Higher Education Crowne Plaza Hotel, Glasgow 9-11 June 2015 Welcome.
Oxford Centre for Staff and Learning Development Student assessment: lightening the load while increasing the learning Dr Chris Rust Head, Oxford Centre.
An investigation of the impact of student support initiatives on the retention of computer science students Clem O’Donnell 1, James Murphy 2, Abdulhussain.
Oral pathology in a PBL course Dr Mary Toner School of Dental Science Trinity College, Dublin.
The Student Experience Project Overview for Kosovo Higher Education visit Mark Wilkinson October 2014.
Formal Appraisal of Undergraduates – worth the effort? Deborah Murdoch-Eaton Professor of Medical Education Leeds.
Five things I have learned from students about feedback Dr Georgina Fyfe School of Biomedical Sciences Curtin University Western Australia.
1 Beyond the Library: i-Skills for University Administration © Netskills, Quality Internet Training, Newcastle University Partly.
Principal’s teaching award Scheme- Students as Partners Sally Anne Argyle, Caroline Hahn, Jessie Paterson, Eoghan Clarkson and Susan Rhind Kristin Timiney,
Problem-based learning in a traditional curriculum
Transition, Engagement and Retention of First Year Computing Students Heather Sayers Mairin Nicell Anne Hinds.
International Conference on Enhancement and Innovation in Higher Education Crowne Plaza Hotel, Glasgow 9-11 June 2015 Welcome.
Faculty of Medical Sciences School of Biomedical Sciences Dr Edwin Williams Director of Studies
Enhancing the first-year learning experience: a case study from Biomedical Sciences Stephen McClean and Paul Hagan School of Biomedical Sciences University.
Lessons learnt and changed understanding: Can Machine Dynamics students survive without my lectures? Ian Howard Mechanical Engineering.
Learners’ Attitudes and Perceptions of Online Instruction Presented by: Dr. Karen S. Ivers Dr. JoAnn Carter-Wells Dr. Joyce Lee California State University.
Evaluation of Respondus assessment tool as a means of delivering summative assessments A KU SADRAS Project Student Academic Development Research Associate.
Lack of Learning or Lack of Studying? An Inquiry into Low Exam Scores Katherine M. Sauer Metropolitan State College of Denver February.
STEM Seminar – 23 rd November 2011 Overview Background Implementation of support initiative Methodology Evaluation Conclusions.
Assessment for Learning: supporting post graduate students’ Margo McKeever.
2005 Performance Development System Survey Human Resources Staff Meeting March 20, 2006.
Postgraduate Taught Experience Survey (PTES) 2010 Interim Results Dr Pam Wells Adviser, Evidence-Informed Practice.
Improving Academic Feedback - Turning the ship around with the power of 1000 students David Hope
2009 Teaching and Learning Symposium John H. Bantham Management & Quantitative Methods Establishing Student-Faculty Expectations in the Classroom.
Interpersonal Skills 4 detailed studies Health Psychology.
Maximizing ROI of Multi-Rater Feedback. Background CSAA had implemented 360 feedback process two years prior to my coming into my role. No formal mechanisms.
Knowledge and experiences about health literacy needs and assessment among fourth year nursing students in Thailand Sinsakchon Aunprom-me, MPH, Ph.D. (Target.
Final FRCA VIVA Course Evaluation 11 th and 12 th June 2009.
UNDERSTANDING 2012 NATIONAL SURVEY OF STUDENT ENGAGEMENT (NSSE) RESULTS Nicholls State University October 17, 2012.
Planning for NSS 2013 School of Medicine Manjit Bansal.
The Use of Formative Evaluations in the Online Course Setting JENNIFER PETERSON, MS, RHIA, CTR DEPARTMENT OF HEALTH SCIENCES.
Simulated Patients Improve Medical Student Comfort Level with Breaking Bad News and End of Life Issues Skotti Church, MD Carl J Fichtenbaum, MD, FACP University.
Final FRCA VIVA Course Evaluation 25 th and 26 th November 2009.
30/10/2006 University Leaders Meeting 1 Student Assessment: A Mandatory Requirement For Accreditation Dr. Salwa El-Magoli Chair-Person National Quality.
Presented by Dr. Vivek Indramohan Faculty of Health / CHSCR BCU Evaluating student experiences when undertaking an objective structured clinical examination.
Enhanced Induction to Postgraduate Learning for Geographically Diverse Students. Nigel Coates & Dr Diane Sloan Newcastle Business School Northumbria Conference.
Graduate School of Medicine University of Wollongong Educators learning together and modeling interprofessional collaboration Sue Vella, Sal Sanzone, Helen.
Student Experience in Optometry Education in Mozambique: Initial challenges in an international collaborative program Shah, K. 1, Naidoo, K. 2,3, Loughman,
Race for Equality – A report on the experiences of Black students in further and higher education
Case of the Month – using the virtual learning environment to consolidate learning in the final year of medical school CaseoftheMonth Dr Sarah Bennett,
Postgraduate Taught Experience Survey (PTES) 2010 Interim Results
What Works at Wolves? A roll out in the Institute of Sport
Formal Feedback about the ‘Bridge’ (Internally collated)
Irish Survey of Student Engagement (ISSE) Results 2016
Factors facilitating academic success: a student perspective
Readiness Consultations
David Hope, Avril Dewar and Helen Cameron Background
Jade Purtell Stewart Muir
Derek Herrmann & Ryan Smith University Assessment Services
How “Appeals” and Exam Estimates work
Learning and Teaching Conference
Presentation transcript:

Developing Satisfying and Effective Feedback for Medical Students David Hope, Avril Dewar, Kyle Gibson, Jamie Davies & Helen Cameron June 2014

Background Feedback is often ad-hoc, untested, sporadically delivered and does not actively solicit student views What do students want from their feedback? Can feedback improve performance? How do we evaluate the needs of at-risk students? What is everyone else in the UK doing?

Background £15,000 to investigate feedback in the medical school How does satisfaction with feedback link to performance change? What do our ‘at-risk’ students think about feedback? Can detailed feedback improve performance?

Edinburgh Feedback Project 750 students from years 1, 3 and 5 took an online questionnaire (repeated over 3 years) Tests completed: – 50-item personality inventory (IPIP) measuring Extraversion, Emotional Stability, Agreeableness, Conscientiousness and Openness to Experience – Measure of satisfaction with feedback – Assessment information – Edinburgh Feedback Inventory

The Edinburgh Feedback Inventory We scored feedback from a series of individual statements – “Being given motivation to improve is …” – Getting feedback from a staff member who knows you is …”

Feedback maturation Question (1 to 6 – 6 is maximum)LeedsEdinburgh I consider it important to get feedback on my work5.45.1* I think I receive enough feedback3.7*3.0* I understand what types of feedback are appropriate for the type of course/project/core unit I am studying 3.7*3.9 I know what to do when I get feedback I know where to get more feedback if I need it3.1*2.5*

Summary of results Students who answered the questionnaire were typically half a grade better than students who did not answer There was no association between feedback rating, satisfaction with feedback and academic performance (rs = < 0.1)

‘At risk’ students - Dr Kyle Gibson Junior Doctor in Intensive Care Medicine Significant interest in Medical Education Graduated from Edinburgh Medical School 2010 Have personal (and fairly recent) insight into curriculum and therefore relate well to students Developed interest in enhancing feedback experience for students particularly as I regularly teach and assess across the curriculum

At-risk students One-to-one interviews with nine students – all had recently failed assessment or narrowly passed it Main findings – Desire for frequent formative exams – Tailored one-to-one feedback – Reluctance to seek help due to perceived stigma of exam failure – Desire for allocated member of staff to provide feedback and pastoral support throughout their time at medical school – Guidelines for meeting with students after failing were not followed – Administrative staff were seen as safer and less threatening – Metacognitive skills among struggling students seem especially poor

At-risk students “I feel like feedback should be limited to what they’ve either seen you do or a piece of work they’re marking you on.” There was some...there was exam feedback lectures but, again, I found them...they didn’t really help me at all. It was just kind of like fair enough, it’s past now.” “They... ed out a list to the people that failed of all the different study groups. And I know for a fact that some people were very upset about that because they didn’t want everyone to know that they failed.”

What I learned Development of medical education skills (designing, implementing and evaluating assessments) Qualitative research skills (all my previous work had been quantitative) Development of project and time management skills (whilst working concurrently as a junior doctor) Whilst analysing and disseminating results from our project work, I can use the insights I have learned when providing feedback to my own students

Is it useful to give detailed feedback? 196 students (completion rate over 95%) sat a 30 minute formative cardiovascular exam OSCA-FM: 50% received detailed, immediate feedback explaining each option for each question but they could not take the feedback away Tagging: 50% received a summary of their performance on a number of domains and they could keep this indefinitely Students sat a further 30 minute formative exam three weeks later

Evaluating two feedback mechanisms

TagScoreAverageMax Cardiovascular Biomedical Sciences Presentation Diagnosis and Management Clinical pharmacology and therapeutics Anatomy999 Physiology Pathology81116 Chest pain323 Congenital Heart Disease81116 Lipids333 Biochemistry ECG999

Evaluating two feedback mechanisms Students performed similarly on the follow up formative exam – OSCA-FM 73.0 – Tagging 68.8 – Previous cohort 69.5 No significant differences on this or final cardiovascular exam performance after controlling for prior academic ability

What I learned Students cannot accurately self-rate the EFFECTIVENESS of feedback Personalised feedback significantly increases student satisfaction and well-being Students cannot improve performance without focusing on metacognitive skills BUT most students believe their metacognitive skills are fine

Further work We saved a significant sum when the year 1 cardiovascular module released their questionbank to us We tackled two new topics with the extra funds: – What do other UK institutions – with significantly higher feedback ratings – do? – What do our (practicing UK) graduates think of our feedback, looking back on the Edinburgh experience?

Background Graduated from University of Edinburgh 2007 with an MA (Hons) degree in Psychology and Sociology Coaching and mentoring role in private sector Graduated from Edinburgh Napier University in 2012 with BN in Adult Nursing.

Edinburgh Graduates What is good feedback – to a graduate? How to prepare for the junior doctor years How feedback can be improved in light of practise

Graduate Findings Overall positive experience “The course prepared me to be a lot more than an FY1. It prepared me to almost be a registrar” “I think from the clinical perspective formative feedback is fairly lacking” “From the time that I failed the [finals] I felt I got support in terms of feedback”

OSCE Survey Project Audit of UK medical schools Use of written/formative feedback Use of supporting exam software

OSCE Survey Project Findings 17 of 31 schools responded Big variation in feedback usage 1 school gave no formative OSCE feedback Some of the highest rated for feedback gave very little formal feedback

Conclusions Feedback increases student happiness, but engagement with metacognitive skills is required to increase performance At-risk students struggle with metacognitive skills Many staff practices are unhelpful, and cause distress among students: we can fix this while trying to improve the quality of feedback “Poor” feedback goes beyond the academic: many places that give objectively poor feedback get excellent ratings!