Formal Appraisal of Undergraduates – worth the effort? Deborah Murdoch-Eaton Professor of Medical Education Leeds

The workshop Who are we? What do you want to get out of the day?

Proposed Outcomes Explored whether formal appraisal might be of benefit for students &/or faculty developed a model for undergraduates Considered whether it might be applicable in your institution ie “worth the effort”?

What is appraisal? 1 to 1 interview Purpose? (Sam Scallan review of educational appraisal)

Purpose of Appraisal Review performance Provide opportunity to discuss progress Develop personal and professional development needs (contribute to University requirements for PDP)

Is appraisal different for UG and PG and CPD? Purpose ? Type of goals set? Skills of reflection ? Students? Appraiser Appraiser skills

Group work Benefits to student Benefits to faculty What stage of UG career might it be considered most valuable? Who should do the appraisals?

Feedback from groups

Benefits to student? Individuality Reassurance re progress and grades Life at university Feedback – on more than academic criteria Achieve potential Model for future ? Reflective practice?

“A more positive feeling about how I am doing. I truly believed I was doing badly, and without this appraisal I would have maintained this disheartening and negative feeling”

Benefits to faculty? Detection Alternative view on course Enhance student support system?

Who gets appraised Does it depend on the medical school Type of curriculum? All students? Selected? Performance Year Self-need

Who are the appraisers? Different from PG and / or CPD model? Chosen Knowledge of curriculum Interest / understanding trained

Coffee break

methodology When do you do it? Preparation Documentation of outcomes?

methodology How and when Preparatory material Student Appraiser outcomes

Leeds Model When After Easter break Who All 1 st and 2 nd years, “selected” 3 rd years Is it “optional”? (Murdoch Eaton & Levene Med Teacher 2004)

Student preparation Written guidelines Preparatory questions Summative and formative assessment marks Individual (Study habits survey – data from available )

Preparatory questions for students – work / course do you think the results you obtained were a fair reflection of the work you put in? Think about how you worked – was it always appropriate and efficient? Give an example of where it went well, what you are pleased about and why.

Preparatory questions for students work / course Is there anything over the year you didn’t do as well in or learn as much from as you could have? How might you approach something like this differently again?

Preparatory questions for students – career choice Has anything this year changed your views on what being a doctor is about? Either made you less or more enthused about your career choice?

Preparatory questions for students – personal change How would you measure a successful next year? What would you like to do to ensure this happens (both personally and in your UG career)? Is there anything else you would like to discuss – personal or professional development?

Study Habits Survey Who knows how hard a student works? What is the “norm”?

Study Habits Survey One representative week in term 2 Document Number of hours private study (all course related activities which are unsupervised) Where, with whom Employment? Hours/week on sport / social activities? (Whittle S R & Murdoch Eaton D G. Study Habits audit. Medical Education 2004 (38) )

Study Habits Survey Number of Hours spent on self-directed learning

Study Habits Survey Hours spent sports / societies

Use of Study Habits Survey Results Not the absolute values for comparison Relative in relation to Academic results Outcomes Personal objectives i.e. A diagnostic tool??

Personal Tutor involvement Sent copy of students results Form to complete Relevant information Not breaking confidentiality

For the Appraisers Environment Information for appraisers Student specific: Individual results appraisal record – (yr 2 + 3) PT report General : preparatory guidelines prev results study habits survey Appraisal record form to complete with student

methodology How and when Preparatory material Student Appraiser Recording the outcomes

Qns on bottom of Appraisal record form Option to refer to Dean / Sub-Dean of students Use of progress file Contact / met with Personal Tutor Consider for 3 rd year appraisal

Appraisal record form What happens to the information? Student Personal tutor Student file Signed by both appraiser and student

Lunch break

Issues raised in appraisal 2 main outcome themes altered perception on personal progress Altered perception of faculty …….feel like a person, not just a statistic – the Med school does care!

Video clips

Tea break

So is appraising undergraduates worth the effort? Cost effectiveness ? Students ~ 600 Appraisers ~ 28 84X 3 hour sessions = 252 hours senior academics time Administration - complex

Appraisers perspective all would do it again Personally rewarding Timing about right Training of future appraisers?

Effect on Students? Student Feedback Questionnaire Quantitative and qualitative questions Student Outcomes Analysis of Record of Appraisal Outcomes

Outcomes – student perspective Questionnaire – quantitative analysis response rate (86%) 96% well informed about process 99% they could be honest in answers 99% topics covered were relevant 96% feedback was useful and accurate

Outcomes – student perspective Questionnaire – quantitative analysis (2) 94% able to discuss academic problems 98% appraisal record fair representation of the points discussed 62% appraisal feedback would alter their study habits 33% appraisal feedback would alter attitude towards medical career

Outcomes – student perspective Questionnaire – qualitative analysis Majority very satisfied with appraisal session Perspective of medical school changed “feel like I’m a person, not just a statistic” “the Med School does care”

“2 things you will take away” Personal Confidence, reassurance Work/social balance Healthy balance Study skills advice Time management, efficiency of working Academic progress C grade is fine

“Two things you will take away” “A more positive feeling about how I am doing. I truly believed I was doing badly, and without this appraisal I would have maintained this disheartening and negative feeling”

But does it change anything? Longer term study To identify from Appraisal Record Forms Pattern of advice and areas discussed Year 1 appraisal Year 2 appraisal What of year 1 goals are achieved by year 2 What are new goals for year 2, and what goals are carried forward from year 1 to 2 Correlation between factors

Analysis of Appraisal Outcomes 2 cohorts of students completed both 1 st and 2 nd year appraisals N=511 1.Anonymous student feedback questionnaires (88 % response rate) 2. Appraisal record forms (100% completion)

Feedback questionnaire - key points Overwhelming positive perception of the value of appraisal in both years 59.2% overall considered issues covered would alter study habits 51% of 2 nd years considered they had made changes based on previous years appraisal

Practice Points – key advice required early in undergraduates’ career Study skills guidance was the commonest need Some significant correlations were seen between types of advice required Evolution of personal goals - changing course requirements and maturation Early opportunity to address healthy work- leisure balance

Undergraduate appraisal Worth the effort? We think so Data would suggest that it is, from both students and faculty perspective The future? Later years????

Any questions / comments Thank you for coming and participating! I hope you found it valuable.