Realist evaluation Dr Geoff Wong Clinical Research Fellow geoffrey.wong@phc.ox.ac.uk 6th July 2016

Acknowledgements RAMESES II Project – to quality and reporting standards and training materials for realist evaluations (March 2015 to Feb 2017) Funded by NIHR HS&DR Programme (14/19/19) RAMESES I Project – to quality and publication standards and training materials for realist reviews (completed) Funded by NIHR HS&DR Programme (10/1008/07) The views and opinions expressed therein are those of the presenters and do not necessarily reflect those of the UK's National Institute of Health Research Health Services and Deliver Research (NIHR HS&DR), NIHR, National Health Service (NHS) or the Department of Health 2

Structure of this presentation Objectives Illustrating the value of realist evaluation Clarifications Summary

Objectives By the end of this presentation hopefully you will have an understanding of the value of undertaking a realist evaluation

Realist evaluation: A type of theory driven evaluation Start Evaluation questions General design Designing the evaluation ‘End’ Policy relevant theory Formulating the initial programme theory (PT) Refined programme theory (PT) Synthesis Translation of CMOCs into PT Refined evaluation design Design follows PT Data analysis Context-Mechanism-Outcome Configurations (CMOC) Data collection Multiple methods ADAPTED FROM: Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Bruno Marchal, Sara van Belle, Josefien van Olmen, Tom Hoerée and Guy Kegels. Evaluation 2012 18: 192

Illustration Grades in formative workplace-based assessment: a study of what works for whom and why. Janet Lefroy, Ashley Hawarden, Simon P Gay, Robert K McKinley & Jennifer Cleland Medical education 2015;49:307-320 Feedback with grades is important to students But a paradox is that “giving grades as part of feedback to students in school settings and elsewhere in undergraduate education can reduce the effectiveness of feedback and may reduce student performance.” In particular students might focus more on the grade than the feedback on how they might improve. From existing literature several factors impact seem to be in play: relationships between the recipient and giver state of mind and maturity of the recipient feedback process feedback content etc 6

Illustration 1 medical school, Year 3 students, 3 Work Based Assessment (WBA) feedback sessions (face to face with tutor +/- Grade) Randomised study Randomised Group 1: Grade, No grade, choice Group 2: No grade, Grade, choice Not randomised Grade, Grade, Grade Mixed data collected: Interviews at exit OSCE results Written summaries of WBA x3 7

Illustration 8

Illustration Built a programme theory from the literature: … if the student has a trusted assessor (external context) and a learning goal approach (internal context), he or she will find that grades (external context) clarify (mechanism) and energise (mechanism) his or her efforts to find strategies to improve (outcome). If students are more performance-oriented (internal context), they will find that satisfactory grades (external context) reassure (mechanism) and therefore reduce efforts to improve (outcome). … less sure about what we expected to find as the outcome of lower than expected formative grades ... 9

Illustration Some illustrative examples from their work: Effects of grades: Grades (C) help students to locate themselves clearly (M) so they know what needs improving (M), in the context of a trusted and consistent assessor (C) Grades diminishing effort (O) by complacency (M) in the context of a satisfactory grade (C) or by demoralisation (M) with a low grade (C), in a student damaged by previous assessments (internal C) How students handle their grades: Filtering (M) for self-protection from risk (O) in the context of low grades by exit standard (C)… 10

Illustration Moved from a list … relationships between the recipient and giver state of mind and maturity of the recipient feedback process feedback content etc. … to configurational explanations – realist context-mechanism-outcome-configurations (for more details see Tables 2 and 3 in the paper) 11

Illustration … to more nuanced guidance Grades are important to many medical students for a number of reasons. These reasons can be explored by a supportive tutor who can encourage a learning approach towards WBA, aiming for self-awareness of competency and the prioritising of areas for improvement. If it seems that the receipt of grades will enhance the seeking of strategies for improvement, grades should be offered as an element of formative feedback. The criteria by which grades are allocated must be understood by the students who receive them. Not all students will accept the offer of grades where choice is available, and this decision should be respected but perhaps explored by tutors. Feedback is processed in the light of previous feedback; hence all tutors must understand that today’s feedback affects the response to tomorrow’s feedback. 12

Illustration … and are potentially transferable. … understanding how and why feedback and grades ‘work’ for other learners could be explored by elucidating the behaviour of the following mechanisms in differing contexts: locating self clarifying progress focusing attention challenging complacency show progress self protection resolve dissonance meaning of grades filtering pumping etc. 13

Closing summary Realist research approaches help to partially unpack the ‘black box’ of what is going on: why things happen, how, for whom, in what contexts and to what extent To do so it uses a realist logic of analysis: C+M=O Produces findings that explain how context affects outcomes Explicit warrant for why knowledge from one study may be transferable to another setting – via mechanisms. But ‘testing’ is needed in the new setting!! Knowledge produced is partial but cumulative

Any questions for clarification?

Thank you for listening and for your questions Suggested readings: Pawson R., Tilley N. Realistic Evaluation. London: Sage, 1999 Pawson R. Evidence-based Policy. A Realist Perspective. London: Sage, 2006. Pawson R. The Science of evaluation: A realist manifesto. London: Sage, 2013 RAMESES on JISCM@il – www.jiscmail.ac.uk/RAMESES The RAMESES Project – www.ramesesproject.org