Evidence-based Feedback George Bergus, LuAnne Stockton, Susan Labuda Schrop, Tracy Kedian

Introduction Who we are Why we are giving this presentation We desire to go beyond the simple rules about effective feedback. We want our learners to practice evidence-based medicine. Shouldn’t we be evidence-based educators?

What we will be doing Define feedback Review effectiveness of feedback How do motivation and feedback relate? Effects of different types of feedback What do medical learners value about feedback? Putting it together

What is feedback? What do you think?

What is feedback? Kluger and DeNisi: Actions taken by (an) external agent(s) to provide information regarding some aspect(s) of one’s task performance. Hattie and Timperley: Information provided by an agent (e.g., teacher, peer, parent, self, experience) regarding aspects of one’s performance or understanding. Feedback is a “consequence” of performance. Shute: Information communicated to the learner that is intended to modify his or her thinking or behavior for the purpose of improving learning.

How effective is feedback? Cohen (1985) Feedback "is one of the more instructionally powerful and least understood features in instructional design." Kluger and DeNisi (1996, 1998) “double edged sword” FB improves performance on average with an effect size of 0.41. BUT, in more than one third of the studies FB results in reduce performance. Kluger and DeNisi idenitified over 2,500 published papers and 500 technical reports. Selected papers had to have a control group, had to one group with a single intervention so as not to confound results, had to be in English. 131 selected from which 607 effect sizes were extracted based on 12,652 subjects and 23,663 observations. Average sample size per effect was 39 subjects.

Effect size = difference in mean outcomes SDpooled What is an effect size? Effect size = difference in mean outcomes SDpooled Effect size Medical Intervention 0.04 ASA and MI risk 0.06 HTN treatment and CVA risk 0.16 CABG and 5-year survival G Norman. The effectiveness and the effects of effect sizes (editorial). Advances in Health Sciences Education 2003;8(3):183-187.

Effect sizes of FB (Kluger and DeNisi, 1996)

How can we make our feedback more effective? We have data from studies to guide us but no comprehensive FB theory. Perhaps we can come up with explanations.

FB focus on correct vs. incorrect Moderator Number of effects Effect size  ’Tis correct 114 0.43  ’Tis incorrect 197 0.25

FB about individual’s change from previous trials Moderator Number of effects Effect size  Yes 50 0.55  No 380 0.28

FB designed to discourage the student Moderator Number of effects Effect size  Yes 49 –0.14  No 388 0.33

FB that threatens self-esteem Moderator Number of effects Effect size  Much threat 102 0.08  Little threat 170 0.47

Praise FB about the task Moderator Number of effects Effect size  Yes 80 0.09  No 358 0.34

FB which includes goal setting Moderator Number of effects Effect size  Difficult goals 37 0.51  Easy, do your best goals 373 0.30

Feedback models Many questions about FB not well answered Timing Amount How complex Delivery Type of task

FB oral vs. written/computer Moderator Number of effects Effect size  Oral 194 0.23  Non-oral 221 0.37 Need to expand to personal vs. written

Number of times FB provided Moderator Number of effects Effect size Lots 97 0.32  Little 171 0.39

FB and complexity of task Moderator Number of effects Effect size Very Complex 107 0.03 Not Complex 114 0.55

FB and type of task Moderator Number of effects Effect size Memory 43 0.69 Follow the rule 100 0.19 Physical 65 -0.11

FB immediate vs. delayed Moderator Number of effects Effect size  Immediate 194 0.28  Delayed 53 0.36 immediate error correction during task acquisition (FT) can result in faster rates of acquisition, whereas immediate error correction during fluency building can detract from the learning of automaticity and the associated strategies of learning (FP). High performing learners might do better with delayed feedback particularly with challenging tasks Low performing learners might do better with immediate feedback

Delayed FB Moderator Effect size Easy Task -0.06 Intermediate Task 0.35 Hard Task 1.17

Summary of what we know Shute VJ. Focus on formative feedback. Review of Educational Research 2008;78(1):153-189.

Hattie and Timperley FB model Effective feedback must answer three major questions. Where am I going? (the goals, Feed Up) How am I going? (assessment, what progress is being made towards the goals, Feed Back) Where to next? (what activities should be undertaken to make better progress, Feed Forward)

Medical students and feedback Student reactions to feedback Students rated praise FB as higher quality than focused FB on specific procedural skills, although the specific skills FB resulted in improved performance while praise FB did not. Suggests: Student satisfaction with FB is not an accurate measure of quality of feedback. Boehler ML, Rogers DA, Schwind CJ, Mayforth R, Quin J, Williams RG, Dunnington G. An investigation of medical student reactions to feedback: A randomized controlled trial. Medical Education 2006;40(8):746-49.

Medical students and feedback What do students want feedback on? Feedback about presentation skills and fund of knowledge was most frequently requested by M3 students on their clerkships. FB on interviewing, communication, or physical exam skills less frequently requested. Suggests: Students want FB in domains which they think have the most impact on their grades. Kogan JR, Shea JA. Implementing feedback cards in core clerkships. Medical Education 2008;42(11):1071-79.

FB effects on other outcomes FB can have benefits other than positive effects on immediate performance. May increase learner satisfaction May contribute to long-range persistence on the task May contribute to transfer of learning

Evidence-based FB How will this discussion affect your delivery of FB?

Feedback summary Specificity of FB Normative FB Discouraging FB FB containing information about correctness, details of how the learner has improved, and new challenging goals will enhance performance. Normative FB Giving feedback and comparing the learner to the group tends to undermine learning. Discouraging FB is discouraging Timing of FB Research on timing is not consistent. Delayed FB may be more effective with difficult tasks and high performing learners, but…