1. Evidence-based Feedback
George Bergus, LuAnne Stockton, Susan Labuda Schrop, Tracy Kedian

2. Introduction Who we are Why we are giving this presentation
We desire to go beyond the simple rules about effective feedback.
We want our learners to practice evidence-based medicine. Shouldn’t we be evidence-based educators?

3. What we will be doing Define feedback Review effectiveness of feedback
How do motivation and feedback relate?
Effects of different types of feedback
What do medical learners value about feedback?
Putting it together

4. What is feedback?
What do you think?

5. What is feedback?
Kluger and DeNisi: Actions taken by (an) external agent(s) to provide information regarding some aspect(s) of one’s task performance.
Hattie and Timperley: Information provided by an agent (e.g., teacher, peer, parent, self, experience) regarding aspects of one’s performance or understanding. Feedback is a “consequence” of performance.
Shute: Information communicated to the learner that is intended to modify his or her thinking or behavior for the purpose of improving learning.

6. How effective is feedback?
Cohen (1985)
Feedback "is one of the more instructionally powerful and least understood features in instructional design."
Kluger and DeNisi (1996, 1998)
“double edged sword”
FB improves performance on average with an effect size of 0.41.
BUT, in more than one third of the studies FB results in reduce performance.
Kluger and DeNisi idenitified over 2,500 published papers and 500 technical reports. Selected papers had to have a control group, had to one group with a single intervention so as not to confound results, had to be in English.
131 selected from which 607 effect sizes were extracted based on 12,652 subjects and 23,663 observations. Average sample size per effect was 39 subjects.

7. Effect size = difference in mean outcomes SDpooled
What is an effect size?
Effect size = difference in mean outcomes SDpooled
Effect size Medical Intervention ASA and MI risk HTN treatment and CVA risk CABG and 5-year survival
G Norman. The effectiveness and the effects of effect sizes (editorial). Advances in Health Sciences Education 2003;8(3):

8. Effect sizes of FB (Kluger and DeNisi, 1996)

9. How can we make our feedback more effective?
We have data from studies to guide us but no comprehensive FB theory.
Perhaps we can come up with explanations.

10. FB focus on correct vs. incorrect
Moderator
Number of effects
Effect size
 ’Tis correct
114
0.43
 ’Tis incorrect
197
0.25

11. FB about individual’s change from previous trials
Moderator
Number of effects
Effect size
 Yes 50
0.55
 No
380
0.28

12. FB designed to discourage the student
Moderator
Number of effects
Effect size
 Yes 49
–0.14
 No
388
0.33

13. FB that threatens self-esteem
Moderator
Number of effects
Effect size
 Much threat
102
0.08
 Little threat
170
0.47

14. Praise FB about the task
Moderator
Number of effects
Effect size
 Yes 80
0.09
 No
358
0.34

15. FB which includes goal setting
Moderator
Number of effects
Effect size
 Difficult goals 37
0.51
 Easy, do your best goals
373
0.30

16. Feedback models Many questions about FB not well answered Timing
Amount
How complex
Delivery
Type of task

17. FB oral vs. written/computer
Moderator
Number of effects
Effect size
 Oral
194
0.23
 Non-oral
221
0.37
Need to expand to personal vs. written

18. Number of times FB provided
Moderator
Number of effects
Effect size
Lots 97
0.32
 Little
171
0.39

19. FB and complexity of task
Moderator
Number of effects
Effect size
Very Complex
107
0.03
Not Complex
114
0.55

20. FB and type of task Moderator Number of effects Effect size Memory 43
0.69
Follow the rule
100
0.19
Physical 65
-0.11

21. FB immediate vs. delayed
Moderator
Number of effects
Effect size
 Immediate
194
0.28
 Delayed 53
0.36
immediate error correction during task acquisition (FT) can result in faster rates of acquisition,
whereas immediate error correction during fluency building can detract from the
learning of automaticity and the associated strategies of learning (FP).
High performing learners might do better with delayed feedback particularly with challenging tasks
Low performing learners might do better with immediate feedback

22. Delayed FB Moderator Effect size Easy Task -0.06 Intermediate Task
0.35
Hard Task
1.17

23. Summary of what we know
Shute VJ. Focus on formative feedback. Review of Educational Research 2008;78(1):

24. Hattie and Timperley FB model
Effective feedback must answer three major questions.
Where am I going? (the goals, Feed Up)
How am I going? (assessment, what progress is being made towards the goals, Feed Back)
Where to next? (what activities should be undertaken to make better progress, Feed Forward)

25. Medical students and feedback
Student reactions to feedback
Students rated praise FB as higher quality than focused FB on specific procedural skills, although the specific skills FB resulted in improved performance while praise FB did not.
Suggests: Student satisfaction with FB is not an accurate measure of quality of feedback.
Boehler ML, Rogers DA, Schwind CJ, Mayforth R, Quin J, Williams RG, Dunnington G. An investigation of medical student
reactions to feedback: A randomized controlled trial. Medical Education 2006;40(8):

26. Medical students and feedback
What do students want feedback on?
Feedback about presentation skills and fund of knowledge was most frequently requested by M3 students on their clerkships.
FB on interviewing, communication, or physical exam skills less frequently requested.
Suggests: Students want FB in domains which they think have the most impact on their grades.
Kogan JR, Shea JA. Implementing feedback cards in core clerkships. Medical Education 2008;42(11):

27. FB effects on other outcomes
FB can have benefits other than positive effects on immediate performance.
May increase learner satisfaction
May contribute to long-range persistence on the task
May contribute to transfer of learning

28. Evidence-based FB
How will this discussion affect your delivery of FB?

29. Feedback summary Specificity of FB Normative FB Discouraging FB
FB containing information about correctness, details of how the learner has improved, and new challenging goals will enhance performance.
Normative FB
Giving feedback and comparing the learner to the group tends to undermine learning.
Discouraging FB
is discouraging
Timing of FB
Research on timing is not consistent.
Delayed FB may be more effective with difficult tasks and high performing learners, but…