Principles of feedback delivery and the role of self- assessment Kevin W. Eva Department of Clinical Epidemiology and Biostatistics Program for Educational Research and Development McMaster University

Principles of feedback delivery 1.There are no universal principles

“There is no “best” type of formative feedback for all learners and all learning outcomes” Shute (2008) “There is no “best” type of formative feedback for all learners and all learning outcomes” Shute (2008)

Principles of feedback delivery 1.There are no universal principles 2.Self-assessment is insufficient

The Rhetoric Almost every article on self-assessment begins with the same basic sentence:Almost every article on self-assessment begins with the same basic sentence: “The ability to self-assess is a cornerstone of professional self-regulation”

The self-regulating professional Reflect regularly on daily practice Self-assess gaps in knowledge or skill Seek opportunities to redress gaps Invest energy to learn (or relearn) Incorporate knowledge into practice Repeat (Handfield-Jones, et al., 2002)

Eva, et al. (2004)

The self-assessment literature Hundreds of articlesHundreds of articles Many literature reviewsMany literature reviews One conclusionOne conclusion Self-assessment ability is generally poor

Boehler, Rogers, et al. (2006)

“In a majority of relevant studies physicians do not appear to accurately self-assess” Davis, et al. (2006) “In a majority of relevant studies physicians do not appear to accurately self-assess” Davis, et al. (2006)

The self-assessment literature Either the conclusions of hundreds of studies using multiple methodologies are all wrong or a critical premise underlying the concept of self-regulation in the professions is unsupportableEither the conclusions of hundreds of studies using multiple methodologies are all wrong or a critical premise underlying the concept of self-regulation in the professions is unsupportable

If you don’t think you’re susceptible, you’re not alone … Pronin, Lin, Ross (2002) …

“Evolution deserves the Microsoft Windows Award for installing these mental processes in every one of us without asking permission.” (Gilbert, 2006)

Principles of feedback delivery 1.There are no universal principles 2.Self-assessment is insufficient 3.Self-assessment is vital

“It is impossible for a man to learn what he thinks he already knows” Epictetus (circa 100AD) “It is impossible for a man to learn what he thinks he already knows” Epictetus (circa 100AD)

Predicting generation of learning goals (Overall R 2 =.45; Eva, et al., In press)

“Physician agreement with colleagues’ scores correlated positively with overall mean scores from colleagues (R=.48)” Sargeant et al. (2003) “Physician agreement with colleagues’ scores correlated positively with overall mean scores from colleagues (R=.48)” Sargeant et al. (2003)

The ironic nature of self-assessment We can’t trust it … but we should encourage itWe can’t trust it … but we should encourage it We need to influence it … without changing itWe need to influence it … without changing it

Self Efficacy Influenced by feedback, but also influences future performanceInfluenced by feedback, but also influences future performance Not merely a passive reflection of performance, but part of a self-fulfilling prophecy that affects performanceNot merely a passive reflection of performance, but part of a self-fulfilling prophecy that affects performance (see Dweck, 1999)

Implications for self-regulation “Rose colored glasses” approach to reflection understandable and necessary“Rose colored glasses” approach to reflection understandable and necessary Important for ability to function and succeedImportant for ability to function and succeed Can never create a “good self-assessor”Can never create a “good self-assessor” And probably shouldn’t tryAnd probably shouldn’t try

Principles of feedback delivery 1.There are no universal principles 2.Self-assessment is insufficient 3.Self-assessment is vital 4.The goal is to enable discovery

% of physicians for whom it is difficult/impossible to generate practice data about… (Audet, et al., 2005)

% of physicians who have access to any quality of care data (Audet, et al., 2005)

The need for difficulties Learners tend to request feedback after successful experiencesLearners tend to request feedback after successful experiences (Chiviacowsky and Wulf, 2002) Learning strategies perceived as beneficial are often sub-optimalLearning strategies perceived as beneficial are often sub-optimal (Simon and Bjork, 2001) Proficiency tends to be inferred from the rate of learning rather than performanceProficiency tends to be inferred from the rate of learning rather than performance (Eva, 2009)

Jowett, et al. (2007) Learner Material

Test-enhanced learning See Larsen, Butler, and Roediger (2008)

“The phrase self-assessment should not imply an isolated or individualistic activity; it should commonly involve peers, teachers, and other sources of information” (Boud, 1999 )

“Education is learning what you didn’t even know you didn’t know” (Boorstin, 1914) “Education is learning what you didn’t even know you didn’t know” (Boorstin, 1914)

Principles of feedback delivery 1.There are no universal principles 2.Self-assessment is insufficient 3.Self-assessment is vital 4.The goal is to enable discovery 5.Numbers can be harmful

“Feedback interventions that draw attention to the self via normative cues are largely ineffective” (Kluger and DeNisi, 1998) “Feedback interventions that draw attention to the self via normative cues are largely ineffective” (Kluger and DeNisi, 1998)

“Studies that augmented feedback with consultation produced substantially larger differences, but other methodological variations had little effect” (Marsh and Roche, 1997) “Studies that augmented feedback with consultation produced substantially larger differences, but other methodological variations had little effect” (Marsh and Roche, 1997)

Principles of feedback delivery 1.There are no universal principles 2.Self-assessment is insufficient 3.Self-assessment is vital 4.The goal is to enable discovery 5.Numbers can be harmful 6.Credibility is likely to be the key

Credibility “With your clinical partner … they know your strengths and weaknesses and they’re probably going to give you quite useful feedback. Maybe a consultant watched you one time. They might not have the whole picture of what you’re capable of and what you normally do” (Sargeant, et al., 2010)

Credibility Of relationships: Engagement and awareness Of the process: Validity and authenticity Of intent: Beneficence and non-maleficence (Sargeant, et al., 2010)

Thanks

Kruger and Dunning (1999)

Self-assessment broadly conceived of as a prequel to self-directed learning activities aimed at performance improvementsSelf-assessment broadly conceived of as a prequel to self-directed learning activities aimed at performance improvements With respect to patient safety, we’re much more concerned about situation specific awareness (i.e., “reflection in practice”)With respect to patient safety, we’re much more concerned about situation specific awareness (i.e., “reflection in practice”) Continuous monitoring of whether or not one has sufficient skills to effectively solve the current problemContinuous monitoring of whether or not one has sufficient skills to effectively solve the current problem The self-assessment literature (as summarized by Eva and Regehr, 2008)

Self-monitoring Knowing when to defer Participants clearly knew when they were likely to respond in errorParticipants clearly knew when they were likely to respond in error (Eva and Regehr, 2007)

Self-monitoring

Self-monitoring Self-assessment (as overall judgment of ability) is importantly different from self-awareness (monitoring)Self-assessment (as overall judgment of ability) is importantly different from self-awareness (monitoring) Difficulty with self-assessment is not that we can’t judge our performance, but that we don’t aggregate well over eventsDifficulty with self-assessment is not that we can’t judge our performance, but that we don’t aggregate well over events