Christy K. Boscardin, PhD Pat O’Sullivan, EdD Formative Assessment Christy K. Boscardin, PhD Pat O’Sullivan, EdD March 30, 2011: Office of Medical Education

Creative Commons License Attribution-NonCommercial-Share Alike 3.0 Unported You are free: to Share — to copy, distribute and transmit the work to Remix — to adapt the work Under the following conditions: Attribution. You must give the original authors credit (but not in any way that suggests that they endorse you or your use of the work). Noncommercial. You may not use this work for commercial purposes. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. See http://creativecommons.org/licenses/by-nc-sa/3.0/ for full license.

Timeline 9:00-9:10 Introduction 9:10-10:10 Formative Assessment 10:10-10:50 Small Group Exercise 1 10:50-11:00 Break 11:00-11:20 Share results 11:20-11:40 Small Group Exercise 2 11:40-12:00 Wrap-up and Evaluations

Pair-Share: With a partner Identify the characteristics of formative assessment as you currently use it Hint: How often to you assess your learners? Hint: What types of tools do you use? Hint: How do you use the assessment data? Collect on a board to revisit Frequent Specific information for next steps in improveemtn Evolves a plan that is checked at a later time by the faculty Learner must identify/commit to areas needing improvement Assessment must provide sufficient data to inform next steps

The word ‘assess’ Comes from the Latin verb ‘assidere’ meaning to ‘to sit with’. In assessment, one is supposed ‘to sit with’ the learner. This implies it is something we do with and for students and not to students. (Green, 1998) Reflect on thinking about what you currently do and/or what you think is formative assessment What frightens them about this definition? If you can’t be present, what can you do?

Summative vs. Formative The garden analogy: If we think of our learners as plants… Summative assessment: you may be concerned with how many tomatoes, taste of the tomatoes, size of the tomatoes, etc. Formative assessment: you may be more concerned about whether the plant is getting enough water, light, and adequate temperature, etc. Make sure that they make the connection that the formative assessment is not just the daily measures of the summative assessment; How does this contrast with their definitions of formative assessment

Possible misconceptions about formative assessment Mirror image of the summative assessment Learner will recognize need for improvement from formative score Feedback of the assessment data is sufficient to help learner improve; motivator Instructor’s responsibility met by giving the formative assessment and communicating performance score “You need to improve” is adequate

What is formative assessment? Provides sufficient information to guide next steps of improvement Embedded as part of the learning and not as an isolated activity. Develops self-reflection skills of the learner to form a plan Requires feedback from instructor and/or learning community involved in helping with next steps Allows for and guides change for both instructor and learner; follow-up Again, go back to list and see how this correlates (may have beaten list to death by this point and no need)

Additional Differences Summative: Given periodically (at a particular point in time) to determine what learners know and can do (e.g. USMLE Step 2) Takes place at the end of the learning process Impacts extrinsic learner motivation (graduation, matching, etc) Formative: Designed as part of the instructional process. Provide feedback to both learners and instructors. It’s a cyclical process. Examples: (informal observations, peer assessment, feedback loop with learning plan, open-ended questions, mini-CEX) Stimulates intrinsic learning motivation (self- motivation)

Why Formative Assessment? 80% of residents ~ only one observed clinical encounter & receive no or infrequent feedback (Isaacson et al. 1995) Faculty failed to detect 68% of error committed by residents & 2/3 scored marginal residents as satisfactory or superior (Noel et al, 1992) Frequent feedback ~ 12 meta-analysis illustrate ES = .79 Hattie, 1999 Feedback & Clinical performance ~ positive effect for 106 out of 132 studies (77%)

Formative Assessment or Not? The teacher provides students with an open- ended questions related to the concept they are studying and asks the students to identify the information necessary for a response to demonstrate full understanding of the concept. A list of these are recorded on the board. The teacher then provides examples of past student responses. Students are then asked to determine if the responses show full understanding, partial understanding, or no understanding. Students must justify their answers. The list is refined until a set of criteria emerges that students can use to self-assess and peer- assess their progress. Engages learners in the learning process by creating target/goal and the criteria for reaching it. Establishing and defining quality work together Using exemplars of what is expected helps learners understand where they are and where they need to be. Asking better questions allows an opportunity for deeper thinking and provides instructors with significant insight into the degree and depth of understanding. Create environment where learners can reflect while engaged in metacognitive thinking are involved in their learning.

Formative Assessment or Not? The band teacher worked through a section of a musical piece with the class that was selected because it contained several challenging sections. As the band played, several students perform some off-notes towards the end of the section. The teacher asked everyone to play that measure and hold each note until directed to play the next one. By doing this, he was able to help students isolate the instruments that were playing the wrong pitch. However, students were still unsure about what exactly needed to be corrected. The teacher asked the trumpets to hold down the fingering for the notes. He then had the students observe each other’s fingerings. It became obvious where incorrect fingerings were applied when student looked at each other’s instrument. The correction was made immediately. Engages learners in the learning process by creating target/goal and the criteria for reaching it. Establishing and defining quality work together Using exemplars of what is expected helps learners understand where they are and where they need to be. Asking better questions allows an opportunity for deeper thinking and provides instructors with significant insight into the degree and depth of understanding. Create environment where learners can reflect while engaged in metacognitive thinking are involved in their learning.

Light bulb example Consider a student who initially holds a “source consumer” view of electricity and tries to light a bulb like this… It is relatively easy to convince him that the way to light a bulb is actually this… But, without addressing his underlying model, he is likely to think about series circuits like this

Steps in Formative Assessment Establish common objectives/goals and performance standards. Develop assessment tools to gather data (information) for feedback on performance Debrief and analyze the performance data together Develop a learning plan Follow-up

Examples of Existing Tools that can evolve into Formative Assessments Mini-CEX Clinical Encounter Cards (CEC) Clinical Work Sampling (CWS) Blinded Patient Encounters (BPE) Direct Observation of Procedural Skills (DOPS) Case-based Discussion (CbD) Multi-Source Feedback (MSF) Multiple Choice Tests (MCQ) Learning Issues (LI) Goal of this slide is that they actually have many things that can be developed as formative assessments or if currently used as formative assessments, built to be stronger formative assessments. Get CEC BOSCO MCQ LI

Take one: Mini CEX for History Taking Does this provide the learner with adequate feedback to make improvement on his/her performance?

Example of Revised Tool

Small Group Activity 1 Select a tool and describe how as an instructor/faculty would make it part of a formative assessment process Follow the guide to help you

Small Group Activity 2 Each member select a literature guided tip on feedback and decide how it can be applied to the situation just analyzed

Key References Black & Wiliam (2009) Developing the theory of formative assessment. Educ Asses Eval Acc, 21, 5-31 Rudolph et al (2008) Debriefing as formative assessment: Closing performance gaps in medical education. Acad. Emerg Med 15, 1010-1016. Shute (2008) Focus on formative feedback. Review of Educational Research 78, 153-189.