1. Christy K. Boscardin, PhD Pat O’Sullivan, EdD
Formative Assessment
Christy K. Boscardin, PhD
Pat O’Sullivan, EdD
March 30, 2011: Office of Medical Education

2. Creative Commons License
Attribution-NonCommercial-Share Alike 3.0 Unported
You are free:
to Share — to copy, distribute and transmit the work
to Remix — to adapt the work
Under the following conditions:
Attribution. You must give the original authors credit (but not in any way that suggests that they endorse you or your use of the work).
Noncommercial. You may not use this work for commercial purposes.
Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.
See for full license.

3. Timeline 9:00-9:10 Introduction 9:10-10:10 Formative Assessment
10:10-10:50 Small Group Exercise 1
10:50-11:00 Break
11:00-11:20 Share results
11:20-11:40 Small Group Exercise 2
11:40-12:00 Wrap-up and Evaluations

4. Pair-Share:
With a partner Identify the characteristics of formative assessment as you currently use it
Hint: How often to you assess your learners?
Hint: What types of tools do you use?
Hint: How do you use the assessment data?
Collect on a board to revisit
Frequent
Specific information for next steps in improveemtn
Evolves a plan that is checked at a later time by the faculty
Learner must identify/commit to areas needing improvement
Assessment must provide sufficient data to inform next steps

5. The word ‘assess’
Comes from the Latin verb ‘assidere’ meaning to ‘to sit with’. In assessment, one is supposed ‘to sit with’ the learner. This implies it is something we do with and for students and not to students. (Green, 1998)
Reflect on thinking about what you currently do and/or what you think is formative assessment
What frightens them about this definition?
If you can’t be present, what can you do?

6. Summative vs. Formative
The garden analogy: If we think of our learners as plants…
Summative assessment: you may be concerned with how many tomatoes, taste of the tomatoes, size of the tomatoes, etc.
Formative assessment: you may be more concerned about whether the plant is getting enough water, light, and adequate temperature, etc.
Make sure that they make the connection that the formative assessment is not just the daily measures of the summative assessment;
How does this contrast with their definitions of formative assessment

7. Possible misconceptions about formative assessment
Mirror image of the summative assessment
Learner will recognize need for improvement from formative score
Feedback of the assessment data is sufficient to help learner improve; motivator
Instructor’s responsibility met by giving the formative assessment and communicating performance score
“You need to improve” is adequate

8. What is formative assessment?
Provides sufficient information to guide next steps of improvement
Embedded as part of the learning and not as an isolated activity.
Develops self-reflection skills of the learner to form a plan
Requires feedback from instructor and/or learning community involved in helping with next steps
Allows for and guides change for both instructor and learner; follow-up
Again, go back to list and see how this correlates (may have beaten list to death by this point and no need)

9. Additional Differences
Summative:
Given periodically (at a particular point in time) to determine what learners know and can do (e.g. USMLE Step 2)
Takes place at the end of the learning process
Impacts extrinsic learner motivation (graduation, matching, etc)
Formative:
Designed as part of the instructional process. Provide feedback to both learners and instructors. It’s a cyclical process.
Examples: (informal observations, peer assessment, feedback loop with learning plan, open-ended questions, mini-CEX)
Stimulates intrinsic learning motivation (self- motivation)

10. Why Formative Assessment?
80% of residents ~ only one observed clinical encounter & receive no or infrequent feedback (Isaacson et al. 1995)
Faculty failed to detect 68% of error committed by residents & 2/3 scored marginal residents as satisfactory or superior (Noel et al, 1992)
Frequent feedback ~ 12 meta-analysis illustrate ES = .79 Hattie, 1999
Feedback & Clinical performance ~ positive effect for 106 out of 132 studies (77%)

11. Formative Assessment or Not?
The teacher provides students with an open- ended questions related to the concept they are studying and asks the students to identify the information necessary for a response to demonstrate full understanding of the concept. A list of these are recorded on the board. The teacher then provides examples of past student responses. Students are then asked to determine if the responses show full understanding, partial understanding, or no understanding. Students must justify their answers. The list is refined until a set of criteria emerges that students can use to self-assess and peer- assess their progress.
Engages learners in the learning process by creating target/goal and the criteria for reaching it.
Establishing and defining quality work together
Using exemplars of what is expected helps learners understand where they are and where they need to be.
Asking better questions allows an opportunity for deeper thinking and provides instructors with significant insight into the degree and depth of understanding.
Create environment where learners can reflect while engaged in metacognitive thinking are involved in their learning.

12. Formative Assessment or Not?
The band teacher worked through a section of a musical piece with the class that was selected because it contained several challenging sections. As the band played, several students perform some off-notes towards the end of the section. The teacher asked everyone to play that measure and hold each note until directed to play the next one. By doing this, he was able to help students isolate the instruments that were playing the wrong pitch. However, students were still unsure about what exactly needed to be corrected. The teacher asked the trumpets to hold down the fingering for the notes. He then had the students observe each other’s fingerings. It became obvious where incorrect fingerings were applied when student looked at each other’s instrument. The correction was made immediately.
Engages learners in the learning process by creating target/goal and the criteria for reaching it.
Establishing and defining quality work together
Using exemplars of what is expected helps learners understand where they are and where they need to be.
Asking better questions allows an opportunity for deeper thinking and provides instructors with significant insight into the degree and depth of understanding.
Create environment where learners can reflect while engaged in metacognitive thinking are involved in their learning.

13. Light bulb example
Consider a student who initially holds a “source consumer” view of electricity and tries to light a bulb like this… It is relatively easy to convince him that the way to light a bulb is actually this… But, without addressing his underlying model, he is likely to think about series circuits like this

14. Steps in Formative Assessment
Establish common objectives/goals and performance standards.
Develop assessment tools to gather data (information) for feedback on performance
Debrief and analyze the performance data together
Develop a learning plan
Follow-up

15. Examples of Existing Tools that can evolve into Formative Assessments
Mini-CEX
Clinical Encounter Cards (CEC)
Clinical Work Sampling (CWS)
Blinded Patient Encounters (BPE)
Direct Observation of Procedural Skills (DOPS)
Case-based Discussion (CbD)
Multi-Source Feedback (MSF)
Multiple Choice Tests (MCQ)
Learning Issues (LI)
Goal of this slide is that they actually have many things that can be developed as formative assessments or if currently used as formative assessments, built to be stronger formative assessments.
Get CEC
BOSCO
MCQ LI

16. Take one: Mini CEX for History Taking
Does this provide the learner with adequate feedback to make improvement on his/her performance?

17. Example of Revised Tool

18. Small Group Activity 1
Select a tool and describe how as an instructor/faculty would make it part of a formative assessment process
Follow the guide to help you

19. Small Group Activity 2
Each member select a literature guided tip on feedback and decide how it can be applied to the situation just analyzed

20. Key References
Black & Wiliam (2009) Developing the theory of formative assessment. Educ Asses Eval Acc, 21, 5-31
Rudolph et al (2008) Debriefing as formative assessment: Closing performance gaps in medical education. Acad. Emerg Med 15,
Shute (2008) Focus on formative feedback. Review of Educational Research 78,