1. www.cirtl.net Formative Assessment as a Means of Gauging Student Understanding Session begins at 1PM ET/12PM CT/11AM MT/10AM PT. Please configure your audio by running the Audio Set Up Wizard: Tools>Audio>Audio Set Up Wizard. Barbara Masi Director of Education Innovation & Assessment Initiatives; Arts, Sciences & Engineering, University of Rochester Welcome to today’s CIRTLCast What does “formative assessment” mean to you? Respond below by using the textbox tool (the 4 th icon down on the vertical toolbar to the left) Today’s speaker

2. www.cirtl.net Formative Assessment as a Means of Gauging Student Understanding Welcome & introductions Discussion with Barbara Masi Event feedback (1 minute survey) Upcoming CIRTLCasts

3. www.cirtl.net January CIRTLCast Series: Assessing What Students Are Learning January 13: Facilitating Cooperative Work: How to Assess Active Learning Experiences Featuring Rique Campa, Michigan State University January 20: Formative Assessment as a Means of Gauging Student Understanding Featuring Barbara Masi, University of Rochester January 27: Effectively Assessing (and Developing) Your Students’ Problem-Solving Skills Featuring Amy Godert, Cornell University and Jared Danielson, Iowa State University

4. FORMATIVE ASSESSMENT AS A MEANS OF GAUGING STUDENT UNDERSTANDING Barbara Masi, Ph.D. Director of Education Innovation and Assessment Initiatives Arts, Sciences and Engineering University of Rochester January 14, 2016 4

5. Questions for Reflection Situation 1: It’s the beginning of the term and a new course has begun! Teacher hands out syllabus, begins first lecture. Lecture begins by laying out a topic and problem that assumes all are prepared with expected prior knowledge and skill. Students side- eye one another since few do, in fact, recall prior knowledge/skill. – If you or others did not recall appropriate knowledge or skill, did someone bring it up with the teacher? How did the teacher respond? Situation 2: You give, or take, an exam. Class average is 65%. The teacher reviews the test answers. – In your course experiences, do most teachers move on to the next unit? – Have there been courses you have taken where the teacher did something different in response to a 65% or lower class average? 5

6. If only every course could draw on the best aspects of one-on-one tutoring and small group seminar… How do you support learning for each student as class size grows? 6

7. Student characteristics KNOWLEDGE A ABILITIES A MOTIVATIONS A ATTITUDES A YOUR COURSE EXPERIENCE Student AFTER Your Course KNOWLEDGE ΔA ABILITIES ΔA MOTIVATIONS ΔA ATTITUDES ΔA Since students are not perfectly uniform inputs, how can you design your course to work for as many as possible YET BE EFFICIENT IN USE OF YOUR TIME? If only every student could be a perfectly uniform input… 7

8. You’ve aligned course objectives with lectures, assignments and tests or projects. Is this enough to help all students achieve your challenging course objectives, especially higher order Bloom’s objectives? You’ve written your course learning objectives that include higher and lower order Bloom’s taxonomy. 8

9. How to break down learning objectives to help students “build” skills and connect them with course knowledge How to use a mix of formative assessments to improve student learning and move all toward “mastery” Use diagnostics effectively Choose formative assessment measures that gather data from individuals versus groups Today’s goals 9

10. 10 Formative vs. summative assessment methods Formative assessment methods – opportunities to observe how well students are achieving one or more of course learning outcomes. – faculty provide constructive feedback so that a student can improve performance – may or may not be graded Summative assessment methods – formal assessment or judgment of student work that is used in assigning grades 10

11. Learning is like building a house. You start with a strong foundation. The process of developing deep foundational knowledge. Accessing that knowledge to address diverse problems and contexts is built one layer at a time. For each house, a master builder has a plan, knows where she’s going, what to do next. A novice watches and just tries to copy the master. When that fails…. 11

12. 12 Did you ever take lessons on a musical instrument? The teacher listens to the student play a piece. Then she provides feedback and lets the student try again. And so on until the student masters the piece. MASTERY LEARNING….

13. Keller, Bloom and Mastery Learning Model 13 Formative assessment is paramount Large lecture - one-on-one tutoring spectrum INTRODUCE UNIT LEARNING GOALS, CONCEPTS AND EXAMPLES PRACTICE PROBLEMS ASK QUESTIONS OF PEERS/ TAS UNIT TEST 1 PASS NO PASS IDENTIFY LEARNING ISSUES WITH TA PRACTICE TARGET PROBLEMS UNIT TEST 2 PASS MASI, B. et al., “Comparison of Mastery Learning…” ASEE Annual Conference 2015

14. High stakes testing versus mastery testing 14

15. QUESTION Has anyone experienced a mastery learning course? If yes, what did you think of the course as a learning experience? Mastery learning model is super, but cumbersome, so how can we design a course to draw on the spirit of mastery and use it in “traditional” course design? 15

16. Where we’re going…traditional course 16 INTRODUCE TOPIC/ KNOWLEDGE INTRODUCE SAMPLE PROBLEMS THAT ILLUSTRATE NEEDED SKILLS TO USE HAT KNOWLEDGE FORMATIVE: HOMEWORK/ WORKSHOPS FOR PRACTICE SUMMATIVE: EXAM TO ASSESS LEARNING

17. Where we’re going…formative assessment moves students toward mastery 17 TEACHER: INTRODUCE NEW TOPIC/ KNOWLEDGE TEACHER: INTRODUCE SAMPLE PROBLEMS FORMATIVE: REINFORCE WITH IN- CLASS PRACTICE AND DISCUSSION FORMATIVE: QUIZ TO ASSESS LEARNING FORMATIVE: HOMEWORK/ WORKSHOPS FORMATIVE: PREREQ DIAGNOSTIC SUMMATIVE: SHOW MASTERY ON EXAM FORMATIVE: IN-CLASS PRACTICE AND DISCUSSION TEACHER: REVIEW PREREQ KNOWLEDGE/ SKILLS

18. 18 Rationale for more formative in-class Don’t leave all of students’ learning processes to workshops or homework where you can’t observe process. Low-stakes, formative assessment Immediate pulse on student understanding Immediate feedback to students to move them forward in learning

19. But good formative assessment needs more focus…how people learn Prior Knowledge: Students construct new knowledge based on what they already know (or don’t know). Deep Foundational Knowledge: Students need a deep knowledge base and conceptual frameworks. Think like an expert! Metacognition: Students must identify learning goals and monitor their progress toward them. Awareness of process. Bransford, J., Brown, A., & Cocking, R. (Eds.). (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press. 19

20. 20 Formative in-class assessment methods for measuring student learning Indirect Course evaluations (pre-course, midpoint, end of term) Direct Pre-requisite/ diagnostic tests Standardized concept tests (eg. Force Concept Inventory) Mini-quiz (for credit or no) Oral exam of problem solving Documented solutions Clicker questions Observations of student performance Angelo and Cross, Classroom Assessment Techniques (CATs): eg. minute paper, think/pair/share, muddiest point

21. 21 Diagnostic assignment/ test Good foundation: Prior knowledge Don’t guess readiness A diagnostic can test: background recall of concepts, math retention, writing, scientific method, programming, experimental data analysis. Use the diagnostic process to convey the course tone as formative! Introduce learning journey! Bring diagnostic results back to classroom for refresher lectures and in-class practice.

22. 22 Building a foundation… Fuzzy biology problem: You observe that robins seem to be dying in your yard at a higher rate than those in your friend’s yard a few blocks away. Design an experiment to determine why. Design an experiment, gather and analyze data | | Experimental design Statistics Requires integration of 2 essential skills, but no knowledge of biology Practice breaking down your course learning objectives into blocks! Good diagnostic breaks down each bit of desired knowledge and skills into bite size blocks.

23. Introduction to Biology Example Challenging course goals required strong incoming frosh foundation Scientific reasoning: Make inferences or draw conclusions about biological cell mechanism based on quantitative data gathered from biological experimental data Faculty concerned with incoming frosh background. Diagnostic test: – Algebra, logs, reading log graphs – Bio definitions of cell parts – Conceptual understanding of cell functions and feedback 23

24. Introduction and follow up were key… Bio faculty Introduced idea of diagnostic for gauging knowledge and skills Low pressure, promised to engage students in results and refreshers if needed 20 minute in-class quiz Graded in 1 week and returned Reviewed results in next class Reinforced areas of performance concern in next homework and workshops Identified students whose high school bio background was limited, created mentoring groups 24

25. 25 Diagnostics: Conceptual misunderstandings Standardized concept inventory tests can elucidate key conceptual misunderstandings. Easy to interpret and use results. Examples: Physics – http://modeling.asu.edu/R%26E/InterFCI.pdf http://modeling.asu.edu/R%26E/InterFCI.pdf Stats – https://engineering.purdue.edu/Archive/SCI https://engineering.purdue.edu/Archive/SCI Chemistry – https://www.chemedx.org/JCEDlib/QBank/collection/CQandChP/CQs/Con ceptsInventory/CCIIntro.html https://www.chemedx.org/JCEDlib/QBank/collection/CQandChP/CQs/Con ceptsInventory/CCIIntro.html Biology – http://www.sci.sdsu.edu/CRMSE/files/Concept_Inventories_in_Biology_2 0110325.pdf http://www.sci.sdsu.edu/CRMSE/files/Concept_Inventories_in_Biology_2 0110325.pdf 25

26. Introduction to Physics Example Faculty concerned with incoming frosh background – Algebra, vectors, trigonometry – Basic Newtonian physics conceptual understanding/ misconceptions Introduced idea of diagnostic for gauging knowledge and skills Diagnostics – Provided few days math refresher time, then 20 minute math quiz – ½ hour in-class Force Concept Inventory (FCI) to test high school level mechanics concept understanding Graded in a few days and returned Identified students’ key misconceptions, addressed in lectures and workshop problems Students participated in mandatory tutoring if math quiz score below 50% 26

27. REFLECTION Think about your course experiences with diagnostic tests. How did faculty introduce them? How were were results used? Did the diagnostic test and/or review process bring you and your peers up to speed? Did students drop the course after the diagnostic? Can you think of ways to use diagnostics in a course you dream of designing? 27

28. 28 Right tool for the job: Classroom Assessment Techniques (CAT) are not all the same… Learning outcomeTechnique Recall and understanding – quick check Background knowledge probe Think/pair/share Minute paper/ Muddiest point Concept map Problem solving- probingDocumented problem solutions What’s the principle Videotape oral protocol Application- probingApplication cards Analysis/ critical thinking - deep Analytic memo or class discussion (connect to case study) Synthesis / creative thinking- deep Project with annotated portfolio of conceptual framework, problem solving and synthesis process

29. In-class formative assessments: mix individual and group data gathering Observe each student’s response – Takes more time to review but tells you about how each student is doing – Documented solutions (explain problem solving process) – Oral problem solving, videotape (switch up response mode) – Mini-quizzes (test setting practice, low pressure, rapid feedback) Observe general trend/ statistics of all student responses – Takes less time, can run in class in real time, but lose individual picture – Clickers (use with CAT methods) – Think-Pair-Share – Concept maps in class (making connections among concepts, feedback) 29

30. 30 CAT Documented solution – Useful for assessing problem solving in quantitative course. – Develop ability to apply principles and generalizations to new problems. – Prompts students to keep track of steps they take to solve problem. – Ask students to apply CAT method to 3 homework problems. – Helps you to see who uses appropriate, concise process that those who are vague or misunderstand steps. PREP TIME LOW, REVIEW TIME MEDIUM/ HIGH 30

31. 31 [P] To minimize the work you do in carrying a bag of groceries from first to second floor of a building, which would you do and why? What concepts and calculations did you use to choose your solution? 1.Carry bag up stairs 2.Carry bag up elevator 3.Put bag on floor of elevator, ride up, then pick up bag again 4.Carry bag up a ramp 5.Put bag in cart and push it up a ramp Documented solution example

32. CAT Concept maps: – Drawings or diagrams showing mental connections among concepts. – Each linkage should be labeled with how concepts are linked. – Helps students organize and synthesize lecture notes – Give clear directions of primary concept to be mapped – Compare student maps with your map – Use to ask “what if” questions where students must make reasonable inferences of what would happen if a change is made – http://cmap.ihmc.us/publications/research-publications.php PREP TIME LOW/ MEDIUM, REVIEW TIME MEDIUM/HIGH IF GRADED 32

33. 33 Alignment of learning objectives and assessments Essential step: pull it all together Examine the balance of type and number of assessments for each course learning objective If there are too many assignments that cover one important learning and too few on another important one, shift the balance. Make sure there are sufficient assessments to INTRODUCE, REINFORCE, MASTER introducereinforcemaster Learning objective 1 Homework 1.q.2 and 3 In-class clicker questions Workshop 1, q.1 In class minute paper Mid-term survey Exam 1, q.4 Final exam, q.2 Final survey 33

34. Where we’re going: COURSE PLAN MATRIX 34 Intro - duce Reinforcemaster LO 1: UNDERSTAN D, CONCEPTU AL AND PROCEDUR E LEC. 1, 2 FORMATIVE: Workshop group work 1, q.1, 2 FORMATIVE: In class individual clicker questions FORMATIVE: HW 1, q. 1,2,3,4,5 Exam 1, q.1,2 Final exam, q.2 LO 2: APPLY, PROCEDUR E LEC. 3,4 FORMATIVE: In class Individual minute paper FORMATIVE: Workshop group work 2, q. 1,2 FORMATIVE: Individual, HW 2, q. 1,2,4 Exam 1. q. 3,4 LO 2: EVALUATE, PROCEDUR E LEC. 1, 3, 4 FORMATIVE: Workshop group work 2: q. 1,2,3 FORMATIVE: In class individual minute paper HW: 2, q. 6, 7 Exam 2, q. 1 Less time to cover lots of topics, but greater development of mastery! Less class time talking, more class time students doing and you observing

35. QUESTIONS? 35