2. Formative Assessment: A Method to Close the Feedback Loop Eugenia Etkina, Graduate School of Education (GSE) Rutgers University Florida International University April 8, 2005 Supported in part by NSF Grant DUE #0241078

3. Rutgers PER Group David Brookes (Physics, Student) Suzanne Brahmia (Physics, Staff) Eugenia Etkina (GSE, Professor) Mike Lawrence (High School Physics Teacher) Sahana Murthy (GSE, Post Doc) David Rosengrant (GSE, Student) Maria Ruibal (GSE, Student) Alan Van Heuvelen (Physics, Professor) Aaron Warren (Physics, Student) http://paer.rutgers.edu/scientificabilities/ http://paer.rutgers.edu/PT3

4. Outline Assessment Formative vs summative Three steps of formative assessment Self-assessment Scientific abilities and assessment rubrics Examples of assessment tasks and student performance

5. We all do assessment Summative Formative

6. Formative Assessment Enhances learning during learning. Gains are the largest reported for an educational intervention (Black and Wiliam, 1998).

7. Components of formative assessment Teacher giving feedback to the students The teacher and students taking action to improve learning during learning Self - assessment

8. Three essential steps Where are you now? Where do you need to go? How can you get there?

9. Role of feedback Feedback Grade Written comments Grade and comments EFFECTEFFECT Negative Positive Zero

10. What are our goals? Conceptual understanding Problem solving Scientific abilities

11. Formative assessment project What are some scientific abilities? Activities in which students develop abilities Sub-abilities Assessment of abilities through rubrics Rubrics as guidelines and for self-assessment Implementing activities with rubrics (3 steps of FA) Using rubrics to score Do students improve?

12. What are some scientific abilities? Ability to represent a process in multiple ways Ability to design an experimental investigation (an observational experiment; a testing experiment; an investigation to solve a problem) Ability to collect and analyze experimental data Ability to construct and modify explanations Ability to evaluate all of the above All of them are content dependent!!!

13. Activities Multiple representation tasks (D. Rosengrant, A. Van Heuvelen, E. Etkina) Experimental design tasks (S. Murthy, M. Ruibal, E. Etkina) Anomalous data tasks (D. Brookes, M. Lawrence, E. Etkina) Video problems tasks (D. Brookes, E. Etkina) Evaluation tasks (A. Warren, A. Van Heuvelen) http://paer.rutgers.edu/scientificabilities http://paer.rutgers.edu/pt3

14. Activity: design an experiment to solve a problem Experiment 1 Design an experiment to determine the coefficient of static friction between your shoe and the carpet. Experiment 2 Design an experiment to determine the specific heat of the given unknown object.

15. Sub-abilities Ability to design an experiment to solve a problem Some sub-abilities To suggest multiple experiments to accomplish desired goals. To design a reliable experiment that solves the problem. To identify assumptions made in the procedure. To determine specifically the way assumptions might affect the outcome. To identify sources of experimental uncertainties. To evaluate specifically how experimental uncertainties may affect data. To evaluate results by an independent method.

16. Experiments Design two independent experiments to determine the coefficient of static friction between your shoe and the carpet. Include in your report for each experiment: a)an outline of the experimental procedure and a labeled sketch; b)a mathematical procedure to solve the problem; c)assumptions about the objects, interactions, and processes used in the math procedure and how these assumptions could affect the outcome; d)sources of experimental uncertainty, how these could affect the result and how you could minimize them. e)Perform the experiment and record your observations in an appropriate format. What is the outcome of your experiment? f)Compare the two values you obtained. Explain the difference using assumptions and uncertainties. g)Describe a real life situation in which you need to figure out things similar to this experiment. Design two independent experiments to determine the specific heat of the given unknown object.

17. Scientific abilities rubrics Contain descriptors of different levels of development of scientific abilities. Scale of 0-3. Very high inter-rate reliability. Can be used as guidelines for students, grading lab reports, self- assessment of lab reports by students, research on developing and assessing scientific abilities. Lab tasks contain guidelines which match rubrics. SCORE ABILITY 0123 To design a reliable experiment that solves the problem The experiment does not solve the problem. The experiment attempts to solve the problem but due to the nature of the design, the data will not lead to an accurate solution. The experiment attempts to solve the problem but due to the nature of the design, there is a moderate chance that the data will not lead to an accurate solution. The experiment solves the problem and has a high likelihood of producing data that will lead to an accurate solution

18. Developing rubrics Identifying important sub-abilities Writing descriptors (scale 0-3) Finding clear wording Scoring student work Discussing the items with a disagreement Revising wording Name (9 rows) Able to Suzanne31132 Sahana31322

19. Using rubrics SCORE ABILITY 0123 To identify assumptions made in the chosen procedure No attempt is made to identify any assumptions. An attempt is made to identify assumptions, but most are missing, described vaguely, or incorrect. Most assumptions are correctly identified All assumptions are correctly identified. Design two independent experiments to determine the specific heat of the given unknown object. Student writing samples … the block and water will reach equilibrium after 10 minutes … no heat goes in and out of the calorimeter… minimal heat is lost to the environment during transfer of the block … the temperature inside the beaker is homogenous SCORE: 3 … No heat exchange between system and surroundings. SCORE: 1

20. Using rubrics Design two independent experiments to determine the specific heat of the given unknown object. SCORE ABILITY 0123 To determine specifically the way in which assumptions might affect the results No attempt is made to determine the effects of assumptions. An attempt is made to determine the effects of some assumptions, but most are missing, incorrect or described vaguely Effects of most assumptions are determined correctly, though a few contain minor errors, inconsistencies, or omissions. The effects of all assumptions are correctly determined … if the heat is lost from the block during transfer, the specific heat obtained in the experiment would be smaller than what it should be. SCORE: 2 … the assumptions will affect the results, the actual value may be different from the experimental one. SCORE: 1

21. Implementation Two large enrollment introductory algebra-based classes - - 500 and 190 students, 30 TAs Interactive engagement - ISLE curriculum Labs are integrated [L-r-l] Lab sections of 25 students who work in groups of 3 or 4 TA Training, one hour per week: TAs perform and analyze design tasks Students perform and write a report in a 3-hour lab

22. Pilot Study (spring 2004) Students do not use rubrics for self assessment 35 students randomly selected from 4 sections Scored student responses from week 3 (initial) and week 10 (final) Found significant improvement

23. Student writing samples Week 3Week 10

24. New Study (Fall 2004) A typical lab session Students download lab tasks and rubrics from course webpage Student report: brainstorming Student report: self- assessment scores Student report: outcomes, effect of assumptions, uncertainties Student report: data and analysis Groups perform experiment; collect, record and analyze data TA Students perform self-assessment of their experiment and analysis TA Groups brainstorm solutions for task, decide best design TA

25. Findings N=102

26. Findings

27. Findings

28. Findings N=47

29. What did we learn? It is possible to implement and assess open-ended design tasks in large enrollment class. Assessment rubrics serve as goals for writing new design tasks. Preparation of TA matters. Evaluating assumptions and uncertainties is very difficult for the students. Abilities are content-dependent. Students self-assessment is closer to ours on the abilities that they developed. Students self-assessment is farther from ours on the abilities that they did not develop.

30. Ability to represent phenomena in multiple ways Free-body diagrams: where do you need to go? An elevator is slowing down An elevator is slowing down on on its way up:its way down: y a y a Earth, cable F Earth on elevator F cable on elevator y F Earth on elevator y

31. Rubric for self assessement Scientific Ability 0123 Ability to construct free-body diagrams No FBD is drawn FBD is constructed but contains major errors such as an incorrect relative length, wrong direction, extra incorrect force vectors, a missing force vector, or a force exerted on a different object. FBD contains no errors in forces but lacks a key feature such as labels or forces are mislabeled or axes are missing or mis labeled. The diagram contains no errors and each force is labeled, axes are present and labeled. Free-body diagrams: How to get there?

32. Formative assessment task in lecture Draw a free-body diagram for the ball in the air.. Where are you now? 2 4 1 3 F ball on person F person on ball F Earth on ball F person on ball F Earth on ball F person on ball

33. Do students actually use FBDs?

34. Ability to test an explanation Ability to make a prediction based on a relationship/explanation Ability to revise the explanation based on the results of the experiment Where do you need to go?

35. How to get there? Scientific Ability 0 123 Is able to make a reasonable prediction based on a relationship or explanation. No attempt to make a prediction is made. A prediction is made but it doesnt follow from the relationship or explanation being tested, or it ignores or contradicts some of the assumptions inherent in the relationship or explanation. A prediction is made that follows from the relationship or explanation and incorporates the assumptions, but it contains minor errors, inconsistencies or omissions. A correct prediction is made that follows from the relationship or explanation and incorporates the assumptions.

36. Ability to revise the explanation or assumptions based on the results of the experiment Make a prediction about an outcome of a particular experiment Explain why you made the prediction. Watch the experiment, record the outcome, compare to the prediction. Revise the explanation or the assumptions if necessary.

37. How to get there Is able to revise the explanation of a prediction, based on the results of an experiment. No attempt is made to explain the outcome of the experiment, revise explanation or assumptions. An attempt is made to explain the outcome and revise the previous explanation or assumptions but is (a) mostly incomplete and/or (b) based on incorrect reasoning. The revision of the previous explanation or assumptions is partially complete and correct, yet still lacking in some relevant details. The revision of the explanation or assumptions is explained completely and correctly.

38. Ability to design an experiment to solve a problem 1.Video problems. 2.Design experiments with scaffolding. 3.Design experiments with no scaffolding (practicals)

39. The END Thank you!