1. Michigan State University Gathering and Timely use of Feedback from Individualized On-Line Work. Matthew HallJoyce ParkerBehrouz Minaei-BigdoliGuy AlbertelliGerd KortemeyerEdwin KashyMatthew HallJoyce ParkerBehrouz Minaei-BigdoliGuy AlbertelliGerd KortemeyerEdwin Kashy Matthew Hall Joyce Parker Behrouz Minaei-Bigdoli Guy Albertelli Gerd Kortemeyer Edwin Kashy

2. Michigan State University Alfred P. Sloan Foundation Michigan State University Andrew W. Mellon Foundation National Superconducting Cyclotron Laboratory National Science Foundation Acknowledgements

3. Michigan State University How can we keep instructors informed about how their students are doing? How can an instructor tell where their students are going wrong? How can an instructor use this feedback to provide a better learning experience for their students? Questions

4. Michigan State University Physics 183 Class time traditional lecture demonstrations lecture exercises quizzes Resources for students physics learning center office hours First year Calculus based Physics

5. Michigan State University First Year Calculus based Physics Course Assessment individualized on-line homework with feedback supplemental problems, and mid-term and final exams Evaluation mid-term tests and final exam

6. Michigan State University Course Management System LON-CAPA Individualized problems highly randomized Multiple response types numerical, classification, ranking, click-on-image, etc Immediate feedback multiple attempts

7. Michigan State University Class Statistics Students (S = 400) Total number of problems (P = 250) Multiple attempts are allowed (A <= 15) Problems can have many responses (R = 1..10) Volume of data is significant –S x A x R for a given problem –S x P x A x R for all problems in the course

8. Michigan State University Variety of data Each student receives their own version of a problem –Different statements –Different pictures –Different answers

9. Michigan State University Cutting through the complexity and volume of data General statistics of class performance on a single problem Feedback on responses to complex conceptual problems Feedback to highlight specific student errors Evaluation of resources for exams Evaluation of overall student performance

10. Michigan State University General Statistics of Class Performance on a Single Problem For a given problem number of students attempting number of attempts number of correct attempts number of attempts per correct answer

11. Michigan State University Number of attempts per correct answer

12. Michigan State University A Complex Conceptual Problem Classify Statements 14 statements grouped into 6 concepts One statement is presented from each concept Each statement must be classified as one of 5 options All classifications must be correct

13. Michigan State University Feedback on a Complex Conceptual Problem

14. Michigan State University Final week Following weekBefore discussion Correct Answers by Concept

15. Michigan State University Feedback to Highlight Student Errors Vector addition problem Classify statements based on comparison of two quantities 14 statements, 7 of which are the reverse of the other statements, grouped into 5 concepts

16. Michigan State University Feedback to Highlight Student Errors 3. Time to row across for K is.. for C 4. Time to row across for C is.. for K

17. Michigan State University Evaluation of Resources used on Exams Degree of difficulty DIFF = 1 – (earned points / possible points) Degree of discrimination Classify students by performance on other exam questions Use same ratio as above DISC = (ratio for good students) – (ratio for bad students) -1 to +1 +1 Only good students earned points 0 everyone did the same -1 Only poor students earned points

18. Michigan State University Mid-term Exam Statistics ProblemExam 1 DIFF Exam 1 DISC Exam 2 DIFF Exam 2 DISC 1 2 3 4 5 6 7 8 9 10 11 0.20 0.16 0.40 0.44 0.32 0.00 0.23 0.21 0.36 0.40 0.25 0.40 0.31 0.40 0.57 0.38 0.00 0.33 0.24 0.63 0.59 0.31 0.70 0.13 0.19 0.41 0.52 0.18 0.70 0.57 0.55 0.87 0.24 0.20 0.31 0.57 0.11 0.26 0.36 0.35 0.58 0.14

19. Michigan State University Evaluating Class Performance Cumulative homework scores Cumulative exam scores

20. Michigan State University 3-D grade plot

21. Michigan State University Evaluating Class Performance Correlation is moderate (r=0.43) Most students do well on homework Students think doing well on homework means success in the class The instructor implemented a “grade extrapolator” to allow students to interactively pose their future performance and view the resulting grade

22. Michigan State University Presented tools for feedback The feedback reflects the complexity of the problems More feedback tools needed –different problem types –different presentations of student answers Conclusion

23. Michigan State University Extra Slide - Atwood Problem Analysis, Massless Pulley