1. 1 Effective Feedback to the Instructor from Online Homework Michigan State University www.lon-capa.org Mark Urban-Lurain urban@msu.edu Gerd Kortemeyer korte@lite.msu.edu Copyright Gerd Kortemeyer & Mark Urban-Lurain, 2005. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. 2 Abstract LON-CAPA allows instructors to efficiently create and distribute educational materials, assignments, assessments, etc. These include numerous types of formative conceptual and algorithmic exercises for which prompt feedback and assistance can be provided to students as they work on assigned tasks. LON-CAPA records and dynamically organizes a vast amount of information on students' interaction with and understanding of these materials LON-CAPA helps faculty interpret this data to identify students' misconceptions and other areas of difficulty, so that timely corrective action can be taken. This information also facilitates detailed studies of the educational resources help faculty redesign the instructional materials and courses.

3. 3 Course or Problem Overview

4. 4 Excerpt of homework performance –Students in the rows –Problems in the columns –Each character is one online problem for one student Number of attempts each for student to get that particular problem correct –* means more than nine attempts –. unsolved problem, blank an un-attempted problem. Useful before problem deadline –Columns with large number of dots or blank spaces indicate problems students have difficulties with

5. 5 Individual Student Responses

6. 6 Student thought that the force would increase in the presence of a medium between the charges First answer: force would double –No feedback except "incorrect" was provided by the system Next attempt: changed answer on only this statement to "four times the force" Only ten seconds passed between the attempts, indicating guessing by which factor the force increased

7. 7 Item Analysis

8. 8 Course-wide per-problem view Graphs on left: statements answered correctly the first and on the second attempt Graphs on right: which options students chose if the statement was answered incorrectly. Students have most difficulty with concept of how a medium acts between charges –Majority believe force would increase –20% believe medium has no influence Instructors can address these misconceptions in class

9. 9 Difficulty

10. 10 Difficulty Ratio of number of submissions to number of correct responses for 17 problems –5 days before assignment due (15% of the 400 students) Five questions are challenging –Each required more than four submissions per success on average. Instructor can adapt lectures and recitations to meet learner needs

11. 11 Concept Analysis

12. 12 Concept Analysis Problems highly randomized among students Internally, options grouped into six "concept groups," each focusing on a particular physics aspect of the problem. Every student gets one statement (with the correct labels filled in) from each one of these concept groups

13. 13 Concept Group First Submission

14. 14 Concept Group First Submission Item analysis is done by concept group, not by statement, independent of randomization Success rate on each concept group for the initial submission

15. 15 Concept Group Second and Third Submissions

16. 16 Concept Group Second and Third Submissions Examining submission records indicates concept 5 is easiest, across all three submissions Concept 3 is most difficult Consistent regardless of the number of submission attempts Instructor should address concept 3

17. 17 Improved Learning LON-CAPA provides detailed feedback on learner performance on online homework problems LON-CAPA allows faculty to quickly and efficiently analyze this data in several ways Faculty can quickly adjust lectures, recitation sessions and individual help to address learner needs

18. 18 Acknowledgements National Science Foundation (ITR-0085921; CCLI- ASA-0243126). Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Alfred P. Sloan and Andrew W. Mellon Foundations Michigan State University, and its administrators for over a decade of encouragement and support