Retina: Helping Students and Instructors Based on Observed Programming Activities Chris Murphy, Gail Kaiser, Kristin Loveland, Sahar Hasan Columbia University

2 Observation #1 It is difficult for CS1 instructors to get quality answers to such questions as: “How long are students taking to complete the programming assignments?” “What sorts of compilation and runtime errors are they encountering?” “Which students are struggling the most with this assignment?”

3 Observation #2 Students in CS1 courses cannot easily get the answers to such questions as: “Are other students taking the same amount of time that I am to complete the assignment?” “Are other students getting the same compilation and runtime errors?” “Is this assignment hard for everyone else, too?”

4 Observation #3 Both CS1 instructors and students can benefit from organizational memory: “What problems did students encounter with this assignment last semester? How long did it take students to complete this assignment?” “How long will this assignment take me? What sorts of errors should I look out for?”

5 Our Solution: Retina Collects objective observational data about students’ programming activities (focusing primarily on compilation errors) Provides useful and informative reports based on the aggregation of that data Can also make proactive recommendations and suggestions to the students

6 Overview Related Work Retina System Architecture Instructor View Student View Real-time Recommendations Evaluation Conclusions and Future Work

7 Related Work Monitoring and logging student programming activities  ClockIt, Hackystat  EclipseWatcher Collecting data and automating testing  Marmoset, Web-CAT Recommendation systems  Strathcona

8 wrapper plugin Architecture/Dataflow EclipseBlueJ javac Retina Data Collector db Retina Data Manager Retina Instructor View Retina Student View TomcatJClaim IM Client (Yahoo, MSN, Google, etc.)

9 Data Collection For each compilation event, Retina records:  Username  Date & time  File name For each compilation error, Retina also records:  Type of error  Error message  Line number

10 Instructor View Browse Mode  Understanding of individual students’ activities  Compilation errors  Working times and time spent Class Mode  Overview of entire class’ performance  Most common errors  List students sorted by number of errors made or time spent on assignment

11 Student View Historical information  Compilation errors made  Time spent on assignment Class comparisons Suggestions  How long next assignment may take  How to address most common error

12 Real-time Recommendations Proactively sent to users over IM Instructor can configure frequency of recommendations based on:  High rate of errors per compilation  Spending too long on the assignment  Same error made multiple times Student can also interact with system to get other info like how many other students are currently working on the assignment

13 Evaluation 48 students volunteered to have their data collected  21 in Spring 2008  27 in Summer 2008 Instructors were asked to comment on the usefulness of the Instructor View A small number of students used the Student View and Real-time Recommendation

14 Instructor Comments “Retina was useful in the case where a student was asking for one-on-one help, so that I could know in advance what difficulties that student had, could anticipate the questions the student would ask, and could tailor the help appropriately.” “By seeing what errors the students were making as a whole, I could also warn the TAs what to look out for, and discuss with them good ways to help the students address those problems.”

15 Student Comments “I was a bit nervous at first about having my data collected but it was useful to see what errors I had made in the past, and that helped me remember how I fixed them.” “I was really surprised to see that other students were spending more time on the assignment than I was. It made me feel like I wasn’t the worst student in the class.”

16 Data Analysis Did not find any correlation between time spent on the assignment and the grades, but when we considered the performance over the entire the semester, we noticed that students who spent less time on the assignments tended to do better than students who spent more time The highest rates of errors per compilation occurred between 1am and 4am, with these values being substantially higher than all other values for errors per compilation per hour

17 Future Work Empirical studies to measure usefulness of Student View and Recommendation tool Creating ad hoc social networks as students work on assignments Addressing privacy concerns and issues related to plagiarism

18 Special Thanks

19 Conclusion Our contribution is a tool called Retina Retina helps instructors learn about their students’ programming activities Retina helps students learn from their past activities and understand how they relate to other students

20 Retina: Helping Students and Instructors Based on Observed Programming Activities Chris Murphy