1. ___________________ 1 GradeBot School of Computing Copyright 2004, BYU Hawaii – All rights reserved. School of Computing Brigham Young University Hawaii Laie, Hawaii 96762, USA Don Colton Leslie Fife Randy Winters Building a Computer Program Grader

2. ___________________ 2 Introduction  80% of our new students have never programmed before.  With one program per week, there is too much to learn per program.  With more programs per week, there is too much to grade.  Automated grading is a solution.

3. ___________________ 3 Automated Grading Benefits  More Learning – because Students fix their own bugs.  Less Partial Credit – instead, Try Until You Get It Right. This is really a substantial paradigm shift.  Fast Response – Grader responds immediately, 24x7.  More Students per Teacher – without more prep time.

4. ___________________ 4 Grading Model  Students Submit Source Code  GradeBot compiles it and tests it.  Test inputs are constructed randomly (within limits). Students cannot “memorize” the answers.  Expected outputs are created by the “teacher's” program, using the same inputs.  When a test is failed, the student is given the actual input and the required output.

5. ___________________ 5 Log In

6. ___________________ 6 Select a Lab

7. ___________________ 7 Learn About the Assignment

8. ___________________ 8 Key in your Program

9. ___________________ 9 Test Your Program

10. ___________________ 10 Save Your Work

11. ___________________ 11 Status Report

12. ___________________ 12 Student Praise  My mistakes are not penalized. I can try again.  I can work whenever I want to, 24 by 7.  I can submit my labs from home.  I always know my grade in the class.  I always know when things are due.  I love the feeling of getting something right, and knowing that it is right. GradeBot says I am done.  I do not have to wait for the instructor to grade my work and tell me it is wrong and too late to fix it.

13. ___________________ 13 Student Complaints  The spacing has to be exactly right.  No problem; keep trying until you get it to match. There is no penalty for submitting again.  GradeBot is too picky.  The programmer must make the customer happy.  Where is my opportunity for creativity?  GradeBot only specifies the output; You are free to generate it in any way you like.  What if I want to write my OWN program?  Use a regular compiler and do your own testing.

14. ___________________ 14 We Noticed Cheating  Students Help Each Other – Even when such collaboration is explicitly forbidden.  Man vs. Machine – Students seemed less upset about cheating in their interactions with a machine than with interactions with humans.  Student vs. Teacher – In some cultures, there is a strong us- versus-them mentality relating students to teachers.  Keeping a History – Duplicate work can be identified and patterns of copying can be discovered and proven.  Some students were failed or expelled – But does the punishment fit the crime?

15. ___________________ 15 We Redefined Cheating  We gave up on preventing student collaboration – There were too many issues to overcome. And we started to doubt the wisdom in killing collaboration.  We substituted In-Class Tests – The grade rests on in-class tests. This rewards those who learn, whether they collaborated or not.  Collaboration is now Accepted – Students are openly invited to collaborate, but reminded that they must do well on the test.

16. ___________________ 16 Results  GradeBot has been operational for four years handling an average of 400 students per year, 30 programs per student, dozens of submissions per program.  GradeBot is used with many student program languages.  Instructors like GradeBot and want its use expanded. CS101, CS201, CS210, CS301, Math201, Math202  Students report a love/hate relationship with GradeBot. Mostly love.  The quality of student programming skills seems to have improved a lot.

17. ___________________ 17 Questions ?