1. Evaluating the Effectiveness of Feedback in SQL-Tutor Antonija Mitrovic, Brent Martin Intelligent Computer Tutoring Group Computer Science Department University of Canterbury Christchurch, New Zealand

2. Overview SQL-Tutor Feedback types Evaluation study Results

3. Architecture of the standalone version of SQL-Tutor

5. SQLT-Web: a Web-enabled SQL tutor http://ictg.cosc.canterbury.ac.nz:8000/sql-tutor/ Interface developed in CL HTTP-server Used in COSC courses since 1999 Open for outside users on March 29, 2000

6. Architecture of SQLT-Web

8. Feedback types Positive/negative Error flag Hint All errors Partial solution Complete solution

9. Hypothesis: Positive/negative and Error flag too general Low-level feedback contraproductive (partial and complete solution) Constraint-based feedback most effective (hint and all errors)

10. Evaluation study COSC313, May 1999 (Web) Single 2-hour session 6 lectures + 8 hours of labs 33 students

11. Three groups: Detailed (complete and partial solution) General (hint and all errors) Limited (pos/neg and error flag)

12. Results 1 GroupSolved (%)NoAttemptsTime/attempt Detailed87.072.2165.26s General83.492.1647.80s Limited84.102.2478.05s Difference in times is significant

13. Results 2 FeedbackConstSuccFailLearnCorr (%) Pos/neg43625463629%78.0 Error flag1169812644%81.8 Hint43334343%74.4 All errors64728147%80.0 Partial26221069%91.6 Full sol1861627%44.2

15. Initial learning rate All errors0.44 Error Flag0.40 Pos/Neg0.29 Hint 0.26 Partial0.15 Full solution0.13

16. Conclusions General feedback is most effective Detailed feedback is detrimental to learning Feedback level needs to be adapted