1. Introductory Computer Programming, Problem Solving and Computer Assisted Assessment Charlie Daly, DCU John Waldron, TCD

2. Preview: A Problem A Programmable Robot

4. A Problem Solving a maze. Program a Robot to Solve a maze Analyse a program that is supposed to solve a maze

5. But... Anybody can check if the program works. –Create a maze and check if the robot can solve it. And provide feedback if it doesn’t work –Show what the robot did.

6. So... Certain problems are very challenging...... but they can be checked very simply... easy to provide useful feedback to faulty programs End of Preview Mathematical proofs similar, easy to check, difficult to conceive

7. Programming is Problem Solving "programming is a complex intellectual activity based on deep logical principles" (Felleisen 2002) Polya's "How to Solve it" "sheds more light on the activity of software design … than any software design book I've seen" (McConnell 2002)

8. Learning Programming Learner programmers must master –programing language syntax –problem solving techniques. Computer feedback is useful for both –Compiler catches syntax errors –Testing catches logic errors } RoboProf catches both of these

9. Talk Outline Programming Courses are not working. Programming ability is difficult to assess The Solution: Proper Assessment Implementation issues (software and peopleware) Results Conclusions

10. Programming Courses are not working Students in Introductory Programming Courses do not learn to program! An international multi-institutional study of introductory programming courses ITiCSE 2001 “Do students in introductory computing courses know how to program at the expected skill level?”

11. No!

12. What is Wrong? Assessment "The spirit and style of student assessment defines the de facto curriculum" “Assessing Students”, Derek Rowntree '77

13. What is Wrong? It is difficult to assess programming ability in a traditional written exam. Programming exercises are subject to plagiarism; a serious problem in introductory programming courses. If you do not assess something, the students will not learn it.

14. What's wrong with Exams Two sides of programming –language syntax (easy to examine) –problem solving (hard to examine) Unoriginal (repeat) questions Marks for attempting a question Assuming insight where none exists Objectivity Lecturer doesn't want to fail whole class. One exam

15. What's wrong with Assignments Most students don't see a problem with using somebody else's code ("if I understand it it's OK") Plagiarism is a huge problem. Lecturers frequently know it's happening and do nothing. don't understand there's a difference between writing and understanding

16. The Bright Shining Lie Lecturers think the students know how to program –they passed the exams Students think they know how to program –they passed the exams Students have an excuse: for students education is about passing exams

17. Talk Outline Programming Courses are not working. Programming ability is difficult to assess The Solution: Proper Assessment Implementation issues (software and peopleware) Results Conclusions

18. The Solution: Proper assessment Come up with original challenging problems. Mark properly; only give marks if the solution is completely correct. Allow the student to get computer feedback. –(compiler errors, testing) programming is a process Make students aware of the assessment!

19. Context: the course Introductory Programming Course –one semester: 12 weeks of lectures Students –400 (300 pure Computing) –no prior programming experience –education ≡ passing exams Exams –programming exams week 6 and 12 (Weighting 30%) –written exam week 16 (Weighting 40%)

20. RoboProf Automated Program marker WWW interface Runs a student's program using different input and check that the program produces the correct output. Student is shown the result and (if not correct) may modify and resubmit their program. more

21. RoboProf Server Problem Specification The Student logs on An applet compiles and runs the program locally The program and output are sent to the server for marking Results are returned to the student Browser The student writes a program and submits it

22. The Programming Exam 3 Questions (in increasing difficulty) 2 hours (first half hour without computer) Not open book, but may use a 'cheatsheet' During the exam the students –Write a program –Submit it to RoboProf –View feedback –May resubmit without penalty

23. The Programming Exam Students know their result at the end of the exam. –Fewer complaints –Greater insight into their ability –More likely to listen when subsequently shown a correct solution General effect: formative assessment works.

24. RoboProf Vs Manual Exam Standard CAA advantages –Objective –Fast Feedback Avoids the problem of the manual marker interpreting the student solution Models standard program development process (computer feedback) Huge resource requirements (~400 PCs)

27. RoboProf Vs Manual Exam Results: Not much difference. Markers only awarded marks for programs that definitely looked like they might work. Students knew that the marking system was similar to the programming exams.

28. Student Impressions

29. "I liked the web-based test"

30. Student Impressions "The marking system was fair"

31. Comparison with previous years There were too many simultaneous changes to the course to draw firm conclusions (programming language, number of students, lecturers, facilities) Results of last three years –1999: 30% –2000: 20% –2001: ?

32. Conclusions CAA can evaluate problem solving ability –Using approriate problems Programming problems have added advantages –Models the program development process. –Can show errors without showing the solution. –Avoids the inherent problems that humans have assessing programming problems

33. Future Work Improve Feedback Make it adaptable (can be used by other institutions)

34. Question 1 Write a program that asks for a price and makes an assessment: if the price is more than £100.00, then print "too high", if it is less than £50.00 print "that's cheap" otherwise print "OK" A sample run might be What is the price? 120.99 too high or What is the price? 51.50 OK

35. Question 3 (Haggling) A street seller needs a program to help him haggle. He only sells one item, worth £100, as soon as he's offered this (or more) he accepts the offer. If he's offered less, he demands a price such that the average of the offer and the demanded price is £100. He keeps this up until a deal is done. A sample run might be What is your offer? 400 It's a deal! or What is your offer? 50 I'll sell it for 150. What is your offer? 90 I'll sell it for 110. What is your offer? 100 It's a deal! You can assume that a deal is always done. Note that the first demanded price was 150. This was in response to the offered price of 50 (the average of 150 and 50 is 100).

36. Question 1 Write a program that reads in a sequence of integers and calculates the average, then prints out how many elements are greater than the average. A sample run might look like: How many numbers 7 Enter 7 numbers: -6 8 -17 67 10 -7 0 3 numbers are greater than the average. Note that the average is about 7.9, the elements 8, 67 and 10 are greater than that.

37. Question 2 Write a program that finds the tags in a String of characters. Print each tag on a separate line. Tags are contained in braces '{' and '}'. You may assume that the String is well formed, i.e. all braces match. A sample run might look like: Enter a string: Here is a tagged String {one} and {another}. The tags are One another Hint: You may find the following String methods useful.