1. Automated Assessment of Programming Exercises

2. Programming Exercises
Can categorize into:
Large
Small
There are similarities and differences in what the system provides for these use cases.
We will start with systems for “small” exercises

3. Codingbat Pretty basic, but a good yardstick to measure against.
Fairly widely used in CS
Small exercises
Lots of sites have answers.

4. CodeAcademy

5. Dynamic vs. Static Assessment
The goal is to (automatically) assess, and give feedback
Dynamic Assessment: The runtime behavior of the student’s answer
We generally use this as a proxy for “Does it compute the right function?”
A heuristic: Unit tests
Static Assessment: Evaluation of the “quality” of the student’s answer (distinct from correctness)
Does it do things “in the right way”
Heuristics on the code, NOT its behavior

6. OpenDSA Programming Exercises
Geared for small exercises (write/edit 5-20 lines of code).
Evaluated in a “sandbox” on a server.
Java-only right now
Process:
Check for compiler errors, return if any.
Dynamic analysis: Run unit tests. Feedback if there are problems.
Static analysis: Check various heuristics.

7. Feedback: Program Visualization
Algorithm Visualization vs. Program Visualization
AV: Tutorial presentation of an algorithm. The algorithm is “baked into” the AV
PV: Visual feedback generated from a program to help understand that particular program
jGRASP
JhavePOP
Code Mirror (PythonTutor, Guo)
PV does not address the issue of whether the program is “correct”, it helps the user to decide that for themselves

8. Khan Academy (Programming)
HTML5-based
Integrates tutorial content, quizzes, and programming exercises
At least some exercises are HEAVILY scaffolded, and heavily scripted with detailed requirements.
programming/programming

9. Web-CAT Designed to support grading of large programs
Aspects of (support for) manual grading, and automated grading
Style checking
Unit testing (test cases “correct” or not, time constraint)
Testing vs. debugging
Code coverage of student tests