1. Nabeel Alzahrani1, Frank Vahid1,2, Alex Edgcomb1,2
­­­Manual Analysis of Homework Coding Errors for Improved Teaching and Help
Nabeel Alzahrani1, Frank Vahid1,2, Alex Edgcomb1,2
1 Computer Science and Engineering, University of California, Riverside 
2 zyBooks, Los Gatos, California  
Frank Vahid, UC Riverside, 2019
/15

2. Frank Vahid, UC Riverside, 2019
State of CS1
33% mean fail rate
UCR CS1 10+ years ago
Not-uncommon CS1 grades
* 54 articles, 51 institutions, 15 countries, 161 CS1 courses
Watson, Failure rates in introductory programming revisited, ITiCSE, 2014
Frank Vahid, UC Riverside, 2019
/15

3. Frank Vahid, UC Riverside, 2019
Before
UCR CS1
After
students / quarter, 3-5 sections (1 online), ½ majors ½ non-majors
Aggressively seeking improvement
Textbook  interactive content
Auto-graded coding hwk (100s) / labs (50+)
Frank Vahid, UC Riverside, 2019

4. Excessive Struggle: Too many attempts, can demoralize
Solving a small coding problem
2-3x healthy
10x or more frustrating 🡪
Wrong
“Excessive struggle”
Right
Frank Vahid, UC Riverside, 2019
/15

5. Analyzed struggle on small coding activities (CAs) in C++ zyBook
2-5 min / CA
NOT programming assignments
10 CAs per week, 100 CAs per quarter
“Fact-finding” mission
3 undergrad interns, 1 PhD, >200 hours
Results can improve teaching & content
Frank Vahid, UC Riverside, 2019
/15

6. Frank Vahid, UC Riverside, 2019
Student submissions
Submission time
User #
Correct?
Submitted solution
5/7/2017  
11:50:03 PM
0xxxx No
for (i=0; i <= numRows; ++i) {
  for (j=0; j<= numCols; ++j) {
     return static_cast<char>('A' i);
       cout << i << j;}}
...
11:52:54 PM
       cout << i << j << " ";
  }}
5/8/2017  
12:01:37 AM
Yes
for (i=1; i <= numRows; ++i) {
  for (j=0; j< numCols; ++j) {
       cout << i << static_cast<char>('A' + j) << " ";
Submissions recorded in a csv file with:
timestamp,
user id,
Correctness (yes/no),
code
Frank Vahid, UC Riverside, 2019
/15

7. CAs sorted by struggle rate
Manually analyzed top 12
Struggling if
time > 5 min
and time > 2 x baseline time
And
attempts > 3 attempts
and time > 2 x baseline attempts
(min.)
Frank Vahid, UC Riverside, 2019
/15

8. CA 4.3.2: Whitespace replace
Frank Vahid, UC Riverside, 2019
/15

9. CA 4.3.2: Whitespace replace
Whitespace replace (CA 4.3.2).
struggling students vs non-struggling students.
(G): a general error
(S): a problem-specific error.
x/y: x is the number of students, y is the median number of submissions containing that error.
time: the average time spent solving that specific error
Frank Vahid, UC Riverside, 2019
/15

10. Frank Vahid, UC Riverside, 2019
CA 4.3.1: String with digit
Frank Vahid, UC Riverside, 2019
/15

11. Frank Vahid, UC Riverside, 2019
CA 4.3.1: String with digit
String with digit (CA 4.3.1).
struggling students vs non-struggling students.
(G): a general error
(S): a problem-specific error.
x/y: x is the number of students, y is the median number of submissions containing that error.
time: the average time spent solving that specific error
Frank Vahid, UC Riverside, 2019
/15

12. CA 5.4.1: Nested loops indent
Frank Vahid, UC Riverside, 2019
/15

13. CA 5.4.1: Nested loops indent
Nested loops Indent (CA 5.4.1).
struggling students vs non-struggling students.
(G): a general error
(S): a problem-specific error.
x/y: x is the number of students, y is the median number of submissions containing that error.
time: the average time spent solving that specific error
Frank Vahid, UC Riverside, 2019
/15

14. CA 3.10.2: Bool in branching statements
Frank Vahid, UC Riverside, 2019
/15

15. CA 3.10.2: Bool in branching statements
Bool in branching statements (CA ).
struggling students vs non-struggling students.
(G): a general error
(S): a problem-specific error.
x/y: x is the number of students, y is the median number of submissions containing that error.
time: the average time spent solving that specific error
Frank Vahid, UC Riverside, 2019
/15

16. CS 4.7.3: Basic while loop expression
Frank Vahid, UC Riverside, 2019
/15

17. CS 4.7.3: Basic while loop expression
Basic while loop expression (CA 4.7.3).
struggling students vs non-struggling students.
(G): a general error
(S): a problem-specific error.
x/y: x is the number of students, y is the median number of submissions containing that error.
time: the average time spent solving that specific error
Frank Vahid, UC Riverside, 2019
/15

18. Frank Vahid, UC Riverside, 2019
CA 5.5.2: rand() function
Frank Vahid, UC Riverside, 2019
/15

19. Frank Vahid, UC Riverside, 2019
CA 5.5.2: rand() function
Rand function seed and then get random numbers (CA 5.5.2)
struggling students vs non-struggling students.
(G): a general error
(S): a problem-specific error.
x/y: x is the number of students, y is the median number of submissions containing that error.
time: the average time spent solving that specific error
Frank Vahid, UC Riverside, 2019
/15

20. Common errors leading to struggle (Top 15)
User ID CA
Time (min.)
General error
Problem-specific error
One-off error
5680
4.4.2 50
Hard coding
1055
5.4.2 48
Nested loop conditions
0847
5.4.1 46
Wrong for loop num and operator 43
1131 37
Using int instead of char
2055
3.7.1 35
Misunderstanding question (thinks problem is asking for a range of values, when it just wants specific values)
0514
4.8.2 34
No endl at the end or endl inside the while loop
0525
2.10.2 31
Incorrect math (various combinations of division, modulus and subtraction of different numbers)
0765
3.10.2 30
Using = instead of ==
1929 29
Not realizing character has an ASCII value in loop condition (eg. for( seat = 'A' ; seat <= 3; ++seat) this loop will never be entered)
9967
9.1.2 28
Not taking 1 element array into account
9603 26
Outputting ASCII number instead of character
0676
4.3.2 25
Writing "str.isspace()==true" when it actually returns an integer
0241
1902 24
Using "if/else" instead of "if and if"
9602
Wrong if condition logic (e.g. if (!isBalloon && isRed) instead of if(isBalloon && isRed) )
Frank Vahid, UC Riverside, 2019
/15

21. Common errors Generic Specific One-off
Misunderstanding the problem
Not realizing character has an ASCII value in loop condition
Not taking 1 element array into account
Outputting ASCII number instead of character
Writing"str.isspace()==true" when it actually returns an integer
Hard coding
Nested loop conditions
Wrong for loop num and operator
Incorrect math
Using = instead of ==
Using "if/else" instead of multiple "if"
Wrong if condition logic
Using int instead of char
No endl at the end or endl inside the while loop
cout before cin
Not putting condition for if statement
Frank Vahid, UC Riverside, 2019
/15

22. Frank Vahid, UC Riverside, 2019
Conclusions
Analyzed struggle, top 12/89 C++ zyBook coding activities (CAs) (~100 students)
Struggle rates 34-56%
Generic vs. specific vs. one-offs
Possible interventions
Help systems: Auto-detect, provide hint after x tries (include one-off errors if possible)
Detect struggle, provide help (suggest getting help)?
Lecture time, better material, debug examples
Frank Vahid, UC Riverside, 2019
/15

23. Struggle rate as a metric
Intuition:
Baseline time = average time of top 20% students
Baseline attempts = average attempts of top 20% students
Struggling if
time > 5 min
and time > 2 x baseline time
And
attempts > 3 attempts
and time > 2 x baseline attempts
Frank Vahid, UC Riverside, 2019
/15