Nabeel Alzahrani1, Frank Vahid1,2, Alex Edgcomb1,2 ­­­Manual Analysis of Homework Coding Errors for Improved Teaching and Help Nabeel Alzahrani1, Frank Vahid1,2, Alex Edgcomb1,2 1 Computer Science and Engineering, University of California, Riverside  2 zyBooks, Los Gatos, California   nalza001@ucr.edu, vahid@cs.ucr.edu, aedgcomb@cs.ucr.edu Frank Vahid, UC Riverside, 2019 /15

Frank Vahid, UC Riverside, 2019 State of CS1 33% mean fail rate UCR CS1 10+ years ago https://cra.org/wp-content/uploads/2018/05/2017-Taulbee-Survey-Report.pdf Not-uncommon CS1 grades * 54 articles, 51 institutions, 15 countries, 161 CS1 courses Watson, Failure rates in introductory programming revisited, ITiCSE, 2014 Frank Vahid, UC Riverside, 2019 http://reflect.otago.ac.nz/staffpriv/anthony/publications/pdfs/RobinsLEM.pdf /15

Frank Vahid, UC Riverside, 2019 Before UCR CS1 After 300-500 students / quarter, 3-5 sections (1 online), ½ majors ½ non-majors Aggressively seeking improvement Textbook  interactive content Auto-graded coding hwk (100s) / labs (50+) Frank Vahid, UC Riverside, 2019

Excessive Struggle: Too many attempts, can demoralize Solving a small coding problem 2-3x healthy 10x or more frustrating 🡪 Wrong “Excessive struggle” Right Frank Vahid, UC Riverside, 2019 /15

Analyzed struggle on small coding activities (CAs) in C++ zyBook 2-5 min / CA NOT programming assignments 10 CAs per week, 100 CAs per quarter “Fact-finding” mission 3 undergrad interns, 1 PhD, >200 hours Results can improve teaching & content Frank Vahid, UC Riverside, 2019 /15

Frank Vahid, UC Riverside, 2019 Student submissions Submission time User # Correct? Submitted solution 5/7/2017   11:50:03 PM 0xxxx No for (i=0; i <= numRows; ++i) {   for (j=0; j<= numCols; ++j) {      return static_cast<char>('A' - 1 + i);        cout << i << j;}} ... 11:52:54 PM        cout << i << j << " ";   }} 5/8/2017   12:01:37 AM Yes for (i=1; i <= numRows; ++i) {   for (j=0; j< numCols; ++j) {        cout << i << static_cast<char>('A' + j) << " "; Submissions recorded in a csv file with: timestamp, user id, Correctness (yes/no), code Frank Vahid, UC Riverside, 2019 /15

CAs sorted by struggle rate Manually analyzed top 12 Struggling if time > 5 min and time > 2 x baseline time And attempts > 3 attempts and time > 2 x baseline attempts (min.) Frank Vahid, UC Riverside, 2019 /15

CA 4.3.2: Whitespace replace Frank Vahid, UC Riverside, 2019 /15

CA 4.3.2: Whitespace replace Whitespace replace (CA 4.3.2). struggling students vs non-struggling students. (G): a general error (S): a problem-specific error. x/y: x is the number of students, y is the median number of submissions containing that error. time: the average time spent solving that specific error Frank Vahid, UC Riverside, 2019 /15

Frank Vahid, UC Riverside, 2019 CA 4.3.1: String with digit Frank Vahid, UC Riverside, 2019 /15

Frank Vahid, UC Riverside, 2019 CA 4.3.1: String with digit String with digit (CA 4.3.1). struggling students vs non-struggling students. (G): a general error (S): a problem-specific error. x/y: x is the number of students, y is the median number of submissions containing that error. time: the average time spent solving that specific error Frank Vahid, UC Riverside, 2019 /15

CA 5.4.1: Nested loops indent Frank Vahid, UC Riverside, 2019 /15

CA 5.4.1: Nested loops indent Nested loops Indent (CA 5.4.1). struggling students vs non-struggling students. (G): a general error (S): a problem-specific error. x/y: x is the number of students, y is the median number of submissions containing that error. time: the average time spent solving that specific error Frank Vahid, UC Riverside, 2019 /15

CA 3.10.2: Bool in branching statements Frank Vahid, UC Riverside, 2019 /15

CA 3.10.2: Bool in branching statements Bool in branching statements (CA 3.10.2). struggling students vs non-struggling students. (G): a general error (S): a problem-specific error. x/y: x is the number of students, y is the median number of submissions containing that error. time: the average time spent solving that specific error Frank Vahid, UC Riverside, 2019 /15

CS 4.7.3: Basic while loop expression Frank Vahid, UC Riverside, 2019 /15

CS 4.7.3: Basic while loop expression Basic while loop expression (CA 4.7.3). struggling students vs non-struggling students. (G): a general error (S): a problem-specific error. x/y: x is the number of students, y is the median number of submissions containing that error. time: the average time spent solving that specific error Frank Vahid, UC Riverside, 2019 /15

Frank Vahid, UC Riverside, 2019 CA 5.5.2: rand() function Frank Vahid, UC Riverside, 2019 /15

Frank Vahid, UC Riverside, 2019 CA 5.5.2: rand() function Rand function seed and then get random numbers (CA 5.5.2) struggling students vs non-struggling students. (G): a general error (S): a problem-specific error. x/y: x is the number of students, y is the median number of submissions containing that error. time: the average time spent solving that specific error Frank Vahid, UC Riverside, 2019 /15

Common errors leading to struggle (Top 15) User ID CA Time (min.) General error Problem-specific error One-off error 5680 4.4.2 50 Hard coding 1055 5.4.2 48 Nested loop conditions 0847 5.4.1 46 Wrong for loop num and operator 43 1131 37 Using int instead of char 2055 3.7.1 35 Misunderstanding question (thinks problem is asking for a range of values, when it just wants specific values) 0514 4.8.2 34 No endl at the end or endl inside the while loop 0525 2.10.2 31 Incorrect math (various combinations of division, modulus and subtraction of different numbers) 0765 3.10.2 30 Using = instead of == 1929 29 Not realizing character has an ASCII value in loop condition (eg. for( seat = 'A' ; seat <= 3; ++seat) this loop will never be entered) 9967 9.1.2 28 Not taking 1 element array into account 9603 26 Outputting ASCII number instead of character 0676 4.3.2 25 Writing "str.isspace()==true" when it actually returns an integer 0241 1902 24 Using "if/else" instead of "if and if" 9602 Wrong if condition logic (e.g. if (!isBalloon && isRed) instead of if(isBalloon && isRed) ) Frank Vahid, UC Riverside, 2019 /15

Common errors Generic Specific One-off Misunderstanding the problem Not realizing character has an ASCII value in loop condition Not taking 1 element array into account Outputting ASCII number instead of character Writing"str.isspace()==true" when it actually returns an integer Hard coding Nested loop conditions Wrong for loop num and operator Incorrect math Using = instead of == Using "if/else" instead of multiple "if" Wrong if condition logic Using int instead of char No endl at the end or endl inside the while loop cout before cin Not putting condition for if statement Frank Vahid, UC Riverside, 2019 /15

Frank Vahid, UC Riverside, 2019 Conclusions Analyzed struggle, top 12/89 C++ zyBook coding activities (CAs) (~100 students) Struggle rates 34-56% Generic vs. specific vs. one-offs Possible interventions Help systems: Auto-detect, provide hint after x tries (include one-off errors if possible) Detect struggle, provide help (suggest getting help)? Lecture time, better material, debug examples Frank Vahid, UC Riverside, 2019 /15

Struggle rate as a metric Intuition: Baseline time = average time of top 20% students Baseline attempts = average attempts of top 20% students Struggling if time > 5 min and time > 2 x baseline time And attempts > 3 attempts and time > 2 x baseline attempts Frank Vahid, UC Riverside, 2019 /15