Preliminary Analysis of Contestant Performance for a Code Hunt Contest Adrian Clark, Jonathan Wells, Angello Astorga, Andrew Xie Jalen Coleman-Lands, Tao Xie University of Illinois at Urbana-Champaign, IL, USA

Code Hunt Is a serious programming game Works in C# and Java Appeals to coders wishing to hone their programming skills And also to students learning to code Since its release in Spring 2014 till now, for about 3.5 years, more than 6 million (6,112,500) users Including registered users playing on www.codehunt.com, anonymous users and accounts that access api.codehunt.com directly via the documented REST APIs). http://api.codehunt.com/ Stickiness (loyalty) is very high https://www.codehunt.com/

The Story

Behind the Scene of Code Hunt behavior Secret Impl == Player Impl Player Implementation class Player { public static int Puzzle(int x) { return x; } Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } class Test { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }

Background – Code Hunt Contest Imagine Cup Annual competition hosted by Microsoft to foster software innovation Dataset under analysis is based on the Imagine Cup 2014 Code Hunt Challenge https://www.microsoft.com/en-us/research/project/code-hunt-community/ 259 participants were given 48 hours to solve 24 levels (i.e., puzzles) Puzzles are grouped into 4 different sectors, and each sector corresponds to a specific topic (e.g., loops, arithmetic)

Methodology From the dataset, collect information about each user’s performance on each of the 24 puzzles # attempts until the winning submission or last failed attempt Completion status of a level (i.e., puzzle) Programming language(s) used in the attempts The information is then analyzed with R

Findings – Difficulty Based On Completions Because there are some cases where the highest number of attempts per puzzle skews the distribution (e.g., 278 or 330 attempts), we make a cutoff of the highest number of attempts as 100. The x-axis of both charts denote Puzzles 1-24. As levels increase, % completions decreases.

Findings – Difficulty Based On Completions Cont. Sector 1 Sector 2 Sector 3 Sector 4 98% to 41% 27% to 11% 14% to 6% 8% to 3% % completions of each sector (1-4) gradually decreases.

Findings – Difficulty Based On Completions Cont. Only 40% users completed Level 3, lower than Levels 4-6.

Solution Code for Level 3 -Users averaged seventy-five attempts before their winning submission (significantly more than the other puzzles in this sector) -Analysis showed that the C# users struggled more with this level than Java users C# users spent more time and averaged more attempts perhaps because of the unfamiliarity of that language compared to Java which is more commonly taught in classes Solution is based on using bit-wise operations

Score for a Completed Level Reflect how elegant the student solution is, measured by # instructions in the compiled .NET intermediate language Score 1: the solution is much longer than all other so far submitted student solutions Score 2: the solution is about average Score 3: the solution is significantly shorter

Findings – Difficulty Based on Score of User Cont. Findings – Difficulty Based on Score of User Sector 1 Sector 2 Sector 3 Sector 4 Users tend to get score 3 In the first sector and last sector, users obtaining score 3 are especially dominating

Findings – Difficulty Based on Score of User Cont. % users obtaining score 1 is similar to % users obtaining score 3 in Levels 6, 8, 17.

Findings – Difficulty Based on Score of User Cont. Interestingly, % users containing score 2 in Level 8 is higher than % users containing score 1 or 3.

Level 8’s Partial Secret Solution int openClose = 0; int maxDepth = 0; foreach (char c in s) { if (c == '(') { openClose++; if (openClose > maxDepth) maxDepth = openClose; } else if (c == ')') { openClose--; if (openClose < 0) return 0; return (openClose == 0) ? maxDepth : 0; Users failed to write the last return statement in a succinct way as adopted in the secret solution

Language Choice among 259 Users 143 C# users Attempting more than half (12) levels with C# 7 C# users attempted with both Java and C# 113 Java users Attempting more than half (12) levels with Java 11 C# users attempted with both Java and C# 3 users using C# to attempt 12 levels and Java to attempt the other 12 levels #C# users are 20% more than #Java users!

Findings – %Completion (C# vs Java) C# users performed worse than Java users with respect to % completions Levels 3 and 17 are the only two exceptions.

Solution Code for Level 17 The difficulty primarily comes from the relatively complex logic in the secret solution requires a combination of techniques and algorithms in order to successfully complete the puzzles The solution to Puzzle 17 in Sector 3 is to emulate the movement of a knight piece in a chess game. We hypothesize that the difficulty faced by many users when solving Puzzle 17 came from requiring the use of bit-wise operations to solve the puzzle, a similar situation for Puzzle 3.

Conclusion Code Hunt being used for contests or for fun/learning by a large number of users. Data analysis conducted on Code Hunt contest data, focusing on puzzle difficulty (%completion and %score 1, 2, or 3) and C# users vs. Java users Main findings: Trend of increasing difficulty from Levels 1 to 24/from Sectors 1 to 4 as expected; some outlier cases exist (puzzles that are much more difficult than others in the same sector). Java users generally outperform C# users; Java users may be more experienced coders than C# users(?).

Thank you! Questions?

Conclusion Code Hunt being used for contests or for fun/learning by a large number of users. Data analysis conducted on Code Hunt contest data, focusing on puzzle difficulty (%completion and %score 1, 2, or 3) and C# users vs. Java users Main findings: Trend of increasing difficulty from Levels 1 to 24/from Sectors 1 to 4 as expected; some outlier cases exist (puzzles that are much more difficult than others in the same sector). Java users generally outperform C# users; Java users may be more experienced coders than C# users(?).

Motivation Code Hunt is a web-based gaming platform from Microsoft Research Popularly used for various programming contests No existing analysis to better understand how users perform in these programming contests Analysis of the users in these programming contests can help by Showing areas where Code Hunt can be improved to help students improve their coding abilities Providing evidence that Code Hunt can be used as teaching and training tool To increase the number of participants in contests Transition: To reap the benefits of analyzing users in these programming contests we analyzed the dataset collected by Microsoft Research for the 2014 Imagine Cup. To better understand how users perform on puzzles Use the gathered info to help reaction of puzzles in the Codehunt Game improve the c We hope our analysis can help show any areas where Codehunt can be improved to better help students improve their coding abilities and to increase the number of people who participate in Codehunt