1. Preliminary Analysis of Contestant Performance for a Code Hunt Contest
Adrian Clark, Jonathan Wells, Angello Astorga, Andrew Xie Jalen Coleman-Lands, Tao Xie
University of Illinois at Urbana-Champaign, IL, USA

2. Code Hunt Is a serious programming game Works in C# and Java
Appeals to coders wishing to hone their programming skills
And also to students learning to code
Since its release in Spring 2014 till now, for about 3.5 years, more than 6 million (6,112,500) users
Including registered users playing on anonymous users and accounts that access api.codehunt.com directly via the documented REST APIs).
Stickiness (loyalty) is very high

3. The Story

8. Behind the Scene of Code Hunt
behavior
Secret Impl == Player Impl
Player Implementation
class Player {
public static int Puzzle(int x) {
return x; }
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1); }
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”); }

9. Background – Code Hunt Contest
Imagine Cup
Annual competition hosted by Microsoft to foster software innovation
Dataset under analysis is based on the Imagine Cup 2014 Code Hunt Challenge
259 participants were given 48 hours to solve 24 levels (i.e., puzzles)
Puzzles are grouped into 4 different sectors, and each sector corresponds to a specific topic (e.g., loops, arithmetic)

10. Methodology
From the dataset, collect information about each user’s performance on each of the 24 puzzles
# attempts until the winning submission or last failed attempt
Completion status of a level (i.e., puzzle)
Programming language(s) used in the attempts
The information is then analyzed with R

11. Findings – Difficulty Based On Completions
Because there are some cases where the highest number of attempts per puzzle skews the distribution (e.g., 278 or 330 attempts), we make a cutoff of the highest number of attempts as 100.
The x-axis of both charts denote Puzzles 1-24.
As levels increase, % completions decreases.

12. Findings – Difficulty Based On Completions
Cont.
Sector 1 Sector 2 Sector 3 Sector 4 98% to 41% 27% to 11% 14% to 6% 8% to 3%
% completions of each sector (1-4) gradually decreases.

13. Findings – Difficulty Based On Completions
Cont.
Only 40% users completed Level 3, lower than Levels 4-6.

14. Solution Code for Level 3
-Users averaged seventy-five attempts before their winning submission (significantly more than the other puzzles in this sector)
-Analysis showed that the C# users struggled more with this level than Java users
C# users spent more time and averaged more attempts perhaps because of the unfamiliarity of that language compared to Java which is more commonly taught in classes
Solution is based on using bit-wise operations

16. Score for a Completed Level
Reflect how elegant the student solution is, measured by # instructions in the compiled .NET intermediate language
Score 1: the solution is much longer than all other so far submitted student solutions
Score 2: the solution is about average
Score 3: the solution is significantly shorter

17. Findings – Difficulty Based on Score of User
Cont.
Findings – Difficulty Based on Score of User
Sector Sector Sector Sector 4
Users tend to get score 3
In the first sector and last sector, users obtaining score 3 are especially dominating

18. Findings – Difficulty Based on Score of User
Cont.
% users obtaining score 1 is similar to % users obtaining score 3 in Levels 6, 8, 17.

19. Findings – Difficulty Based on Score of User
Cont.
Interestingly, % users containing score 2 in Level 8 is higher than % users containing score 1 or 3.

20. Level 8’s Partial Secret Solution
int openClose = 0; int maxDepth = 0; foreach (char c in s) { if (c == '(') { openClose++; if (openClose > maxDepth) maxDepth = openClose; } else if (c == ')') { openClose--; if (openClose < 0) return 0; return (openClose == 0) ? maxDepth : 0;
Users failed to write the last return statement in a succinct way as adopted in the secret solution

21. Language Choice among 259 Users
143 C# users
Attempting more than half (12) levels with C#
7 C# users attempted with both Java and C#
113 Java users
Attempting more than half (12) levels with Java
11 C# users attempted with both Java and C#
3 users using C# to attempt 12 levels and Java to attempt the other 12 levels
#C# users are 20% more than #Java users!

22. Findings – %Completion (C# vs Java)
C# users performed worse than Java users with respect to % completions
Levels 3 and 17 are the only two exceptions.

23. Solution Code for Level 17
The difficulty primarily comes from the relatively complex logic in the secret solution
requires a combination of techniques and algorithms in order to successfully complete the puzzles
The solution to Puzzle 17 in Sector 3 is to emulate the movement of a knight piece in a chess game.
We hypothesize that the difficulty faced by many users when solving Puzzle 17 came from requiring the use of bit-wise operations to solve the puzzle, a similar situation for Puzzle 3.

24. Conclusion
Code Hunt being used for contests or for fun/learning by a large number of users.
Data analysis conducted on Code Hunt contest data, focusing on puzzle difficulty (%completion and %score 1, 2, or 3) and C# users vs. Java users
Main findings:
Trend of increasing difficulty from Levels 1 to 24/from Sectors 1 to 4 as expected; some outlier cases exist (puzzles that are much more difficult than others in the same sector).
Java users generally outperform C# users; Java users may be more experienced coders than C# users(?).

25. Thank you! Questions?

26. Conclusion
Code Hunt being used for contests or for fun/learning by a large number of users.
Data analysis conducted on Code Hunt contest data, focusing on puzzle difficulty (%completion and %score 1, 2, or 3) and C# users vs. Java users
Main findings:
Trend of increasing difficulty from Levels 1 to 24/from Sectors 1 to 4 as expected; some outlier cases exist (puzzles that are much more difficult than others in the same sector).
Java users generally outperform C# users; Java users may be more experienced coders than C# users(?).

27. Motivation
Code Hunt is a web-based gaming platform from Microsoft Research
Popularly used for various programming contests
No existing analysis to better understand how users perform in these programming contests
Analysis of the users in these programming contests can help by
Showing areas where Code Hunt can be improved to help students improve their coding abilities
Providing evidence that Code Hunt can be used as teaching and training tool
To increase the number of participants in contests
Transition: To reap the benefits of analyzing users in these programming contests we analyzed the dataset collected by Microsoft Research for the 2014 Imagine Cup.
To better understand how users perform on puzzles
Use the gathered info to help reaction of puzzles in the Codehunt Game improve the c
We hope our analysis can help show any areas where Codehunt can be improved to better help students improve their coding abilities and to increase the number of people who participate in Codehunt