Code Hunt: Experience with Coding Contests at Scale Judith Bishop, R Nigel Horspool, Tao Xie, Nikolai Tillmann, Jonathan de Halleux Microsoft Research, University of Victoria and UIUC

Working and learning for fun Enjoyment adds to long term retention on a task Discovery is a powerful driver, contrasting with direct instructions Gaming joins these two, and is hugely popular Can we add these elements to coding? Code Hunt can!

Websites Game Project Community Data Release Blogs research.microsoft.com/codehunt research.microsoft.com/codehuntcommunity github.com/microsoft/code-hunt Linked on the Project page

The Code Hunt Game

Code Hunt Is a serious programming game Works in C# and Java (Python coming) Appeals to coders wishing to hone their programming skills And also to students learning to code Code Hunt has had over 300,000 users since launching in March 2014 with around 1,000 users a day Stickiness (loyalty) is very high

The Story

Gameplay 1.User writes code in browser 2.Cloud analyzes code – test cases show differences As long as there are differences: User must adapt code, repeat When they are no more differences: User wins level! secret code test cases

It’s a game! 1.iterative gameplay 2.adaptive 3.personalized 4.no cheating 5.clear winning criterion Score is based on how many puzzles solved, how well solved, and when solved secret code test cases

void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } a.Length>0 a[0]==123… T F T F F a==null T Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]== Input null {} {0} {123…} Execute&Monitor Solve Choose next path Observed constraints a==null a!=null && !(a.Length>0) a==null && a.Length>0 && a[0]!= a==null && a.Length>0 && a[0]== Done: There is no path left. Dynamic Symbolic Execution

Available in Visual Studio since 2010 (as Pex and Smart Unit Tests) Powered by a IntelliTest in Visual Studio 2015 Nikolai Tillmann, Jonathan de Halleux, Tao Xie: Transferring an automated test generation tool to practice: from pex to fakes and code digger. ASE 2014: Jonathan de HalleuxTao XieASE 2014

Code Hunt Contests

Code Hunt PropertiesCode Hunt Contests Browser-basedHave world-wide reach on all platforms Automatic gradingAre cost effective Cloud based hostingScale to 100,000s players Clear scoring criteriaDetermine winners precisely Results availableCan identify top coders The game is fun, fresh and different.

The ah-ha moment Code Hunt can identify top coders

30 Contests in the past year

What the player sees

Dashboard and Leaderboard Publically visible when the contest ends Visible only to the organizer

Puzzles and Analytics

Puzzles About 400 puzzles Classified and rated by difficulty Puzzle Bank is highly protected PuzzleGroupDescriptionSource Difficult y Calculat edPlayers P249-1numbersCompute 23*X-42NH21335 P277stringConvert N to base 4 stringNH23332 P115arrayConvert array of scores to letter gradesAPCS32329 P002arrayCount occurrences of a string in an arrayAPCS12325 P260-1boolsCompute x && (y || !z)NH12325 Number: P067 Group: Arrays Description: Remove duplicates from an array Source: NH Difficulty: 2

Puzzle Bank Statistics Modify the Calculated rating based on user experience. D = a + tries * b + tries * c * distance where a, b and c are constants with values a = 1 b = 0.02c = 0.05 tries is the average attempts for all players who solved the puzzle distance is the number of levels solved so far. Tries does depend on Mix of students who enter The internet speed Whether students use an IDE in addition

Laser, Sept 2014 Original Difficulty Average Calculated Difficulty s Total Try Count Average Try Count Max Try Count Total Solved Users

How players perceive difficulty Beauty of ProgrammingSubjective difficulty Perceived difficulty Players who started Qualification Preliminary A Preliminary B Semi-Final CSTA and TEALSSubjective difficulty Calculated difficulty Players who scored Students Teachers

Code Hunt - the APCS (default) Zone Opened in March problems covering the Advanced Placement Computer Science course By August 2014, over 45,000 users started.

Effect of difficulty on drop off in sectors 1- 3 Yellow – Division Blue – Operators Green - Sectors

Aug 2014 and Feb 2015 PuzzleLevelAugFeb-A Compute -X Compute 4 / X Compute X-Y Compute X/Y Compute X% Compute 10%X Construct a list of numbers 0..N Construct a list of multiples of N Compute x^y Compute X! the factorial of X Compute sum of i*(i+1)/

One week 3-10 October, 2014 One year May 2014 – April 2015

Survey results – before and after contest season How much did the puzzle aspect of Code Hunt keep you interested in reaching a solution? Till Sept 2014 Sept 2014 – May 2015

Survey Results cont. In your opinion, were your final solutions well-structured code? Till Sept 2014 Sept 2014 – May 2015

Gender and Code Hunt FemaleMale Respondents12%80% Played as practice for a course in C# 11%18% Played as practice for a course in Java 40%23% Played for own enjoyment 47%57% Open survey with 850 respondents

Beyond contests

Towards a Course Experience

Total Try Count Average Try Count Max Try Count Total Solved Users Public Data release in open source For ImCupSept 257 users x 24 puzzles x approx. 10 tries = about 13,000 programs For experimentation on how people program and reach solutions Github.com/microsoft/code-hunt

Upcoming events PLOOC 2015 PLOOC 2015 at PLDI 2015, June , Portland, OR, USA PLDI 2015 CHESE 2015CHESE 2015 at ISSTA 2015, July 14, 2015, Baltimore, MD, USAISSTA 2015 Worldwide intern and summer school contests Public Code Hunt Contests are over for the summer Special ICSE attendees Contest. Register at aka.ms/ICSE2015 Code Hunt Workshop February 2015

Summary: Code Hunt: A Game for Coding 1.Powerful and versatile platform for coding as a game 2.Unique in working from unit tests not specifications 3.Contest experience fun and robust 4.Large contest numbers with public data sets from cloud data Enables testing of hypotheses and making conclusions about how players are mastering coding, and what holds them up 5.Has potential to be a teaching platform collaborators needed

Websites Game Project Community Data Release Blogs Office Mix ICSE Contest research.microsoft.com/codehunt research.microsoft.com/codehuntcommunity github.com/microsoft/code-hunt Linked on the Project page mix.office.com aka.ms/icse2015 to register Aka.ms/icse2015codehunt to play Grazie!

Appendix

Results and rewards Detailed results are presented Ranking is always based on score, not on time or attempts Score is based on how many puzzles solved, how well solved, and when solved Special rewards for contests

Comparing Code Hunt Code Hunt A serious online coding game Built on symbolic execution Addressing various audiences Data available in the cloud IPCC Annual programming contest Scored by unit testing Addressing university students Questions available on the web

BoP China 2014