1. Tutorial Pex4Fun: Teaching and Learning Computer Science via Social Gaming http://pex4fun.com Nikolai Tillmann, Jonathan de Halleux, Judith Bishop, Michal Moskal Microsoft Research Tao Xie NCSU http://bit.ly/ioa4qf

2. Pex4Fun: Teaching and Learning Computer Science via Social Gaming http://pex4fun.com Agenda Background: Code Analysis by Dynamic Symbolic Execution Writing Code in a Browser Coding Duels Social Experience Teaching Platform Conclusion

3. DYNAMIC SYMBOLIC EXECUTION IN THE CLOUD Pex – The Pex4Fun Engine – History

4. Before the cloud… http://research.microsoft.com/pex/

6. http://pex4fun.com/CoverMe

7. void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } a.Length>0 a[0]==123… T F T F F a==null T Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==123456890 Input null {} {0} {123…} Execute&Monitor Solve Choose next path Observed constraints a==null a!=null && !(a.Length>0) a==null && a.Length>0 && a[0]!=1234567890 a==null && a.Length>0 && a[0]==1234567890  Generates test data systematically Done: There is no path left. Dynamic Symbolic Execution in Pex http://pex4fun.com/HowDoesPexWork

8. Dynamic Symbolic Execution Exercises ArrayIndexLength Pex knows about all implicit, exception- throwing control-flow branches ArrayHeap Pex models the heap Assert, Assert123 Assertions connect code coverage and correctness http://pex4fun.com/DynamicSymbolicExecutionExercises

9. Limitations 16K characters of code (single file / editor window) * Single-threaded only No environment interactions * No non-determinism * Try to avoid floating-point computations * Limitation only applies to pex4fun.com, but not Pex tool

10. Dynamic Symbolic Execution Summary “Ask Pex” sends code to cloud Code is compiled and analyzed in cloud Dynamic Symbolic Execution automatically finds relevant interesting test inputs that achieve high code coverage Results are shown in browser

11. WRITING CODE IN A BROWSER Code Auto-Completion

12. Auto-Completion

13. Writing Code in Browser Exercise 1.Go to http://pex4fun.comhttp://pex4fun.com 2.Click “New” 3.Write some code with public static “Puzzle” method using System; class Vector { int x, y; public Vector(int x, int y) { this.x = x; this.y = y; } public int Length() { return (int)Math.Sqrt(x*x+y*y); } public Vector Normalize() { return new Vector(x/Length(), y/Length()); } public bool Equals(Vector other) { return this.x==other.x && this.y == other.y; } } public class Program { public static void Puzzle(Vector v) { var n1 = v.Normalize(); var n2 = n1.Normalize(); if (!n1.Equals(n2)) throw new Exception(“bug in normalize”); } } 4.“Ask Pex!” http://bit.ly/lOgPZJ http://pex4fun.com/ParameterizedUnitTesting

14. Writing Code in Browser Summary Code is compiled in cloud Code is executed in cloud Auto-completion via cloud/Javascript in browser

15. CODING DUELS Fun and Engaging Serious Game – Win by Writing Code

16. Coding Duels

17. Pex computes “semantic diff” in cloud  code written in browser vs.  secret reference implementation You win when there are no differences secret

18. Coding Duels class VisibleProgram { public static int Puzzle1(int x) { return x; } class SecretProgram { public static int Puzzle2(int x) { return x * x; } public class MetaProgram { public static void Puzzle(int x) { if (VisibleProgram.Puzzle1(x) != SecretProgram.Puzzle2(x)) throw new System.Exception("mismatch!"); } class VisibleProgram { public static int Puzzle1(int x) { return x; } class SecretProgram { public static int Puzzle2(int x) { return x * x; } public class MetaProgram { public static void Puzzle(int x) { if (VisibleProgram.Puzzle1(x) != SecretProgram.Puzzle2(x)) throw new System.Exception("mismatch!"); }

19. Coding Duels Teaching Debugging/Problem Solving Skills From Hypothesis to Theorem – Step 1: Inference of hypothesized (delta) requirement When delta: Inference of failure-inducing conditions If difficult: forced generation of new data points – By adding data-point-fitting conditionals – Step 2: Writing of requirement-satisfying solution When delta: – Derivation of hypothesized faulty line(s) of code – Derivation of patches on the faculty line(s) of code – Step 3: “Ask Pex” to validate the hypothesis Initial almost-complete (delta) working implementation helps teach debugging skills

20. Coding Duels Teaching Abstraction Skills Step 1: Inference of hypothesized (delta) requirement from observed data points – Pattern recognition – Generalization from data points to formula – Inversion of constraint solving

21. Teaching Programming, Algorithm, and Program Understanding Skills Step 2: Writing of requirement-satisfying solution Many choices, e.g. recursion vs. loop implementation Teacher can give guidance: Providing hints in comments Providing initial almost-complete working implementation Teacher can give guidance: Providing hints in comments Providing initial almost-complete working implementation

22. Coding Duels Exercise 1.Go to http://pex4fun.com; optionally: sign in with Windows Live IDhttp://pex4fun.com 2.Click “New” 3.Write secret implementation with public static “Puzzle” method using System; public class Program { public static int Puzzle(int x, int y) { return x + y – 42; } } 4.“Ask Pex!” 5.Enter Coding Duel Name 6.Click “Turn This Puzzle Into A Coding Duel” 7.Open Permalink 8.Edit Description 9.Click “Permalink” 10.Use it, or create ulta-short version, e.g. with http://bit.lyhttp://bit.ly http://pex4fun.com/CreatingAndPublishingCodingDuels

23. Coding Duels Fun and Engaging Iterative gameplay Adaptive Personalized No cheating Clear winning criterion

24. Coding Duels Summary Coding Duel: write function that matches specification given by another implementation Semantic equivalence checked in cloud Student’s tasks: – Inference of hypothesized (delta) requirement – Writing of requirement-satisfying solution http://pex4fun.com/CreatingAndPublishingCodingDuels http://pex4fun.com/TipsAndTricksForGreatDuels

25. SOCIAL EXPERIENCE Connecting Students – Competitive yet Self-Paced Environment

26. Social Experience

27. Social Experience Summary Community High score lists, leaderboard Live feed http://pex4fun.com/Community.aspx http://pex4fun.com/Livefeed.aspx

28. PEX4FUN, A TEACHING PLATFORM Pages – Courses – Automated Assessments

29. Teaching

30. Teaching Summary How to become a teacher: – Sign in, choose nickname – Send us email with nickname Teachers can – Create courses – Reuse existing or author existing pages – Get automated assessments of students – See students code http://pex4fun.com/TeachingWithPexForFun

31. Outlook I Analyzing code submissions – Detecting duplicate submissions – Ranking submissions (how correct, how beautiful) – If faulty, give user guidance on “the next step”.

32. Outlook II Better code development experience – Debugging – Dynamic input/output table – More languages Beyond the web – Phone

33. More Reading Nikolai Tillmann, Jonathan de Halleux, and Tao Xie Pex for Fun: Engineering an Automated Testing Tool for Serious Games in Computer Science Microsoft Research, Technical report MSR-TR-2011-41, March 2011 http://research.microsoft.com/apps/pubs/?id=147143

34. Conclusion Taking programming into the browser, cloud –.NET: C# (with Intellisense), Visual Basic, F# – Execution, Code analysis with Pex Fun learning experience with Coding Duels Social experience: Live feed, sharing duels Teaching with automatic grading Create your own courses! http://pex4fun.com