1. Dimensions in Synthesis Part 2: Applications (Intelligent Tutoring Systems) Sumit Gulwani sumitg@microsoft.com Microsoft Research, Redmond May 2012

2. Domain Insight Bit-vector Algorithms Geometry Constructions Testing & Verification Symbolic verification runs in reasonable time Testing is probabilistically sound Synthesis Strategy Counter-example guided inductive synthesis Brute-force search: Generate and Test 1 Recap

3. Students and Teachers End-Users Algorithm Designers Software Developers Most Transformational Target Potential Users of Synthesis Technology 2 Most Useful Target Vision for End-users: Enable people to have (automated) personal assistants. Vision for Education: Enable every student to have access to free & high-quality education.

4. Motivation –Online learning sites: Khan academy, Edx, Udacity, Coursera Increasing class sizes with even less personal attention –New technologies: Tablets/Smartphones, NUI, Cloud Various Aspects –Solution Generation –Problem Generation –Automated Grading/Feedback –Content Entry Various Domains –K-12: Mathematics, Physics, Chemistry –Undergraduate: Introductory Programming, Automata Theory –Language Learning 3 Intelligent Tutoring Systems

5. 4 Joint work with: Cerny, Henzinger, Radhakrishna, Zufferey

6. 5 Classic Problem in Automata Theory Course Solution Generation Engine Let L be the language containing all strings over {a,b} that have the same number of occurrences of “ab” as occurrences of “ba”. Construct an automata that accepts L, or prove that L is non-regular. “Regular”,

7. 6 Classic Problem in Automata Theory Course Solution Generation Engine Let L be the language containing all strings over {a,b} that have the same number of occurrences of “a” as occurrences of “b”. Construct an automata that accepts L, or prove that L is non-regular. “Non-regular”,

8.  Formal Description of Input Regular Languages –Algorithm for automata synthesis. Non-regular Languages –Formal description of non-regularity proof. –Algorithm for proof synthesis. 7 Outline

9. Problem Description Languages Examples User-friendly logic Context-free grammar Interfaces Membership Test –Required for inductive synthesis of automata or non-regularity proof. Symbolic Membership Test –Required for verification of non-regularity proof. 8 Formal Description of Input

10. 9 User-friendly Logic

11. Formal Description of Input Regular Languages  Algorithm for automata synthesis. Non-regular Languages –Formal description of non-regularity proof. –Algorithm for proof synthesis. 10 Outline

12. 11 Automata Generation for Regular Languages

13. 12 Distribution of Automata sizes The automata size of educational problems is small!

14. 13 Distribution of counterexample lengths (relative to automata size) The counterexample lengths are quite smaller than worst-case possibility of twice the automata size.

15. Formal Description of Input Regular Languages –Algorithm for automata synthesis. Non-regular Languages  Formal description of non-regularity proof. –Algorithm for proof synthesis. 14 Outline

16. 15 Myhill-Nerode Theorem: Non-regularity Condition

17. 16 Examples of Congruence/Witness Functions

18. 17 Language for Congruence/Witness Functions

19. Formal Description of Input Regular Languages –Algorithm for automata synthesis. Non-regular Languages –Formal description of non-regularity proof.  Algorithm for proof synthesis. 18 Outline

20. 19 Algorithm for constructing Congruence/Witness Fns.

21. 20 Experimental Results IdPDLGeneration Congruence Fn.Witness Fn. TestVerify 1UFPDL50.1 2UFPDL50.20.1 3UFPDL50.90.1 3CFG50.40.1 4UFPDL51.60.10.2 5CFG100.20.1 6CFG50.1 9CFG100.1 11UFPDL812.10.40.1 12UFPDL510.10.40.2 13UFPDL56.40.20.1 15UFPDL77.81.10.1 15CFG70.1 16CFG50.20.1 17UFPDL51.60.40.1 18CFG50.30.1 19UFPDL1015.84.51.7 21CFG50.50.1

22. 21 Intelligent Tutoring Systems AAAI 2012: Singh, Gulwani, Rajamani.

23. 22 Trigonometry Problem

24. 23 Algebra Problem Generation Example Problem Query Generation Query Query Execution New Problems Query Refinement Results OK? Refined Query No Yes Similar Problems

25. 24 Limits/Series Problem

26. 25 Integration Problem

27. 26 Determinant Problem

28. 27 Intelligent Tutoring Systems Arxiv TR 2012: Rishabh Singh, Gulwani, Armando Solar-Lezama.

29. 28 Background: PexForFun

30. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 29 Buggy Program for Array Reverse 6:28::50 AM

31. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 30 Buggy Program for Array Reverse 6:32::01 AM

32. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 31 Buggy Program for Array Reverse 6:32::32 AM No change! Sign of Frustation?

33. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i <= a.Length; i--) { b[count] = a[i]; count++; } return b; } 32 Buggy Program for Array Reverse 6:33::19 AM

34. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 33 Buggy Program for Array Reverse 6:33::55 AM Same as initial attempt except Console.Writeline!

35. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 34 Buggy Program for Array Reverse 6:34::06 AM No change! Sign of Frustation?

36. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i <= a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 35 Buggy Program for Array Reverse 6:34::56 AM The student has tried this before!

37. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 36 Buggy Program for Array Reverse 6:36::24 AM Same as initial attempt!

38. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 37 Buggy Program for Array Reverse 6:37::39 AM The student has tried this before!

39. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i > 0; i--) { b[count] = a[i]; count++; } return b; } 38 Buggy Program for Array Reverse 6:38::11 AM Almost correct! (a[i-1] instead of a[i] in loop body)

40. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i >= 0; i--) { b[count] = a[i]; count++; } return b; } 39 Buggy Program for Array Reverse 6:38::44 AM Student going in wrong direction!

41. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 40 Buggy Program for Array Reverse 6:39::33 AM Back to bigger error!

42. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 41 Buggy Program for Array Reverse 6:39::45 AM No change! Frustation!

43. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 42 Buggy Program for Array Reverse 6:40::27 AM No change! More Frustation!!

44. using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 43 Buggy Program for Array Reverse 6:40::57 AM No change! Too Frustated now!!! Gives up.

45. Provides additional value over counterexample feedback. More friendly feedback. –Helpful for students who give up after several tries (with only counterexample feedback). Grading –Counterexample feedback does not distinguish between a slightly incorrect solution and one that is very far off from being correct. 44 Proposal: Semantic Grading

46. 45 Demo

47. Simplifying Assumptions Correct solution is known. Errors are predictable. Programs are small. Challenging Aspects No logical specification. –Instead program equivalence. Higher density of errors than production code. Generate multiple fixes –To remain faithful to the student’s thought process. 46 Relation with Automated Bug Fixing

48. Teacher provides: –A reference implementation –Model of errors that students make Sketch encoding: –Fuzz the program using error model. –Use a counter to keep track of number of changes. Sketch solving: –To generate minimal fixes, assert (counter = i) for increasing values of i. –Use off-the-shelf SAT solver to explore the state space of possible corrections. 47 Technique

49. Array Index Fuzzing: v[a] -> v[{a+1, a-1, v.Length-a-1}] Initialization Fuzzing: v=n -> v={n+1, n-1, 0} Increment Fuzzing: v++ -> { ++v, v--, --v } Return Value Fuzzing: return v -> return ?v Conditional Fuzzing: a op b -> a’ ops { a+1, a-1, 0 } where ops = {, =, ==, != } 48 Error Model for Array Reverse Problem

50. 49 Effectiveness of Error Models

51. 50 Efficiency of Error Models

52. 51 Generality of Error Models

53. BenchmarkTotalFixedChangesTime(s) Array Reverse3052541.732.69 String Palindrome86641.283.52 Array Maximum99681.257.47 Is Increasing Order51381.723.56 Array Sort74351.1732.46 Factorial70301.254.99 Friday Rush174112.42 52 Experimental Results

54. 53 Potential Workflow Teacher grades an ungraded answer-script. System generalizes corrections into error models. System performs automated grading by considering all possible combinations and instantiations of all error models. Any ungraded answer scripts? Yes No

55. 54 Intelligent Tutoring Systems Joint work with: Alex Polozov and Sriram Rajamani

56. State-of-the-art Mathematical Editors Text editors like Latex –Unreadable text in prefix notation WYSIWIG editors like Microsoft Word –Change of cursor positions multiple times. –Back and forth switching between mouse & keyboard Our proposal: An intelligent predictive editor. Mathematical text has low entropy and hence amenable to prediction! 55 Mathematical Intellisense

57. Terms connected by the same AC operator can be thought of as terms belonging to a sequence. There are 2 opportunities for predicting such terms. Sequence Creation: T 1, T 2, T 3, … Learn a function F such that F(T i ) = T i+1 Sequence Transformation: T 1, T 2, T 3, T 4 -> S 1, S 2, … Learn a function F such that F(T i ) = S i 56 Reducing (Term) Prediction to Learning-By-Examples

58. 57 Mathematical (Syntactic) Intellisense

59. 58 Mathematical (Semantic) Intellisense

60. Long-term Goals Ultra-intelligent computer Model of human mind Inter-stellar travel 59