Dimensions in Synthesis Part 2: Applications (Intelligent Tutoring Systems) Sumit Gulwani Microsoft Research, Redmond May 2012

Domain Insight Bit-vector Algorithms Geometry Constructions Testing & Verification Symbolic verification runs in reasonable time Testing is probabilistically sound Synthesis Strategy Counter-example guided inductive synthesis Brute-force search: Generate and Test 1 Recap

Students and Teachers End-Users Algorithm Designers Software Developers Most Transformational Target Potential Users of Synthesis Technology 2 Most Useful Target Vision for End-users: Enable people to have (automated) personal assistants. Vision for Education: Enable every student to have access to free & high-quality education.

Motivation –Online learning sites: Khan academy, Edx, Udacity, Coursera Increasing class sizes with even less personal attention –New technologies: Tablets/Smartphones, NUI, Cloud Various Aspects –Solution Generation –Problem Generation –Automated Grading/Feedback –Content Entry Various Domains –K-12: Mathematics, Physics, Chemistry –Undergraduate: Introductory Programming, Automata Theory –Language Learning 3 Intelligent Tutoring Systems

4 Joint work with: Cerny, Henzinger, Radhakrishna, Zufferey

5 Classic Problem in Automata Theory Course Solution Generation Engine Let L be the language containing all strings over {a,b} that have the same number of occurrences of “ab” as occurrences of “ba”. Construct an automata that accepts L, or prove that L is non-regular. “Regular”,

6 Classic Problem in Automata Theory Course Solution Generation Engine Let L be the language containing all strings over {a,b} that have the same number of occurrences of “a” as occurrences of “b”. Construct an automata that accepts L, or prove that L is non-regular. “Non-regular”,

 Formal Description of Input Regular Languages –Algorithm for automata synthesis. Non-regular Languages –Formal description of non-regularity proof. –Algorithm for proof synthesis. 7 Outline

Problem Description Languages Examples User-friendly logic Context-free grammar Interfaces Membership Test –Required for inductive synthesis of automata or non-regularity proof. Symbolic Membership Test –Required for verification of non-regularity proof. 8 Formal Description of Input

9 User-friendly Logic

Formal Description of Input Regular Languages  Algorithm for automata synthesis. Non-regular Languages –Formal description of non-regularity proof. –Algorithm for proof synthesis. 10 Outline

11 Automata Generation for Regular Languages

12 Distribution of Automata sizes The automata size of educational problems is small!

13 Distribution of counterexample lengths (relative to automata size) The counterexample lengths are quite smaller than worst-case possibility of twice the automata size.

Formal Description of Input Regular Languages –Algorithm for automata synthesis. Non-regular Languages  Formal description of non-regularity proof. –Algorithm for proof synthesis. 14 Outline

15 Myhill-Nerode Theorem: Non-regularity Condition

16 Examples of Congruence/Witness Functions

17 Language for Congruence/Witness Functions

Formal Description of Input Regular Languages –Algorithm for automata synthesis. Non-regular Languages –Formal description of non-regularity proof.  Algorithm for proof synthesis. 18 Outline

19 Algorithm for constructing Congruence/Witness Fns.

20 Experimental Results IdPDLGeneration Congruence Fn.Witness Fn. TestVerify 1UFPDL50.1 2UFPDL UFPDL CFG UFPDL CFG CFG50.1 9CFG UFPDL UFPDL UFPDL UFPDL CFG CFG UFPDL CFG UFPDL CFG

21 Intelligent Tutoring Systems AAAI 2012: Singh, Gulwani, Rajamani.

22 Trigonometry Problem

23 Algebra Problem Generation Example Problem Query Generation Query Query Execution New Problems Query Refinement Results OK? Refined Query No Yes Similar Problems

24 Limits/Series Problem

25 Integration Problem

26 Determinant Problem

27 Intelligent Tutoring Systems Arxiv TR 2012: Rishabh Singh, Gulwani, Armando Solar-Lezama.

28 Background: PexForFun

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 29 Buggy Program for Array Reverse 6:28::50 AM

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 30 Buggy Program for Array Reverse 6:32::01 AM

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 31 Buggy Program for Array Reverse 6:32::32 AM No change! Sign of Frustation?

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i <= a.Length; i--) { b[count] = a[i]; count++; } return b; } 32 Buggy Program for Array Reverse 6:33::19 AM

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 33 Buggy Program for Array Reverse 6:33::55 AM Same as initial attempt except Console.Writeline!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 34 Buggy Program for Array Reverse 6:34::06 AM No change! Sign of Frustation?

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i <= a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 35 Buggy Program for Array Reverse 6:34::56 AM The student has tried this before!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 36 Buggy Program for Array Reverse 6:36::24 AM Same as initial attempt!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 37 Buggy Program for Array Reverse 6:37::39 AM The student has tried this before!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i > 0; i--) { b[count] = a[i]; count++; } return b; } 38 Buggy Program for Array Reverse 6:38::11 AM Almost correct! (a[i-1] instead of a[i] in loop body)

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i >= 0; i--) { b[count] = a[i]; count++; } return b; } 39 Buggy Program for Array Reverse 6:38::44 AM Student going in wrong direction!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 40 Buggy Program for Array Reverse 6:39::33 AM Back to bigger error!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 41 Buggy Program for Array Reverse 6:39::45 AM No change! Frustation!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 42 Buggy Program for Array Reverse 6:40::27 AM No change! More Frustation!!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 43 Buggy Program for Array Reverse 6:40::57 AM No change! Too Frustated now!!! Gives up.

Provides additional value over counterexample feedback. More friendly feedback. –Helpful for students who give up after several tries (with only counterexample feedback). Grading –Counterexample feedback does not distinguish between a slightly incorrect solution and one that is very far off from being correct. 44 Proposal: Semantic Grading

45 Demo

Simplifying Assumptions Correct solution is known. Errors are predictable. Programs are small. Challenging Aspects No logical specification. –Instead program equivalence. Higher density of errors than production code. Generate multiple fixes –To remain faithful to the student’s thought process. 46 Relation with Automated Bug Fixing

Teacher provides: –A reference implementation –Model of errors that students make Sketch encoding: –Fuzz the program using error model. –Use a counter to keep track of number of changes. Sketch solving: –To generate minimal fixes, assert (counter = i) for increasing values of i. –Use off-the-shelf SAT solver to explore the state space of possible corrections. 47 Technique

Array Index Fuzzing: v[a] -> v[{a+1, a-1, v.Length-a-1}] Initialization Fuzzing: v=n -> v={n+1, n-1, 0} Increment Fuzzing: v++ -> { ++v, v--, --v } Return Value Fuzzing: return v -> return ?v Conditional Fuzzing: a op b -> a’ ops { a+1, a-1, 0 } where ops = {, =, ==, != } 48 Error Model for Array Reverse Problem

49 Effectiveness of Error Models

50 Efficiency of Error Models

51 Generality of Error Models

BenchmarkTotalFixedChangesTime(s) Array Reverse String Palindrome Array Maximum Is Increasing Order Array Sort Factorial Friday Rush Experimental Results

53 Potential Workflow Teacher grades an ungraded answer-script. System generalizes corrections into error models. System performs automated grading by considering all possible combinations and instantiations of all error models. Any ungraded answer scripts? Yes No

54 Intelligent Tutoring Systems Joint work with: Alex Polozov and Sriram Rajamani

State-of-the-art Mathematical Editors Text editors like Latex –Unreadable text in prefix notation WYSIWIG editors like Microsoft Word –Change of cursor positions multiple times. –Back and forth switching between mouse & keyboard Our proposal: An intelligent predictive editor. Mathematical text has low entropy and hence amenable to prediction! 55 Mathematical Intellisense

Terms connected by the same AC operator can be thought of as terms belonging to a sequence. There are 2 opportunities for predicting such terms. Sequence Creation: T 1, T 2, T 3, … Learn a function F such that F(T i ) = T i+1 Sequence Transformation: T 1, T 2, T 3, T 4 -> S 1, S 2, … Learn a function F such that F(T i ) = S i 56 Reducing (Term) Prediction to Learning-By-Examples

57 Mathematical (Syntactic) Intellisense

58 Mathematical (Semantic) Intellisense

Long-term Goals Ultra-intelligent computer Model of human mind Inter-stellar travel 59