Constraint Based Synthesis for Beginners PSY 2012 Armando Solar-Lezama

CAP View of Synthesis Synthesis Methodology Code

A different approach Synthesis Methodology Code Domain Specific Tool

EXHIBIT A: SYNTHESIS OF SQL With Alvin Cheung and Sam Madden

Motivation It turns out SQL is challenging to learn  Who would have thought? Frameworks simplify program/DB interface  You can access the DB without using SQL  Which can lead to some interesting code...

Examples: Explicit Select public Set getUnfinishedProjects() { Set unfinishedP = new HashSet (); List projects = this.projectDao.getAllProjects(); for (Project project : projects) { if (!(project.getIsFinished())) { unfinishedP.add(project); } } return unfinishedP; } Get list of projects from the DB Select unfinished projects SELECT * FROM Projects WHERE isFinished=FALSE

Examples: Explicit Select public List getRoleUser(){ List listUser = new ArrayList (); List user= this.userDao.getUsers(); List role = this.roleDao.getRoles(); for(int i = 0; i < user.size(); i ++){ for(int a = 0; a < role.size(); a++){ if(user.get(i).getRole_id(). equals(role.get(a).getRole_id())){ WilosUser userok = user.get(i); listUser.add(userok); } return listUser; } Find users in the Roles list and add them to the output Start with users and roles SELECT u FROM users u, roles r WHERE u.roleId == r.id

Why is this so bad? These can be performance bottlenecks  Where performance matters, people write SQL More controversial arguments can be made

Is this a synthesis problem? It is Synthesizer Imperative code with loop nests Equivalent Formula in Relational Algebra Domain knowledge about relational algebra

Framing the problem List getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; int i = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } return results; }

Framing the problem We want to synthesize a post-condition! Challenges  Defining the language  Solving the synthesis problem  Generating code

Defining the language Requirements  Should make reasoning about the program easy  Should have expressiveness comparable to SQL

Language Just like relational algebra......but with lists rather than sets  The program is written in terms of lists, not sets

Invariants List getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; int i = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } return results; }

Verification

Synthesis Synthesis works best in a finite domain Invariant generation can be framed as a sketch 1) Model operations in terms of their source code 2) Construct the verification condition leaving invariant and post condition as unknowns 3) Create a sketch for unknowns 4) Solve!

Results so far We have tried this with a dozen loop nests from real open source projects All solve in less than 7 minutes  Slow, but not very optimized.

EXHIBIT B: AUTOMATED GRADING With Rishabh Singh and Sumit Gulwani

The real software problem The Software Quality problem is a symptom The real problem: The demand for software in our society far exceeds the supply of people skilled enough to produce it

Three pronged attack  Make programmers more productive  Make programming more accessible  Reduce the cost of training the next generation

Grading Programming Assignments Test-cases based grading  No precise correctness correlation  No student tailored feedback Manual grading by TAs  Error-prone  Time consuming  Expensive Manual grading will not scale to 100K students

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 22 Buggy Program for Array Reverse 6:28::50 AM

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 23 Buggy Program for Array Reverse 6:32::01 AM

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 24 Buggy Program for Array Reverse 6:32::32 AM No change! Sign of Frustation?

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i <= a.Length; i--) { b[count] = a[i]; count++; } return b; } 25 Buggy Program for Array Reverse 6:33::19 AM

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 26 Buggy Program for Array Reverse 6:33::55 AM Same as initial attempt except Console.Writeline!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 27 Buggy Program for Array Reverse 6:34::06 AM No change! Sign of Frustation?

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i <= a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 28 Buggy Program for Array Reverse 6:34::56 AM The student has tried this before!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 29 Buggy Program for Array Reverse 6:36::24 AM Same as initial attempt!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 30 Buggy Program for Array Reverse 6:37::39 AM The student has tried this before!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i > 0; i--) { b[count] = a[i]; count++; } return b; } 31 Buggy Program for Array Reverse 6:38::11 AM Almost correct! (a[i-1] instead of a[i] in loop body)

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i >= 0; i--) { b[count] = a[i]; count++; } return b; } 32 Buggy Program for Array Reverse 6:38::44 AM Student going in wrong direction!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 33 Buggy Program for Array Reverse 6:39::33 AM Back to bigger error!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 34 Buggy Program for Array Reverse 6:39::45 AM No change! Frustation!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 35 Buggy Program for Array Reverse 6:40::27 AM No change! More Frustation!!

using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 36 Buggy Program for Array Reverse 6:40::57 AM No change! Too Frustated now!!! Gives up.

AutoGrader Automate grading  Find semantic errors  Feedback to fix them Students make similar mistakes

Is this a synthesis problem? It is Synthesizer Buggy implementation Corrections for student program Error Model Reference Solution

Array Reverse i = 1 i <= a.Length

Challenge 1: Different Algorithms

Challenge 2: Scalability different possible candidate corrections

Our Approach

Aren’t rewrite systems hard? Can teachers really write rewrite rules? Angelic non-determinism helps  Ambiguities and redundancies no longer matter

Results: Problems Fixed

Results: Performance

Results: Generalization

Feedback for Tutoring System OK for Grading, But not ideal for teaching

Broad research agenda ahead Transformative for students in under-funded schools  Reduce the resources required to support quality instruction  Enable “true” distance education for programming courses Same technology could be used for automatic tutoring  Identify errors stemming from deep misconceptions (e.g. not understanding difference in values vs. references)  Synthesize small examples that make misconceptions explicit 48

Sketch Tutorial on Saturday