1. Syntax-Guided Synthesis Rajeev Alur Joint work with R.Bodik, G.Juniwal, M.Martin, M.Raghothaman, S.Seshia, R.Singh, A.Solar-Lezama, E.Torlak, A.Udupa 1

2. Program Verification Does a program P meet its specification where is written as a logical formula? Motivation: Correctness of systems, finding bugs Program verification is hard! Formalizing a structured program into logical formulas Using tools (SMT solvers) to verify whether the formalized program meets its specification. SMT-LIB – common standards and library of benchmarks of SMT solvers. 2

3. Program Synthesis Automatically synthesize a program P that satisfies a given specification Can potentially have greater impact than program verification Program synthesis is hard! Lets provide a syntactic template for the program – Syntax-Guided Synthesis (SyGuS) Works on special cases already exist (e.g. Sketch 2008) Lets build a common standard and benchmarks for SyGuS solvers (SYNTH-LIB) 3

4. Talk Outline Background: SMT Solvers Formalization of SyGuS Solution Strategies Conclusions + SyGuS Competition 4

5. What is SMT? SatisfiabilityModulo Theories + Magnus Madsen

6. Recall SAT literal or negated literal Magnus Madsen

7. Recall SAT SAT is NP-complete (solveable in exponential time) Many SAT solvers exist – DPLL (1962) – Chaff (2001) – MiniSAT (2004) Some do remarkably well. Magnus Madsen

8. What is an SMT instance? theory of integers theory of bitwise operators Magnus Madsen

9. Q: Why not encode every formula in SAT? A: Theory solvers have very efficient algorithms Graph Problems: Shortest-Path Minimum Spanning Tree Optimization: Max-Flow Linear Programming (just to name a few) Magnus Madsen

10. Q: But then, Why not get rid of the SAT solver? A: SAT solvers are being studied for a long time Magnus Madsen

11. SAT Theory Formula YES NO YES SMT Solver Magnus Madsen

12. Theories Theory of: – Difference Arithemetic – Linear Arithmetic – Arrays – Bit Vectors – Algebraic Datatypes – Uninterpreted Functions Magnus Madsen

13. C. Barrett & S. A. Seshia ICCAD 2009 Tutorial 13 Equivalence Checking of Program Fragments int fun1(int y) { int x, z; z = y; y = x; x = z; return x*x; } int fun2(int y) { return y*y; } What if we use SAT to check equivalence?

14. C. Barrett & S. A. Seshia ICCAD 2009 Tutorial 14 Equivalence Checking of Program Fragments int fun1(int y) { int x, z; z = y; y = x; x = z; return x*x; } int fun2(int y) { return y*y; } Using SAT to check equivalence (w/ Minisat) 32 bits for y: Did not finish in over 5 hours 16 bits for y: 37 sec. 8 bits for y: 0.5 sec.

15. C. Barrett & S. A. Seshia ICCAD 2009 Tutorial 15 Equivalence Checking of Program Fragments int fun1(int y) { int x, z; z = y; y = x; x = z; return x*x; } int fun2(int y) { return y*y; } Using EUF solver: 0.01 sec

16. Verification Synthesis 16 Program Verification: Does P meet spec ? SMT: Is satisfiable ? SMT-LIB/SMT-COMP Standard API Solver competition Program Synthesis: Find P that meets spec Syntax-Guided Synthesis Plan for SyGuS-comp

17. Talk Outline Formalization of SyGuS Solution Strategies Conclusions + SyGuS Competition 17

18. Syntax-Guided Synthesis (SyGuS) Problem Fix a background theory T: fixes types and operations Function to be synthesized: name f along with its type Inputs to SyGuS problem: Specification (semantic constraint) Typed formula using symbols in T + symbol f Set E of expressions given by a context-free grammar Set of candidate expressions that use symbols in T (syntactic constraint) Computational problem: Output e in E such that [f/e] is valid (in theory T) 18

19. SyGuS Example Theory QF-LIA Types: Integers and Booleans Logical connectives, Conditionals, and Linear arithmetic Quantifier-free formulas Function to be synthesized f (int x, int y) : int Specification: (x f(x,y)) & (y f(x,y)) & (f(x,y) = x | f(x,y) = y) Candidate Implementations: Linear expressions LinExp := x | y | Const | LinExp + LinExp | LinExp - LinExp No solution exists 19

20. SyGuS Example Theory QF-LIA Function to be synthesized: f (int x, int y) : int Specification: (x f(x,y)) & (y f(x,y)) & (f(x,y) = x | f(x,y) = y) Candidate Implementations: Conditional expressions with comparisons Term := x | y | Const | If-Then-Else (Cond, Term, Term) Cond := Term <= Term | Cond & Cond | ~ Cond | (Cond) Possible solution: If-Then-Else (x y, y, x) …. Solving SyGus is hard! 20

21. Talk Outline Solution Strategies Conclusions + SyGuS Competition 21

22. Solving SyGuS as Active Learning: 22 Learning Algorithm Verification Oracle Initial examples I Fail Success Candidate Expression Counterexample Concept class: Set E of expressions Examples: Concrete input values Counter-Example Guided Inductive Synthesis

23. CEGIS Example Specification: (x f(x,y)) & (y f(x,y)) & (f(x,y) = x | f(x,y) = y) Set E: All expressions built from x,y,0,1, Comparison, +, If-Then-Else 23 Learning Algorithm Verification Oracle Examples = { } Candidate f(x,y) = x Example (x=0, y=1)

24. CEGIS Example Specification: (x f(x,y)) & (y f(x,y)) & (f(x,y) = x | f(x,y) = y) Set E: All expressions built from x,y,0,1, Comparison, +, If-Then-Else 24 Learning Algorithm Verification Oracle Examples = {(x=0, y=1) } Candidate f(x,y) = y Example (x=1, y=0)

25. CEGIS Example Specification: (x f(x,y)) & (y f(x,y)) & (f(x,y) = x | f(x,y) = y) Set E: All expressions built from x,y,0,1, Comparison, +, If-Then-Else 25 Learning Algorithm Verification Oracle Examples = {(x=0, y=1) (x=1, y=0) (x=0, y=0) (x=1, y=1)} Candidate ITE (x y, y,x) Success

26. SyGuS Solutions CEGIS approach (Solar-Lezama, Seshia et al) Related work: Similar strategies for solving quantified formulas and invariant generation Coming up: Learning strategies based on: Enumerative (search with pruning): Udupa et al (PLDI13) Symbolic (solving constraints): Gulwani et al (PLDI11) Stochastic (probabilistic walk): Schkufza et al (ASPLOS13) 26

27. Enumerative Learning Find an expression consistent with a given set of concrete examples Enumerate expressions in increasing size, and evaluate each expression on all concrete inputs to check consistency Key optimization for efficient pruning of search space (examples): Expressions e 1 and e 2 are equivalent if e 1 (a,b)=e 2 (a,b) on all concrete values (x=a,y=b) in Examples Only one representative among equivalent subexpressions needs to be considered for building larger expressions Fast and robust for learning expressions with ~ 15 nodes 27

28. Symbolic Learning Use a constraint solver for both the synthesis and verification steps 28 Each production in the grammar is thought of as a component. Input and Output ports of every component are typed. A well-typed loop-free program comprising these component corresponds to an expression DAG from the grammar. ITE Term Cond >= Term Cond + Term x y 0 1

29. Symbolic Learning 29 x n1 x n2 y n3 y n4 0 n5 1 n6 + n7 + n8 >= n9 ITE n10 Synthesis Constraints: Shape is a DAG, Types are consistent Spec [f/e] is satisfied on every concrete input values in Examples Use an SMT solver (Z3) to find a satisfying solution. If synthesis fails, try increasing the number of occurrences of components in the library in an outer loop Start with a library consisting of some number of occurrences of each component.

30. Symbolic Learning - example Iteration 1: 30 Learned counter-example: x

31. Symbolic Learning - example Iteration 2: 31 Learned counter-example: x ITE y x x

32. Symbolic Learning - example Iteration 3: 32 Learned counter-example: - x ITE y x y

33. Stochastic Learning Idea: Use the Metropolis-Hastings Method to find desired expression e by probabilistic walk on graph where nodes are expressions and edges capture single-edits. …..in simple words: Let E n be the expressions of size n (n picked randomly). For every expression e in E n set Score(e) between 0 and 1 (Extent to which e meets the spec φ) Score(e) = exp( - 0.5 Wrong(e)), where Wrong(e) = No of examples in I for which ~ [f/e] Score(e) is large when Wrong(e) is small. Expressions e with Wrong(e) = 0 more likely to be chosen in the limit than any other expression 33

34. Initial candidate expression e sampled uniformly from E n When Score(e) = 1, return e Pick node v in parse tree of e uniformly at random. Replace subtree rooted at e with subtree of same size, sampled uniformly Stochastic Learning 34 + z e + y x + z e - 1 z With probability min{ 1, Score(e)/Score(e) }, replace e with e Repeat until finding e such that Score(e) = 1. Outer loop responsible for updating expression size n

35. Specification: (x f(x,y)) & (y f(x,y)) & (f(x,y) = x | f(x,y) = y) Set E: All expressions built from x,y,0,1, Comparison, +, If-Then-Else Stochastic Learning - example 35 Suppose n = 6 ; there are 768 expressions of size 6 e = ITE(x 0,y,x) is picked with probability 1/768 The condition x 0 is mutated to y 0 with probability 1/6 X 1/48 Suppose the set of concrete examples is: {(-1,-4),(-1, 3),(-1, 2),(1,1),(1,2)} Then Score(e)=exp(-0.5X2), and Score(e)=exp(-0.5X4) As e is replaced with e with probability exp(-0.5X2) Note that for e = ITE(x<= y,y,x) we have Score(e)=1.

36. Benchmarks and Implementation Prototype implementation of Enumerative/Symbolic/Stochastic CEGIS Benchmarks: Bit-manipulation programs from Hackers delight Integer arithmetic: Find max, search in sorted array Challenge problems such as computing Mortons number Multiple variants of each benchmark by varying grammar Results are not conclusive as implementations are unoptimized, but offers first opportunity to compare solution strategies 36

37. Evaluation: Integer Benchmarks 37

38. Evaluation 3: Hackers Delight Benchmarks 38

39. Evaluation Summary Enumerative CEGIS has best performance, and solves many benchmarks within seconds Potential problem: Synthesis of complex constants Symbolic CEGIS is unable to find answers on most benchmarks Caveat: Sketch succeeds on many of these Choice of grammar has impact on synthesis time When E is set of all possible expressions, solvers struggle None of the solvers succeed on some benchmarks Morton constants, Search in integer arrays of size > 4 Bottomline: Improving solvers is a great opportunity for research ! 39

40. Plan for SyGuS-Comp Proposed competition of SyGuS solvers at FLoC, July 2014 Organizers: Alur, Fisman (Penn) and Singh, Solar-Lezama (MIT) Website: excape.cis.upenn.edu/Synth-Comp.html Mailing list: synthlib@cis.upenn.edusynthlib@cis.upenn.edu Call for participation: Join discussion to finalize synth-lib format and competition format Contribute benchmarks Build a SyGuS solver 40