1. Tackling Ambiguity in PBE Rishabh Singh risin@microsoft.com

2. Programming ByExamples Intuitive Natural Accessible Ambiguity

3. 15171954 Art $75 M Ambiguous!  Modern Art

4. 19702020 Programming Ambiguous!

5. Adding Mr. to Last Name InputOutput Rajeev AlurMr. Alur Satya Nadella Harry Shum Jeanette Wing Eric Horvitz

6. Ambiguity InputOutput Rajeev AlurMr. Alur Satya Nadella

7. Prefer non-constants? InputOutput Rajeev AlurMr. Alur Satya Nadella

8. Prefer non-constants? InputOutput Rajeev AlurMr. Alur Satya NadellaMs. Nadella Prefer smaller substrings as constants?

9. Prefer smaller constants? InputOutput Satya NadellaS. Nadella Harry Shum 2 nd word, last word, 2 nd capital followed by 2 nd lowercase string….

10. Machine Learning for Ranking Probabilistic Semantic Transformations Techniques 2 [ Singh, Gulwani CAV 2015 ] [ Singh, Gulwani POPL 2016 ]

11. Machine Learning for Ranking [CAV 2015] “With great power comes great responsibility.”

12. FlashFill DSL VSA Program Heuristics Benchmarks

13. DSL VSA Program Ranking Benchmarks FlashFill

14. Labelled Training Data Machine Learning Algorithm Efficient Ranking Algorithm Challenges 3

15. Training Data Generation InputOutput Rajeev AlurMr. Alur Satya NadellaMr. Satya Harry ShumMr. Harry

16. Learn DAGs 0124 8 3 5 6 7 Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya 0124 8 3 5 6 7

17. Intersect DAGs Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya

18. Assign Positive Labels Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya

19. Assign Negative Labels Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya

20. Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya Learn ranking function f(p) that ranks programs higher than programs.

21. Ranking Function f(p) Assume Linear Function f(p) = w 1 * f 1 + w 2 *f 2 + … + w k *f k

22. Learning To Rank Logistic Regression Listwise Approach Didn’t work well  Too strong a constraint All relevant pages over irrelevant

23. A New Loss Function Positive ProgramsNegative Programs Rank any positive program over all negative programs

24. Hierarchical Ranking Atomic Expression Substring Expression Concat Expression Frequency of tokens, context, neighborhood,… Length of substring, input, output, constant,… Number of Arguments, sum, max, min, prod

25. Evaluation 175 benchmarks 30-70 train-test partition Baseline (Occam’s razor): Smallest & Simplest programs

26. LearnRank needs 1.48 examples vs 4.17

27. LearnRank needs 1 example for 79% benchmarks

28. LearnRank has overhead of only 25% (8ms)

29. Semantic Data Type Transformations [POPL 2016]

30. Semantic Transformations Transformations over strings that have some interpretation (e.g. Dates, Names, Phone Numbers, Addresses, Currency, Units)

31. Learning Robust Transformations Non-uniform Data Ambiguous Data Noisy and Missing Data 5/10/2015, 2015- -09--25 5/10/2015, 08092015 July 2014, Feburary 2015 5/21/2014, 6.21

32. Semantic Transformations InputOutput 08/21/2015{date: 21 st Aug} 2015-24-08 5.6.99 11.12.09 2015-06-07

33. InputOutput 08/21/2015{date: 21 st Aug} 2015-24-08 5.6.99 11.12.09 2015-06-07 Semantic Transformations

34. InputOutput 08/21/2015{date: 21 st Aug} 2015-24-08 5.6.99 11.12.09 2015-06-07 Semantic Transformations

35. InputOutput 08/21/2015{date: 21 st Aug} 2015-24-08 5.6.99 11.12.09 2015-06-07

36. Semantic Transformations InputOutput 08/21/2015{date: 21 st Aug} 2015-24-08 5.6.99 11.12.09 2015-06-07

37. Semantic Transformations InputOutput 08/21/2015{date: 21 st Aug} 2015-24-08{date: 24 th Aug} 5.6.99{date: 5 th Jun} 11.12.09{date: 11 th Dec} 2015-06-07{date: 6 th Jul}

38. 08/21/2015  {date: 21 st Aug} (m 2 /d 1 /y 2 )  (“{date: ” d 3 “ ” m 4 “}”) Simple Rule Language

39. 08/21/2015  {date: 21 st Aug} Probabilistic Semantics (m 2 /d 1 /y 2 )  (“{date: ” d 3 “ ” m 4 “}”) w3w3 w1w1 w2w2

40. Semantic Constraints InputOutput 08/21/2015{date: 21 st Aug} 2015-20-09 5.6.99 11.12.09 2015-06-07 {date: 20 th Sep}

41. Approximate Matching InputOutput 08/21/2015{date: 21 st Aug} 2015-20-09 5.6.99 11.12.09 2015-06-07 {date: 20 th Sep} {date: 6 th May} m 2 /d 1 /y 2

42. Joint Input Learning InputOutput 08/21/2015{date: 21 st Aug} 2015-20-09 5.6.99 11.12.09{date: 11 th Dec} 2015-06-07 {date: 20 th Sep} {date: 6 th May} y 2 –d 1 –m 2 {date: 6 th Jul}

43. Probabilistic Semantics solves more benchmarks (40 vs 26) with fewer examples (1.5 vs 2.3 )

44. Ambiguity in PBE Machine Learning + Synthesis Probabilistic Semantic Transformations Approximate Matching & Joint Learning General Loss Function for PBE Thanks! risin@microsoft.com