Tackling Ambiguity in PBE Rishabh Singh

Programming ByExamples Intuitive Natural Accessible Ambiguity

Art $75 M Ambiguous!  Modern Art

Programming Ambiguous!

Adding Mr. to Last Name InputOutput Rajeev AlurMr. Alur Satya Nadella Harry Shum Jeanette Wing Eric Horvitz

Ambiguity InputOutput Rajeev AlurMr. Alur Satya Nadella

Prefer non-constants? InputOutput Rajeev AlurMr. Alur Satya Nadella

Prefer non-constants? InputOutput Rajeev AlurMr. Alur Satya NadellaMs. Nadella Prefer smaller substrings as constants?

Prefer smaller constants? InputOutput Satya NadellaS. Nadella Harry Shum 2 nd word, last word, 2 nd capital followed by 2 nd lowercase string….

Machine Learning for Ranking Probabilistic Semantic Transformations Techniques 2 [ Singh, Gulwani CAV 2015 ] [ Singh, Gulwani POPL 2016 ]

Machine Learning for Ranking [CAV 2015] “With great power comes great responsibility.”

FlashFill DSL VSA Program Heuristics Benchmarks

DSL VSA Program Ranking Benchmarks FlashFill

Labelled Training Data Machine Learning Algorithm Efficient Ranking Algorithm Challenges 3

Training Data Generation InputOutput Rajeev AlurMr. Alur Satya NadellaMr. Satya Harry ShumMr. Harry

Learn DAGs Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya

Intersect DAGs Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya

Assign Positive Labels Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya

Assign Negative Labels Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya

Rajeev Alur  Mr. Alur Satya Nadella  Mr. Satya Learn ranking function f(p) that ranks programs higher than programs.

Ranking Function f(p) Assume Linear Function f(p) = w 1 * f 1 + w 2 *f 2 + … + w k *f k

Learning To Rank Logistic Regression Listwise Approach Didn’t work well  Too strong a constraint All relevant pages over irrelevant

A New Loss Function Positive ProgramsNegative Programs Rank any positive program over all negative programs

Hierarchical Ranking Atomic Expression Substring Expression Concat Expression Frequency of tokens, context, neighborhood,… Length of substring, input, output, constant,… Number of Arguments, sum, max, min, prod

Evaluation 175 benchmarks train-test partition Baseline (Occam’s razor): Smallest & Simplest programs

LearnRank needs 1.48 examples vs 4.17

LearnRank needs 1 example for 79% benchmarks

LearnRank has overhead of only 25% (8ms)

Semantic Data Type Transformations [POPL 2016]

Semantic Transformations Transformations over strings that have some interpretation (e.g. Dates, Names, Phone Numbers, Addresses, Currency, Units)

Learning Robust Transformations Non-uniform Data Ambiguous Data Noisy and Missing Data 5/10/2015, /10/2015, July 2014, Feburary /21/2014, 6.21

Semantic Transformations InputOutput 08/21/2015{date: 21 st Aug}

InputOutput 08/21/2015{date: 21 st Aug} Semantic Transformations

InputOutput 08/21/2015{date: 21 st Aug} Semantic Transformations

InputOutput 08/21/2015{date: 21 st Aug}

Semantic Transformations InputOutput 08/21/2015{date: 21 st Aug}

Semantic Transformations InputOutput 08/21/2015{date: 21 st Aug} {date: 24 th Aug} {date: 5 th Jun} {date: 11 th Dec} {date: 6 th Jul}

08/21/2015  {date: 21 st Aug} (m 2 /d 1 /y 2 )  (“{date: ” d 3 “ ” m 4 “}”) Simple Rule Language

08/21/2015  {date: 21 st Aug} Probabilistic Semantics (m 2 /d 1 /y 2 )  (“{date: ” d 3 “ ” m 4 “}”) w3w3 w1w1 w2w2

Semantic Constraints InputOutput 08/21/2015{date: 21 st Aug} {date: 20 th Sep}

Approximate Matching InputOutput 08/21/2015{date: 21 st Aug} {date: 20 th Sep} {date: 6 th May} m 2 /d 1 /y 2

Joint Input Learning InputOutput 08/21/2015{date: 21 st Aug} {date: 11 th Dec} {date: 20 th Sep} {date: 6 th May} y 2 –d 1 –m 2 {date: 6 th Jul}

Probabilistic Semantics solves more benchmarks (40 vs 26) with fewer examples (1.5 vs 2.3 )

Ambiguity in PBE Machine Learning + Synthesis Probabilistic Semantic Transformations Approximate Matching & Joint Learning General Loss Function for PBE Thanks!