Formal Methods in Invited CBSoft Sep 2015 Sumit Gulwani Data Wrangling & Education.

Slides:



Advertisements
Similar presentations
Synthesizing Number Transformations from Input-Output Examples Rishabh Singh and Sumit Gulwani.
Advertisements

From Verification to Synthesis Sumit Gulwani Microsoft Research, Redmond August 2013 Marktoberdorf Summer School Lectures: Part 1.
Automated Grading of DFA Constructions Rajeev Alur (Penn), Loris D’Antoni (Penn), Sumit Gulwani (MSR), Bjoern Hartmann (Berkeley), Dileep Kini (UIUC),
Teaching Finite Automata with AutomataTutor Rajeev Alur (Penn), Loris D’Antoni (Penn), Sumit Gulwani (MSR), Bjoern Hartmann (Berkeley), Dileep Kini (UIUC),
(non-programmers with access to computers)
Programming Languages Marjan Sirjani 2 2. Language Design Issues Design to Run efficiently : early languages Easy to write correctly : new languages.
Dimensions in Synthesis Part 2: Applications (Intelligent Tutoring Systems) Sumit Gulwani Microsoft Research, Redmond May 2012.
FlashExtract : A General Framework for Data Extraction by Examples
Data Manipulation using Programming by Examples and Natural Language Invited Upenn April 2015 Sumit Gulwani.
April 22, Text Mining: Finding Nuggets in Mountains of Textual Data Jochen Doerre, Peter Gerstl, Roland Seiffert IBM Germany, August 1999 Presenter:
Problem Generation & Feedback Generation Invited ASSESS 2014 Workshop collocated with KDD 2014 Sumit Gulwani Microsoft Research, Redmond.
Programming Fundamentals (750113) Ch1. Problem Solving
1 ES 314 Advanced Programming Lec 2 Sept 3 Goals: Complete the discussion of problem Review of C++ Object-oriented design Arrays and pointers.
Text Mining: Finding Nuggets in Mountains of Textual Data Jochen Dijrre, Peter Gerstl, Roland Seiffert Presented by Drew DeHaas.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 16 Slide 1 User interface design.
Data Structures and Programming.  John Edgar2.
Programming by Example using Least General Generalizations Mohammad Raza, Sumit Gulwani & Natasa Milic-Frayling Microsoft Research.
Semantic Web Technologies Lecture # 2 Faculty of Computer Science, IBA.
CASE Tools And Their Effect On Software Quality Peter Geddis – pxg07u.
Introduction to High-Level Language Programming
Cultivating Research Taste (illustrated via a journey in Program Synthesis research) Programming Languages Mentoring Workshop 2015 Sumit Gulwani Microsoft.
Introduction CSE 1310 – Introduction to Computers and Programming
Copyright R. Weber Machine Learning, Data Mining ISYS370 Dr. R. Weber.
Warren He, Devdatta Akhawe, and Prateek MittalUniversity of California Berkeley This subset of the web application generates new requests to the server.
Introduction CSE 1310 – Introduction to Computers and Programming Vassilis Athitsos University of Texas at Arlington 1.
Programming by Examples Marktoberdorf Lectures August 2015 Sumit Gulwani.
Automatically Synthesizing SQL Queries from Input-Output Examples Sai Zhang University of Washington Joint work with: Yuyin Sun.
Introduction CSE 1310 – Introduction to Computers and Programming Vassilis Athitsos University of Texas at Arlington 1.
Dimensions in Synthesis Part 3: Ambiguity (Synthesis from Examples & Keywords) Sumit Gulwani Microsoft Research, Redmond May 2012.
Just as there are many human languages, there are many computer programming languages that can be used to develop software. Some are named after people,
Analysis of algorithms Analysis of algorithms is the branch of computer science that studies the performance of algorithms, especially their run time.
Major objective of this course is: Design and analysis of modern algorithms Different variants Accuracy Efficiency Comparing efficiencies Motivation thinking.
Problem Solving Techniques. Compiler n Is a computer program whose purpose is to take a description of a desired program coded in a programming language.
Unit-1 Introduction Prepared by: Prof. Harish I Rathod
December 2013 Technology for Education (T4E) Conference Sumit Gulwani Affiliations: Microsoft Research Adjunct Faculty.
Topic #1: Introduction EE 456 – Compiling Techniques Prof. Carl Sable Fall 2003.
1 Compiler Design (40-414)  Main Text Book: Compilers: Principles, Techniques & Tools, 2 nd ed., Aho, Lam, Sethi, and Ullman, 2007  Evaluation:  Midterm.
Data Structures and Algorithm Analysis Introduction Lecturer: Ligang Dong, egan Tel: , Office: SIEE Building.
Data Structures and Algorithms Dr. Tehseen Zia Assistant Professor Dept. Computer Science and IT University of Sargodha Lecture 1.
CASE (Computer-Aided Software Engineering) Tools Software that is used to support software process activities. Provides software process support by:- –
Chapter 5 Ranking with Indexes 1. 2 More Indexing Techniques n Indexing techniques:  Inverted files - best choice for most applications  Suffix trees.
Predicting a Correct Program in PBE Rishabh Singh, Microsoft Research Sumit Gulwani, Microsoft Research.
Automating String Processing in Spreadsheets using Input-Output Examples Sumit Gulwani Microsoft Research, Redmond.
Compositional Program Synthesis from Natural Language and Examples Mohammad Raza, Sumit Gulwani & Natasa Milic-Frayling Microsoft.
FlashMeta Microsoft PROSE SDK: A Framework for Inductive Program Synthesis Oleksandr Polozov University of Washington Sumit Gulwani Microsoft Research.
Introduction CSE 1310 – Introduction to Computers and Programming Vassilis Athitsos University of Texas at Arlington 1.
Onlinedeeneislam.blogspot.com1 Design and Analysis of Algorithms Slide # 1 Download From
Programming by Examples Marktoberdorf Lectures August 2015 Sumit Gulwani.
Dagstuhl Seminar Oct 2015 Sumit Gulwani Applications of Inductive Programming in Data Wrangling.
Programming by Examples applied to Data Wrangling Invited SYNT July 2015 Sumit Gulwani.
Deductive Techniques for synthesis from Inductive Specifications Dagstuhl Seminar Oct 2015 Sumit Gulwani.
CASE Tools and their Effect on Software Quality
Certifying and Synthesizing Membership Equational Proofs Patrick Lincoln (SRI) joint work with Steven Eker (SRI), Jose Meseguer (Urbana) and Grigore Rosu.
Sumit Gulwani Spreadsheet Programming using Examples Keynote at SEMS July 2016.
Sumit Gulwani Programming by Examples Applications, Algorithms & Ambiguity Resolution Keynote at IJCAR June 2016.
Advanced Algorithms Analysis and Design
Outline Core Synthesis Architecture [1 hour by Sumit]
Advanced Computer Systems
Human Computer Interaction Lecture 21,22 User Support
Programming by Examples
Usability Design Space in Programming by Examples
Programming by Examples
Unit# 8: Introduction to Computer Programming
Programming by Examples
Programming by Examples
Lecture 12: Data Wrangling
Objective of This Course
Programming Fundamentals (750113) Ch1. Problem Solving
Information Retrieval and Web Design
Presentation transcript:

Formal Methods in Invited CBSoft Sep 2015 Sumit Gulwani Data Wrangling & Education

1 The New Opportunity Software developer Traditional customer for PL community End Users Two orders of magnitude more computer users. Struggle with repetitive tasks. Formal methods can play a significant role! (in conjunction with ML, HCI)

Spreadsheet help forums

Typical help-forum interaction 300_w5_aniSh_c1_b  w5 =MID(B1,5,2) 300_w30_aniSh_c1_b  w30 =MID(B1,FIND(“_”,$B:$B)+1, FIND(“_”,REPLACE($B:$B,1,FIND(“_”,$B:$B),””))-1) =MID(B1,5,2)

Flash Fill (Excel 2013 feature) demo “Automating string processing in spreadsheets using input-output examples”; POPL 2011; Sumit Gulwani

Data locked up in silos in various formats –Flexible organization for viewing but challenging to manipulate. Wrangling workflow: Extraction, Transformation, Formatting Data scientists spend 80% of their time wrangling data. Programming by Examples (PBE) can enable easier & faster data wrangling experience. 5 Data Wrangling

To get Started! Data Science Class Assignment

FlashExtract Demo 7 “FlashExtract: A Framework for data extraction by examples”; PLDI 2014; Vu Le, Sumit Gulwani

FlashExtract

Trifacta: small, guided steps Start with: End goal: Trifacta provides a series of small transformations: From: Skills of the Agile Data Wrangler (tutorial by Hellerstein and Heer) 1. Split on “:” Delimiter 2. Delete Empty Rows 3. Fill Values Down 4. Pivot Number on Type FlashRelate Table Re-formatting

FlashRelate Demo 11 “FlashRelate: Extracting Relational Data from Semi-Structured Spreadsheets Using Examples”; PLDI 2015; Barowy, Gulwani, Hart, Zorn

Extraction FlashExtract: Extract data from text files, web pages [PLDI 2014; Powershell convertFrom-string cmdlet Transformation Flash Fill: Excel feature for Syntactic String Transformations [POPL 2011, CAV 2015] Semantic String Transformations [VLDB 2012] Number Transformations [CAV 2013] FlashNormalize: Text normalization [IJCAI 2015] Formatting FlashRelate: Extract data from spreadsheets [PLDI 2015, PLDI 2011] FlashFormat: a Powerpoint add-in [AAAI 2014] 12 PBE tools for Data Manipulation

13 Programming by Examples Example-based specification Program Search Algorithm Challenge 1: Ambiguous/under-specified intent may result in unintended programs.

Ranking –Synthesize multiple programs and rank them. 14 Dealing with Ambiguity

Rank score of a program: Weighted combination of various features. Weights are learned using machine learning. Program features Number of constants Size of constants Features over user data: Similarity of generated output (or even intermediate values) over various user inputs IsYear, Numeric Deviation, Number of characters IsPersonName 15 Ranking Scheme “Predicting a correct program in Programming by Example”; [CAV 2015] Rishabh Singh, Sumit Gulwani

FlashFill Ranking Demo 16

“It's a great concept, but it can also lead to lots of bad data. I think many users will look at a few "flash filled" cells, and just assume that it worked. … Be very careful.” 17 Need for a fall-back mechanism “most of the extracted data will be fine. But there might be exceptions that you don't notice unless you examine the results very carefully.”

Ranking –Synthesize multiple programs and rank them. User Interaction Models –Communicate actionable information to the user. 18 Dealing with Ambiguity

Make it easy to inspect output correctness –User can accordingly provide more examples Show programs –in any desired programming language; in English –Enable effective navigation between programs Computer initiated interactivity (Active learning) –Highlight less confident entries in the output. –Ask directed questions based on distinguishing inputs. 19 User Interaction Models for Ambiguity Resolution “User Interaction Models for Disambiguation in Programming by Example”, [UIST 2015] Mayer, Soares, Grechkin, Le, Marron, Polozov, Singh, Zorn, Gulwani

FlashExtract Demo (User Interaction Models) 20

21 Programming by Examples Example-based specification Program Search Algorithm Challenge 1: Ambiguous/under-specified intent may result in unintended programs. Challenge 2: Designing efficient search algorithm

22 Challenge 2: Efficient search algorithm “Spreadsheet Data Manipulation using Examples” [CACM 2012 Research Highlights] Gulwani, Harris, Singh

23 Programming by Examples Example-based specification Program Search Algorithm Challenge 1: Ambiguous/under-specified intent may result in unintended programs. Challenge 2: Designing an efficient search algorithm. Challenge 3: Lowering the barrier to design & development.

Developing a domain-specific robust search method is costly: Requires domain-specific algorithmic insights. Robust implementation requires good engineering. DSL extensions/modifications are not easy. Key Ideas: PBE algorithms employ a divide and conquer strategy, where synthesis problem for an expression F(e1,e2) is reduced to synthesis problems for sub-expressions e1 and e2. –The divide-and-conquer strategy can be refactored out. Reduction depends on the logical properties of operator F. –Operator properties can be captured in a modular manner for reuse inside other DSLs. 24 Challenge 3: Lowering the barrier

A generic search algorithm parameterized by DSL, ranking features, strategy choices. Much like parser generators SyGus [Alur et.al, FMCAD 2013] and Rosette [Torlak et.al., PLDI 2014] are great initial efforts but too general. 25 The FlashMeta Framework “FlashMeta: A Framework for Inductive Program Synthesis” [OOPSLA 2015] Alex Polozov, Sumit Gulwani

PBE technology FlashFill FlashExtractText FlashNormalize FlashExtractWeb 26 Comparison of FlashMeta with hand-tuned implementations OriginalFlashMeta N/A2.5 OriginalFlashMeta N/A1.5 Lines of Code (K) Development time (months) Running time of FlashMeta implementations vary between x of the corresponding original implementation. Faster because of some free optimizations Slower because of larger feature sets & a generalized framework

Other application domains (E.g., robotics). Integration with existing programming environments. Multi-modal intent specification using combination of Examples and NL. 27 Future directions in Programming by Examples

Vu Le Collaborators Dan Barowy Ted Hart Maxim Grechkin Alex Polozov Dileep Kini Rishabh Singh Mikael Mayer Gustavo Soares Ben Zorn

29 The New Opportunity Software developer Traditional customer for our community End Users Students & Teachers Two orders of magnitude more computer users. Struggle with repetitive tasks. Formal methods can play a significant role! (in conjunction with ML, HCI)

Repetitive tasks Problem Generation Feedback Generation Various subject domains Math, Logic Automata, Programming Language Learning 30 Intelligent Tutoring Systems [CACM 2014] “Example-based Learning in Computer-aided STEM Education”;

Motivation Problems similar to a given problem. –Avoid copyright issues –Prevent cheating in MOOCs (Unsynchronized instruction) Problems of a given difficulty level and concept usage. –Generate progressions –Generate personalized workflows Key Ideas  Test input generation techniques 31 Problem Generation

ConceptTrace CharacteristicSample Input Single digit additionL3+2 Multiple digit w/o carryLL Single carryL* (LC) L* Two single carriesL* (LC) L+ (LC) L* Double carryL* (LCLC) L* Triple carryL* (LCLCLCLC) L* Extra digit in i/p & new digit in o/pL* CLDCE Problem Generation: Addition Procedure “A Trace-based Framework for Analyzing and Synthesizing Educational Progressions” [CHI 2013] Andersen, Gulwani, Popovic.

Motivation Problems similar to a given problem. –Avoid copyright issues –Prevent cheating in MOOCs (Unsynchronized instruction) Problems of a given difficulty level and concept usage. –Generate progressions –Generate personalized workflows Key Ideas Test input generation techniques  Template-based generalization 33 Problem Generation

34 Problem Generation: Algebra (Trigonometry) AAAI 2012: “Automatically generating algebra problems”; Singh, Gulwani, Rajamani.

35 Problem Generation: Algebra (Limits)

36 Problem Generation: Algebra (Determinant)

1.The principal characterized his pupils as _________ because they were pampered and spoiled by their indulgent parents. 2.The commentator characterized the electorate as _________ because it was unpredictable and given to constantly shifting moods. (a) cosseted (b) disingenuous (c) corrosive (d) laconic (e) mercurial One of the problems is a real problem from SAT (standardized US exam), while the other one was automatically generated! From problem 1, we generate: template T 1 = * 1 characterized * 2 as * 3 because * 4 We specialize T 1 to template T 2 = * 1 characterized * 2 as mercurial because * 4 Problem 2 is an instance of T 2 Problem Generation: Sentence Completion found using web search! KDD 2014: “LaSEWeb: Automating Search Strategies Over Semi-structured Web Data”; Alex Polozov, Sumit Gulwani

Motivation Make teachers more effective. –Save them time. –Provide immediate insights on where students are struggling. Can enable rich interactive experience for students. –Generation of hints. –Pointer to simpler problems depending on kind of mistakes. Different kinds of feedback: Counterexamples 38 Feedback Generation

Motivation Make teachers more effective. –Save them time. –Provide immediate insights on where students are struggling. Can enable rich interactive experience for students. –Generation of hints. –Pointer to simpler problems depending on kind of mistakes. Different kinds of feedback: Counterexamples  Nearest correct solution 39 Feedback Generation

Feedback Synthesis: Programming (Array Reverse) i = 1 i <= a.Length --back front <= back PLDI 2013: “Automated Feedback Generation for Introductory Programming Assignments”; Singh, Gulwani, Solar-Lezama

13,365 incorrect attempts for 13 Python problems. (obtained from Introductory Programming course at MIT and its MOOC version on the EdX platform) Average time for feedback = 10 seconds Feedback generated for 64% of those attempts. Reasons for failure to generate feedback –Large number of errors –Timeout (4 min) 41 Some Results Tool accessible at:

Motivation Make teachers more effective. –Save them time. –Provide immediate insights on where students are struggling. Can enable rich interactive experience for students. –Generation of hints. –Pointer to simpler problems depending on kind of mistakes. Different kinds of feedback: Counterexamples Nearest correct solution  Strategy-level feedback 42 Feedback Generation

43 Anagram Problem: Counting Strategy Strategy: For every character in one string, count and compare the number of occurrences in another. O(n 2 ) Feedback: “Count the number of characters in each string in a pre-processing phase to amortize the cost.” Problem: Are two input strings permutations of each other?

44 Anagram Problem: Sorting Strategy Strategy: Sort and compare the two input strings. O(n 2 ) Feedback: “Instead of sorting, compare occurrences of each character.” Problem: Are two input strings permutations of each other?

45 Different implementations: Counting strategy

46 Different implementations: Sorting strategy

Teacher documents various strategies and associated feedback. –Strategies can potentially be automatically inferred from student data. Computer identifies the strategy used by a student implementation and passes on the associated feedback. –Different implementations that employ the same strategy produce the same sequence of “key values”. 47 Strategy-level Feedback Generation FSE 2014: “ Feedback Generation for Performance Problems in Introductory Programming Assignments ” Gulwani, Radicek, Zuleger

# of inspection steps # of matched implementations 48 Some Results: Documentation of teacher effort When a student implementation doesn’t match any strategy: the teacher inspects it to refine or add a (new) strategy.

Motivation Make teachers more effective. –Save them time. –Provide immediate insights on where students are struggling. Can enable rich interactive experience for students. –Generation of hints. –Pointer to simpler problems depending on kind of mistakes. Different kinds of feedback: Counterexamples Nearest correct solution Strategy-level feedback  Nearest problem description (corresponding to student solution) 49 Feedback Generation

50 Feedback Synthesis: Finite State Automata Draw a DFA that accepts: { s | ‘ab’ appears in s exactly 2 times } Grade: 6/10 Feedback: The DFA is incorrect on the string ‘ababb’ Grade: 9/10 Feedback: One more state should be made final Grade: 5/10 Feedback: The DFA accepts {s | ‘ab’ appears in s at least 2 times} Attempt 3 Attempt 1 Attempt 2 Based on nearest correct solution Based on counterexamples Based on nearest problem description IJCAI 2013: “Automated Grading of DFA Constructions”; Alur, d’Antoni, Gulwani, Kini, Viswanathan

Tool has been used at 10+ Universities. An initial case study: 800+ attempts to 6 automata problems graded by tool and 2 instructors. 95% problems graded in <6 seconds each Out of 131 attempts for one of those problems: –6 attempts: instructors were incorrect (gave full marks to an incorrect attempt) –20 attempts: instructors were inconsistent (gave different marks to syntactically equivalent attempts) –34 attempts: >= 3 point discrepancy between instructor & tool; in 20 of those, instructor agreed that tool was more fair. Instructors concluded that tool should be preferred over humans for consistency & scalability. 51 Some Results Tool accessible at:

Domain-specific natural language understanding to deal with word problems. Leverage large amounts of student data. –Repair incorrect solution using a nearest correct solution [DeduceIt/Aiken et.al./UIST 2013] –Clustering for power-grading [CodeWebs/Nguyen et.al./WWW 2014] Leverage large populations of students and teachers. –Peer-grading 52 Future Directions in Intelligent Tutoring Systems

Billions of non-programmers now have computing devices. –But they struggle with repetitive tasks. Formal methods play a significant role in developing solutions to automate repetitive tasks for the masses! –Language design, Search algorithms, Test input generation Two important applications with large scale societal impact. End-User Programming using examples: Data wrangling Intelligent Tutoring Systems: Problem & Feedback synthesis Conclusion