SmartSynth: Synthesizing Smartphone Automation Scripts from Natural Language Vu Le (UC Davis) Sumit Gulwani (MSR Redmond) Zhendong Su (UC Davis)

motivation

programming App Inventor Tasker

natural language When I receive a new SMS, if the phone is connected to my car’s bluetooth, it reads the message content and replies the sender “I’m driving.”

script when (number, content) := MessageReceived() if (IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

SmartScript Script P ::= I E T C M Parameter I ::= input (i 1, …, i n ) | ε Event E ::= (r 1, …, r n ) := when Event() | ε Conversions T ::= F 1 ; …; F n ; Condition C ::= if (Π 1 ∧ … ∧ Π n ) then | ε Body M ::= Stmt 1 ; … ; Stmt n ; Conversion F ::= x := Convert(a) Predicate Π ::= Predicate (a 1, …, a n ) Statement Stmt ::= S | foreach x in a do S1; … ; Sn; od Atomic Stmt. S::= A | F Action A ::= (r 1, …, r n ) := Action (a 1, …, a n ) Argument a ::= x | i | r | l

approach Script = Components + Relations

approach Script = Components + Relations

approach Script = Components + Relations

approach Script = Components + Relations

definition o Component API: MessageReceived Entity Literal: “I’m driving” API return value: MessageReceived.Text O o Relation = E.g.,

component discovery DescriptionComponent When I receive a new SMSMessageReceived if the phone is connected to IsConnectedToBTDevice my car’s bluetoothCar_BT readsSpeak the message contentMessageReceived.Text O replies SendMessage the senderMessageReceived.Number O “I’m driving”"I'm driving"

mapping features o Regular expressions o Bag of words o Phrase length o Punctuation o Parse tree (Stanford NLP parser)

relation discovery o Rule-based relation detection Relative locations of APIs and entities Return Value or LiteralAPI Parameter Car_BTIsConnectedToBTDevice.Text I MessageReceived.Text O Speak.Text I MessageReceived.Number O SendMessage.Number I “I’m driving”SendMessage.Text I

script generation when (number, content) := MessageReceived() if (IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

ambiguity DescriptionPossible Component Mappings When I receive a new SMSMessageReceived if the phone is connected to IsConnectedToBTDevice IsConnectedToWifiNetwork IsConnectedToDataService my car’s bluetoothCar_BT readsSpeak the message contentMessageReceived.Text O replies SendMessage Send “I’m driving”"I'm driving"

incompletenes s when (number, content) := MessageReceived() if (IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

insight NLP + Program Synthesis

ambiguity { MessagedReceived, IsConnectedToBTDevice, Car_BT, Speak, MessageReceived.Text O, (SendMessage OR Send ), "I'm driving“ } Ranking Script 1 (SendMessage) SendMessag e Send s Script 1 Script 2 Synthesizer

incompletenes s o Search for most likely missing relations

q & a o Distinguishing multiple choice questions Question: API parameter Answers: Equally-likely related entities What do you want the phone to speak? A.The received message content B.“I’m driving”

architecture

evaluation o 50 scripts collected from user forums o User study: give NL descriptions from scripts 11 students participated Each student was given 25 problems 640 correct descriptions (725 total)

component mapping

relation detection

completing ratio

synthesis time

related work o General purpose programming using natural language o Natural language interfaces o Specification extraction from natural language

thank you Script = Components + Relations NLP Program Synthesis IdentifyDetect ResolveComplet e

v.s. Siri Conversation Single phone API Composition of APIs Siri SmartSynth

v.s. Tasker