1 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Task Learning by Instruction: Benefits and Challenges for Intelligent Interactive Systems Jim Blythe, Prateek Tandon and Mandar Tillu USC Information Sciences Institute

2 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Overview Complex tasks in intelligent assistants Overview of task learning by instruction in Tailor Integration with other task-related capabilities Challenges and new approaches in specifying tasks

3 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Handling complex tasks in an intelligent assistant Assistant UI Task models Domain models Travel Contracts Medication Regime Vacation Activities

4 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Specialized capabilities for complex tasks Assistant UI Task models Domain models Application User invocation Advice/ Preferences Explanation Monitoring Trouble-shooting Autonomy Opportunism/ Synergy …. Plan creation Learning/ Instructability Sharing tasks and goals

5 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Specialized capabilities for complex tasks Assistant UI Task models Domain models Application User invocation Advice/ Preferences Explanation Monitoring Trouble-shooting Autonomy Opportunism/ Synergy …. Plan creation Learning/ Instructability Sharing tasks and goals

6 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Modifying a procedure with Tailor: “Task Learning by Instruction” User enters text instruction, e.g. “You don’t need authorization when the cost is below $2000.” Tailor interprets by (1) recognizing type of modification, (2) mapping terms to its task and world ontology, (3) analyzing the proposed modification. To purchase equipment: 1. find two bids 2. fill out form 10C 3. request authorization from two managers 4. if authorization granted place order with purchasing To purchase equipment: 1. find two one bids 2. fill out form 10C 3. if it costs > $500 request authorization from two managers 3½. record purchase on web site 4. if authorization granted place order with purchasing {defProcedure PurchaseEquip cue: [do: (purchase $obj_id)] precond: (and (user $user) (pSelId $obj $obj_id)) body: [seq: [do: (find_bids $obj $bid)] [context: (use_form $form $obj) do: (complete_form $form $obj)] [select: (and (total_price $bid $price) (< $price 2000)) [do: (get_authorization $obj $user)]]]}

7 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Modifying a procedure with Tailor compared with learning from observation/demonstration Benefits: Challenges: Users directly describe the desired change Users describe steps without taking them Users can ignore implementation details Map user instruction to procedure knowledge using context (e.g. other intentions) Navigating a set of complex alternatives Follow-up questions to resolve ambiguity and help fix introduced problems [Blythe, IUI 05; AAAI 05]

8 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 User describes a new task e.g. “list hotels within +miles of +meeting”

9 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 The first step “List each hotel name”

10 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 0: 1:(Price $sel’n $p+),.. 2: (and (HasDisplay $sel’n $d) (Size $d $s)),.. Aggregating DP approach to find compositions of tasks ComputerTotalPrice, ShippingCost, RAM, … HasDisplay Size 0:$selection (GPA) “the selected laptop” 0:US (constant) 1:, … Vendor 0: 1:(Vendor $sel’n $v+) Country 2:(and (Vendor $sel’n $v) (Country $v US)) match DialCode User instruction: “you don’t need form B when the seller is in the US” Tailor finds matching expression: (and (vendor $selection $v) (country $v US)) Shown as: “when the country of the vendor of the selected laptop is the US” laptop company 0: 1:(HasDisplay $sel’n $d+) display country number Variable and description from GPA Object from user instruction Synonym from WordNet: seller

11 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Recent improvements: extend DP with dynamic grouping of partial matches Group partial expressions by datatype and set of instruction words matched  invariant wrt. propagation and matching Propagate groups, not individual expressions Exponential reduction in propagations, and in iterations required to find a match 0:$selection (GPA) “the selected laptop” Vendor 0: 1:(Vendor {lapt: Ø}) laptop company 0:US (constant) 1: Country 2:(and {cmpny: seller} (Country ? US)) match country $selection ε {laptop: Ø } (Vendor {laptop: Ø}) ε {company: seller } … ε {country: seller, US }

12 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 The last step and resulting procedure

13 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Overview Task learning in desktop systems Overview of task learning by instruction Integration with other task-related capabilities Challenges and new approaches in specifying tasks

14 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Challenges highlighted by Tailor prototype Users unsure of the “next step”  More context-dependent menus and wizards  Fewer options, although there are several valid alternatives Many alternative input sets for primitive procedures  Full list is overwhelming: be selective and summarize  Constraints on inputs of primitive procedures Procedure hard to understand  Combining tasks built by users – inappropriate information  Hide substeps  Hide parameters  Allow drill-down

15 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Task-related capabilities: communication and reasoning Communicating with user Invocation Explanation Advice/ preferences Reasoning Learning by instruction/ demo Autonomy Plan creation Monitoring Learning by observation Opportunism Collaboration

16 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Select modification: Add step Add input In order to find available hotels near the meeting with input a meeting: No steps currently defined Done Context-related guidance Add a step

17 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Describe step to add: In order to find available hotels near the meeting with input a meeting: get hotels OK get hotel chains with frequent flyer miles for American Options get hotel chains with frequent flyer miles for American get hotel chains with frequent flyer miles for United get hotel chains with frequent flyer miles for Delta get hotels from onlineReservationz Hiding steps and parameters Hide parameters: focus on distinct actions

18 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Describe step to add: In order to find available hotels near the meeting with input a meeting, a street and a zip: list hotel if distance < 2 OK get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list Options Heuristic: show external actions (e.g. info access), hide internal book-keeping Show steps that determine information access Highlight new inputs that are automatically suggested

19 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Describe step to add: In order to find available hotels near the meeting with input a meeting, a street and a zip: list hotel if distance < 2 OK get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list Options distance between the street and zip, and the hotel street and zip options that don’t use the hotel street and zip options that don’t use new input variables Show parameters when the user asks to see alternatives Explore only through the inputs, not how they are used Show more detail about steps when the user explores the alternatives

20 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Reduce search space using constraints on inputs Given 3 locations: YahooDistance(street1, zip1, street2, zip2): 81 alternatives Given: ‘location’ = tuple(street, zip) YahooDistance(loc1, loc2): 9 alternatives Given: output is a distance metric: symmetric, d(X,X) = 0 YahooDistance(loc1, loc2): 3 alternatives

21 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Constraints may be reused from task ontology Action Information-generating action Information-generating action: output includes a distance between two inputs A and B YahooDistance(A:loc, B:loc) -> distance_miles Symmetric: only consider one ordering constraints Identity: d(A,A) = 0, so ignore B = A. loc = compound(street, zipcode)

22 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Done get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list Output is the list of hotels Select modification: In order to find available hotels near the meeting given a meeting, a street and zip: High-level organization of outputs Add step before this Add test Remove this step Mark step as output Change step inputs Show details Output is created in list form because the constructor is embedded in an iteration

23 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Done get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list Output is the list of hotels Select modification: In order to find available hotels near the meeting with input a meeting, a street and zip: Instead of Result: simpler description and control

24 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Current work Creating primitive procedures while creating top-level procedure, not before. Using examples from previously-build procedures to select steps, bindings and follow-ups Constraints: representation, input and use

25 USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07 Summary Learning complex activities from instruction. Similar issues in modules that communicate about tasks Many potential matches for an instruction Describing learned tasks: find the ‘right’ level of initial detail and allow drill-down Utilize constraints on possible procedure inputs