1. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. 2. Finding the correct action ● Search for the action that maximizes an objective function: ● Each ° is an action the robot could take. (e.g., driving to a location; picking up an object) 1. Our goal: Robots that follow commands “Put the tire pallet on the truck.” Offload the metal crate from the truck. 3. Defining the objective function ● Probabilistic graphical model that maps between words in language and aspects of the external world. ● Random variables and factors created dynamically based on the structure of the natural language command. 3. Defining the objective function ● Probabilistic graphical model that maps between words in language and aspects of the external world. ● Random variables and factors created dynamically based on the structure of the natural language command. Part I: Understanding Natural Language Commands “Go to the pallet on the truck.” 4. Learning word meanings ● Learn word meanings from a parallel corpus of language paired with a log of a robot acting in the environment. ● Map between symbols in language and perceptual features of the robot's environment. 4. Learning word meanings ● Learn word meanings from a parallel corpus of language paired with a log of a robot acting in the environment. ● Map between symbols in language and perceptual features of the robot's environment. Toward Information Theoretic Human-Robot Dialog “Fly past room 124, face the windows and go up.”

2. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part II: Asking Questions “Offload the metal crate from the truck.” What does 'the truck' refer to? What does 'the metal crate' refer to? Toward Information Theoretic Human-Robot Dialog 1. Choose a question to ask. ● Ask about the most uncertain noun phrase, using entropy to assess the robot's uncertainty. ● Use the Generalized Grounding Graph framework to model the mapping between commands, questions, and answers. 2. Understand the answer. ● Merge graphs based on linguistic coreference.

3. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part III: Corpus of Commands, Questions, and Answers “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog 3. Sample Data from Corpus Command:Move your pallet further right. Question: What do the words 'your pallet' refer to? Answer: Your pallet refers to the pallet you are currently carrying. Command:Move closer to it. Question:What does the word 'it' refer to? Answer:It refers to the empty truck trailer. Command:Place the pallet just to the right of the other pallet. Question:What do the words 'the pallet' refer to? Answer:The wooden crate that the merchandise sits on top of. Command:Take the pallet and place it on the one to the left. Question:What do the words the one refer to? Answer:The one refers to the empty trailer. 3. Sample Data from Corpus Command:Move your pallet further right. Question: What do the words 'your pallet' refer to? Answer: Your pallet refers to the pallet you are currently carrying. Command:Move closer to it. Question:What does the word 'it' refer to? Answer:It refers to the empty truck trailer. Command:Place the pallet just to the right of the other pallet. Question:What do the words 'the pallet' refer to? Answer:The wooden crate that the merchandise sits on top of. Command:Take the pallet and place it on the one to the left. Question:What do the words the one refer to? Answer:The one refers to the empty trailer. 1. Data Collection ● Corpus of deliberately ambiguous commands. ● Generate questions about each concrete noun phrase in the corpus. ● Collect answers using crowdsourcing with Amazon Mechanical Turk. 2. Amazon Mechanical Turk Task Imagine that you tell a forklift operator, “Place the pallet just to the right of the other pallet.” The operator finds your command unclear and asks, “What do the words 'the pallet' refer to?” Provide an answer in your own words, based on what you see happening in the video. 1. Data Collection ● Corpus of deliberately ambiguous commands. ● Generate questions about each concrete noun phrase in the corpus. ● Collect answers on Amazon Mechanical Turk.

4. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part IV: Evaluation “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog 1. Improvement From Questions about Noun Phrases 2. Deciding when to ask a question or take an action 3. Types of Errors ● Answers that fail to disambiguate. ● Question: “What do the words 'the pallet' refer to?” Answer: “The wooden crate the merchandise sits on top of.” ● Failure to understand the answer. ● “the object on the far left.” ● Coreference failures. ● Entropy-based metric achieves higher performance, while asking fewer questions. ● Asking questions improves performance. ● Asking targeted questions improves performance more than asking random questions.

5. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part V: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog “Go to the truck” What should the robot do?

6. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part V: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog “Go to the truck” Approach: 1. Define an objective function. 2. Search for the action that maximizes the function. 3. Execute that action.

7. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part V: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog “Go to the truck”

8. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part V: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog ? More Complex Language

9. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part V: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog More Complex Language

10. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part V: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog More Complex Language ° k are groundings, or objects, places, paths, and events in the external world. Each ° k corresponds to a constituent phrase in the language input.

11. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part V: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog How do we factor the model? Factor the model according to the hierarchical and sequential syntactic structure of the natural language command.

12. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part IV: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog Factoring to Train Efficiently ● Standard approaches don't work. ● We cannot assume all the groundings are independent. – “the pallet on the truck” vs “the pallet on the ground.” ● Using Bayes' Rule and factoring requires summing over all possible sentences during training. ● Undirected models require global normalization, but we have a different partition function for different examples. ● Solution: Introduce a new correspondence variable, ©, which lets us use locally- normalized factors

13. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part IV: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog Model ● Assume that the language and ° 1... ° 2 are independent when we do not know © ● Assume a constant prior on ° 1... ° 2 Bayes' rule.

14. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part IV: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog “Put the pallet on the truck.” “Go to the pallet on the truck.”

15. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part IV: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog Training - Feature weight. - Binary-valued feature function. L-BFGS (Andrew and Gao, 2007) to fit the model to the data. Berger et al. (1996)

16. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part IV: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog

17. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part IV: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog

18. Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, Nicholas Roy What does 'the metal crate' refer to? The box pallet near the ammo pallet. Offload the metal crate from the truck. Part IV: Technical Details “Go to the pallet on the truck.” Toward Information Theoretic Human-Robot Dialog