1. David L. Chen Fast Online Lexicon Learning for Grounded Language Acquisition The 50th Annual Meeting of the Association for Computational Linguistics (ACL) July 9, 2012 The University of Texas at Austin Google

2. Navigation Task Learn to interpret and follow free-form navigation instructions – e.g. Go down this hall and make a right when you see an elevator to your left Learn by observing how humans follow instructions Assume no prior linguistic knowledge Use virtual worlds and instructor/follower data from MacMahon et al. (2006)

4. Sample Instructions Take your first left. Go all the way down until you hit a dead end. Go towards the coat hanger and turn left at it. Go straight down the hallway and the dead end is position 4. Walk to the hat rack. Turn left. The carpet should have green octagons. Go to the end of this alley. This is p-4. Walk forward once. Turn left. Walk forward twice. Start 3 3 H H 4 4 End

5. Sample Instructions 3 3 H H 4 4 Take your first left. Go all the way down until you hit a dead end. Go towards the coat hanger and turn left at it. Go straight down the hallway and the dead end is position 4. Walk to the hat rack. Turn left. The carpet should have green octagons. Go to the end of this alley. This is p-4. Walk forward once. Turn left. Walk forward twice. Observed primitive actions: Forward, Left, Forward, Forward Start End

6. Overall System (Chen and Mooney 2011) Learning system for parsing navigation instructions Learning system for parsing navigation instructions Observation Instruction World State Execution Module (MARCO) Instruction World State Training Testing Action Trace Navigation Plan Constructor Semantic Parser Learner Plan Refinement Semantic Parser Action Trace

7. Potential Navigation Plans Instruction: Turn and walk to the couch Action Trace: Left, Forward, Forward Background knowledge: Layout of the map

8. Potential Navigation Plans Instruction: Turn and walk to the couch Action Trace: Left, Forward, Forward Background knowledge: Layout of the map Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

9. Plan Refinement Turn and walk to the couch Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

10. Plan Refinement Face the blue hall and walk 2 steps Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

11. Plan Refinement Turn left. Walk forward twice. Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

12. Plan Refinement Find the correct subplan that corresponds to the instruction First learn the meaning of words and short phrases Use the learned lexicon to remove parts of the plans unrelated to the instructions

13. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch 1. As an example comes in, break down the sentence and the graph into n-grams and connected subgraphs Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

14. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Connected subgraph of size 1 Connected subgraph of size 2 Turn LEFT Verify … … Turn LEFT Verify Turn Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

15. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

16. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

17. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

18. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

19. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

20. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

21. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

22. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

23. Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

24. Subgraph Generation Online Lexicon Learning (SGOLL) turn Turn … LEFT 2. Increase the counts and co-occurrence count of each n-gram, connected-subgraph pair. Hash the connected-subgraphs for efficient update. 48 22 26 Turn RIGHT

25. Subgraph Generation Online Lexicon Learning (SGOLL) turn Turn … LEFT 2. Increase the counts and co-occurrence count of each n-gram, connected-subgraph pair. Hash the connected-subgraphs for efficient update. 48+1 22+1 26 Turn RIGHT

26. Subgraph Generation Online Lexicon Learning (SGOLL) turn Turn … LEFT 2. Increase the counts and co-occurrence count of each n-gram, connected-subgraph pair. Hash the connected-subgraphs for efficient update. 49 23 26 Turn RIGHT

27. Subgraph Generation Online Lexicon Learning (SGOLL) turn Turn … LEFT 3. Rank the entries by the scoring function 0.56 0.31 0.27 Turn RIGHT

28. Evaluation Data Statistics 3 maps, 6 instructors, 1-15 followers/instruction Hand-segmented into single sentence steps ParagraphSingle-Sentence # Instructions7063236 Avg. # sentences5.01.0 Avg. # words37.67.8 Avg. # actions10.42.1

29. Lexicon Building Time Time in seconds Chen and Mooney (2011) 2227.63 SGOLL 157.3

30. End-to-end Execution Test how well the system can perform the overall navigation task Leave-one-map-out approach Strict metric: Only successful if the final position matches exactly Upper baselines – Training with human annotated gold plans – Complete MARCO system [MacMahon, 2006] – Humans

31. End-to-end Execution Single SentencesParagraphs Chen and Mooney (2011) 54.40%16.18% Chen (2012) 57.28%19.18% Gold Standard Plans 62.67%29.59% MARCO 77.87%55.69% Humans N/A69.64%

33. Example Parse Instruction: “Place your back against the wall of the ‘T’ intersection. Turn left. Go forward along the pink-flowered carpet hall two segments to the intersection with the brick hall. This intersection contains a hatrack. Turn left. Go forward three segments to an intersection with a bare concrete hall, passing a lamp. This is Position 5.” Parse:Turn ( ), Verify ( back: WALL ), Turn ( LEFT ), Travel ( ), Verify ( side: BRICK HALLWAY ), Turn ( LEFT ), Travel ( steps: 3 ), Verify ( side: CONCRETE HALLWAY )

34. Mandarin Chinese Experiment Translated all the instructions from English to Chinese Train and test in the same way Chinese does not include word boundaries (spaces) Naively segment each character Use a trained Chinese word segmenter [Chang, Galley & Manning, 2008]

35. Mandarin Chinese Experiment Single SentencesParagraphs Segmented by character 58.54%16.11% Segmented by Stanford segmenter 58.70%20.13%

36. Conclusion Presented a system that learns to interpret free-form navigation instructions by observing how humans follow instructions Assumes no prior linguistic knowledge  Able to learn from multiple languages Fast online learning makes the system more scalable

37. Thanks to my collaborators: Raymond J. Mooney and Lu Guo More details and data/code: http://www.cs.utexas.edu/~ml/clamp/navigation/ Questions?