David L. Chen Fast Online Lexicon Learning for Grounded Language Acquisition The 50th Annual Meeting of the Association for Computational Linguistics (ACL) July 9, 2012 The University of Texas at Austin Google

Navigation Task Learn to interpret and follow free-form navigation instructions – e.g. Go down this hall and make a right when you see an elevator to your left Learn by observing how humans follow instructions Assume no prior linguistic knowledge Use virtual worlds and instructor/follower data from MacMahon et al. (2006)

Sample Instructions Take your first left. Go all the way down until you hit a dead end. Go towards the coat hanger and turn left at it. Go straight down the hallway and the dead end is position 4. Walk to the hat rack. Turn left. The carpet should have green octagons. Go to the end of this alley. This is p-4. Walk forward once. Turn left. Walk forward twice. Start 3 3 H H 4 4 End

Sample Instructions 3 3 H H 4 4 Take your first left. Go all the way down until you hit a dead end. Go towards the coat hanger and turn left at it. Go straight down the hallway and the dead end is position 4. Walk to the hat rack. Turn left. The carpet should have green octagons. Go to the end of this alley. This is p-4. Walk forward once. Turn left. Walk forward twice. Observed primitive actions: Forward, Left, Forward, Forward Start End

Overall System (Chen and Mooney 2011) Learning system for parsing navigation instructions Learning system for parsing navigation instructions Observation Instruction World State Execution Module (MARCO) Instruction World State Training Testing Action Trace Navigation Plan Constructor Semantic Parser Learner Plan Refinement Semantic Parser Action Trace

Potential Navigation Plans Instruction: Turn and walk to the couch Action Trace: Left, Forward, Forward Background knowledge: Layout of the map

Potential Navigation Plans Instruction: Turn and walk to the couch Action Trace: Left, Forward, Forward Background knowledge: Layout of the map Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

Plan Refinement Turn and walk to the couch Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

Plan Refinement Face the blue hall and walk 2 steps Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

Plan Refinement Turn left. Walk forward twice. Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

Plan Refinement Find the correct subplan that corresponds to the instruction First learn the meaning of words and short phrases Use the learned lexicon to remove parts of the plans unrelated to the instructions

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch 1. As an example comes in, break down the sentence and the graph into n-grams and connected subgraphs Verify Travel Turn Verify LEFT 2 steps front: BLUE HALL BLUE HALL SOFA front: SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Connected subgraph of size 1 Connected subgraph of size 2 Turn LEFT Verify … … Turn LEFT Verify Turn Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) Turn and walk to the couch turn, and, walk, to, the, couch 1-gram 2-gram 3-gram turn and, and walk, walk to, to the, the couch turn and walk, and walk to, walk to the, to the couch … Turn LEFT Verify … … Turn LEFT Verify Turn Connected subgraph of size 1 Connected subgraph of size 2 Verify Travel Turn Verify LEFT 2 steps front : BLUE HALL BLUE HALL SOFA front : SOFA at:

Subgraph Generation Online Lexicon Learning (SGOLL) turn Turn … LEFT 2. Increase the counts and co-occurrence count of each n-gram, connected-subgraph pair. Hash the connected-subgraphs for efficient update Turn RIGHT

Subgraph Generation Online Lexicon Learning (SGOLL) turn Turn … LEFT 2. Increase the counts and co-occurrence count of each n-gram, connected-subgraph pair. Hash the connected-subgraphs for efficient update Turn RIGHT

Subgraph Generation Online Lexicon Learning (SGOLL) turn Turn … LEFT 2. Increase the counts and co-occurrence count of each n-gram, connected-subgraph pair. Hash the connected-subgraphs for efficient update Turn RIGHT

Subgraph Generation Online Lexicon Learning (SGOLL) turn Turn … LEFT 3. Rank the entries by the scoring function Turn RIGHT

Evaluation Data Statistics 3 maps, 6 instructors, 1-15 followers/instruction Hand-segmented into single sentence steps ParagraphSingle-Sentence # Instructions Avg. # sentences Avg. # words Avg. # actions

Lexicon Building Time Time in seconds Chen and Mooney (2011) SGOLL 157.3

End-to-end Execution Test how well the system can perform the overall navigation task Leave-one-map-out approach Strict metric: Only successful if the final position matches exactly Upper baselines – Training with human annotated gold plans – Complete MARCO system [MacMahon, 2006] – Humans

End-to-end Execution Single SentencesParagraphs Chen and Mooney (2011) 54.40%16.18% Chen (2012) 57.28%19.18% Gold Standard Plans 62.67%29.59% MARCO 77.87%55.69% Humans N/A69.64%

Example Parse Instruction: “Place your back against the wall of the ‘T’ intersection. Turn left. Go forward along the pink-flowered carpet hall two segments to the intersection with the brick hall. This intersection contains a hatrack. Turn left. Go forward three segments to an intersection with a bare concrete hall, passing a lamp. This is Position 5.” Parse:Turn ( ), Verify ( back: WALL ), Turn ( LEFT ), Travel ( ), Verify ( side: BRICK HALLWAY ), Turn ( LEFT ), Travel ( steps: 3 ), Verify ( side: CONCRETE HALLWAY )

Mandarin Chinese Experiment Translated all the instructions from English to Chinese Train and test in the same way Chinese does not include word boundaries (spaces) Naively segment each character Use a trained Chinese word segmenter [Chang, Galley & Manning, 2008]

Mandarin Chinese Experiment Single SentencesParagraphs Segmented by character 58.54%16.11% Segmented by Stanford segmenter 58.70%20.13%

Conclusion Presented a system that learns to interpret free-form navigation instructions by observing how humans follow instructions Assumes no prior linguistic knowledge  Able to learn from multiple languages Fast online learning makes the system more scalable

Thanks to my collaborators: Raymond J. Mooney and Lu Guo More details and data/code: Questions?