A Conversational Agent to Navigate in Virtual Worlds CHI 2000 Workshop on Natural Language Interfaces The Hague, The Netherlands Pierre Nugues, Christophe Godéreaux, Pierre-Olivier El Guedj, & Frédéric Revolta

Early Experiment - Shrdlu Shrdlu was written by Terry Winograd at the M.I.T. Artificial Intelligence Laboratory in Moves objects around and makes inferences about the world.

Person: IS THERE A LARGE BLOCK BEHIND A PYRAMID? Computer: YES, THREE OF THEM: A LARGE RED ONE, A LARGE GREEN CUBE, AND THE BLUE ONE. Person: PUT A SMALL ONE ONTO THE GREEN CUBE WHICH SUPPORTS A PYRAMID. Computer: OK. (does it) … Person: HOW MANY THINGS ARE ON TOP OF GREEN CUBES? Computer: I'M NOT SURE WHAT YOU MEAN BY "ON TOP OF "IN THE PHRASE "ON TOP OF GREEN CUBES ". DO YOU MEAN: 1 - DIRECTLY ON THE SURFACE 2 - ANWHERE ON TOP OF ? Person: 2 Computer: THREE OF THEM.... Person: DOES A STEEPLE Computer interrupting: SORRY, I DON'T KNOW THE WORD "STEEPLE". Person: A "STEEPLE" IS A STACK WHICH CONTAINS TWO GREEN CUBES AND A PYRAMID. Computer: I UNDERSTAND. Person: ARE THERE ANY STEEPLES NOW? Computer: NO. Person: BUILD ONE. Computer: OK. (does it)

Graphical Limitations Each command needs an interface Difficult to enumerate all the possible combinations Takes up room to display interface Some commands are hard to find In the virtual world, some commands like “Go through the door” is easier to say than to navigate with a mouse.

Ulysses

Semantics & Syntax Trained on Corpora of 400 utterances 350 words plus morphological variations Phrase-based structure modified to handle unknown and missing words Recognizes: –Commands, such as “go to the house” –deictic clarifications, such as “the yellow one” –motion modifications, such as “faster” –motion repetitions, such as “again”

Separates into five commands Go: Corresponds to a change of location with a possible rotation of the embodiment. Includes advance, enter, leave, climb etc. Return: Same as go, but the object visibility does not matter. Rotate: Corresponds to the rotation of the user’s embodiment head. Stop Continue Some verbs, like “retourne”, are still ambiguous.

Each sentence is broken down into clauses. Each clause has a subject, verb group and list of complements. Each complement has a semantic tag: time, manner, location. Each verb group has the command and possible object and adverbs.

Maps different styles of commands together. Go forward. I want to go forward. Can you go forward? I want you to go forward.

Orders clauses based on connectors and breaks down clauses into commands. Go into the house and then go to the computer. (1) Go into house (2) Go to the computer Go to the house by passing in front of the flag pole (1) Go in front of the flag pole (2) Go to the house Go back into the house (1) Return to the house (2) Go into the house

Geometric Reasoning For now objects very distinct – houses, cars, blocks etc. Give an alignment to everything that has alignment Distance is relative

Reference Resolution Uses a salience algorithm based off work of Carla Huls pdf pdf Makes a list of possible objects. Places weight based on visibility, proximity and focus. Asks for clarification if still ambiguous.

Dialogue Manager Receives a list of possible parses. Gets rid of non-executable ones. If only one command executes it. If not solicits clarification.

Turn around Look at the house ViolaOkay Go to the two cars on the left of the house Viola UserUser UlyssesUlysses

Turn to the rightGo to that car YesVoila Turn to the right Viola UserUser UlyssesUlysses

Go to the cubeGo to the small ones There are manyVoila Go behind them Viola UserUser UlyssesUlysses

Questions?