1. Understanding Agent Knowledge through Conversation
Preeti Ramaraj
My name is Preeti Ramaraj and I. I am working on being able to understand agent knowledge through conversation.

2. Motivation Leverage existing knowledge
Learn nuances of tasks that have been performed
Here comes in one of the places where it can be handy for the robot to explain itself. Given that Rosie currently knows to perform a given set of tasks and play games, I’m working on getting Rosie to be able to share this knowledge n being asked specific questions. This mechanism could also be helpful when you wish to know nuances of a task that the robot has performed before. For instance, I might want to learn where Rosie had been last, where did it last see a given object and also confirm that it knows the right definition of actions that it is aware of.
Let us assume we have a robot that is capable of doing certain tasks and learning new tasks as well. Here is a person who has never interacted with the robot before. How do they start interacting with the robot and leverage this inbuilt knowledge? 1

3. Categories Features and relations Primitive actions Tasks Games
“Is block A bigger than block B?”, “Where is the red triangle?”
Primitive actions
“What is remember?”, “Can you pick up a location?”
Tasks
“If I ask you to deliver this stapler, what are the steps you would take?”
Games
“What actions are possible in Tower of Hanoi?
Concepts
“What is three-in-a-row in Tic-Tac-Toe?”, “Can I place an X on this location that is not clear?”
Experiences
“What object did you last deliver?”, “When did you last lose at Tic-Tac-Toe?”
Given this problem statement, The kind of questions that I would like to glean from the robot broadly fall under the following categories. These categories aren’t exhaustive or strict since the questions you can ask can span multiple categories, but I’ll still try combining them in a given sense.
Given an environment, you could ask the robot about features of objects in the environment and the relations between them. So, essentially you are gaining all the properties of an object such as shape, size, colour as well as location.
Primitive actions are single step actions that involve verbs that it is aware of ,such as put down and pick up. it also includes deliberately interacting and storing things in semantic memory, “What was the answer to the question I had asked before”
Tasks are multiple primitive actions put together. To reach a particular goal. So you might have questions about sections of the task or the task as a whole such as what is the goal for the task, or what is the starting location of the task. You might also want to know what the task is meant to do so you could also ask to describe the task so so you might want to know what all the task entails since the same word can involve multiple meanings.
What are the steps that are involved, what primitive actions it is composed up?
Games would include questions involving initial and final states, actions that are possible as a part of the given game, number of players- final states?
Concepts would come into place in situations where Rosie is aware of the given definition in some form and is able to explain it to to you. where the agent needs to be aware of things such is three-in-a-row means the person has won. The tobot should be able to tell you that remember means it can use this answer in the future.
And at last, we have experiences. Experiences are questions w.r.t what the agent has done before. So questions like what object , or when did you last lose, where did you last see the stapler would be questions that come under experiences and that involve the episodic memory as well. 2

4. Current working example
Instructor - What is on the stove? Rosie – A medium green rectangle Instructor – What is the color of the block on the stove? Rosie – The color is green. Instructor – What shape is the object on the stove? Rosie – The shape is rectangle.
Block
Color: Green
Shape: Rectangle
Size: Medium
I currently have a working implementation based on the features and relations category implemented on the Rosie simulator with an internal defined world. This is one of the objects in this world, where it is a block with those properties on a stove. So if I asked these questions, it can parse the question, query the world and give me an appropriate answer.
Stove 3

5. Looking ahead Explain how to play a game step-by-step
Handle hypothetical scenarios
“What would you do if you don’t find the location?”
Complete description given a partial input
So what kinds of conversation do I see Rosie having in the future?
Currently Rosie is aware of steps that need to be taken in a game given certain conditions. Being able to explain a complete game step-by step would involve Rosie knowing the rules, illegal moves and logical steps to the game including the goal state and explaining it step by step. It would also be interesting to see if Rosie is capable of answering questions about a game such as “Is four aces a good combination for the game rummy?”
There are multiple scenarios to a given task/game. For eg, what would you do if you don’t find a location? This situation the robot probably would handle at a given time but on being asked about such a situation, Rosie should be able to tell you what it would do if the procedure were to vary from the happy path. It is a way of getting Rosie to look at the information it has in multiple ways. This would include illegal moves and scenarios that move away from the ideal scenario. For eg, what if it drops the item it has to deliver? I would pick it up again.
Also, there might be situations where you are trying to explain to it an action/description it is already aware of but in different words. It should be able to parse what you said and be able to suggest what it is aware of. Or if you are describing an object and you missed telling it what shape it is and it is not aware of this shape, it should be able to ask you to mention that as well.
By working towards this, I hope to create a system that is capable of having an intelligible conversation with the instructor based on its existing knowledge and knowledge it can acquire. 4

6. Questions? 5