1. How can Rosie tell me what it can do for me?
Preeti Ramaraj
June 2017

2. Problem statement What does Rosie know? What can Rosie do for you?
<new>We have heard about the various capabilities of rosie especially involving being able to learn tasks and games without requiring previous knowledge. However, these instructions that they use require a particular level of skill and knowledge about Rosie that a new instructor cannot be expected to have.
Also, given the idea that Rosie learns new knowledge everyday, as a user, keeping track of what it has learnt on a daily basis as opposed to being told when asked is a cumbersome task. These point to the thought that right now, Rosie as much as it is capable of responding to mid-instruction situations, cannot answer questions on details required or be able to tell us what it already knows. Hence, my solution is two fold.</new> (improve introduction)
Can an agent reason over its internal representations in order to improve its communication abilitie?
Going back to where I started last year, here is someone like me wanting my assistant robot to perform some tasks. If you think of assistants, virtual assistants on the phone are what come to mind. But if there is one thing we have noticed, as much as it is malleable to our schedule and can suggest things based on tasks on a repeated basis, they mostly are dependent on responses that have been stored and are not created on the fly, because they weren’t built to.. Is attuned to your needs.
Some of the problems that would come up are
, doesn’t know what the robot can do. How can Rosie help this person actually perform some tasks?
Given that Rosie can learn games and tasks on the go by virtue of ITL, you do not wish to come back and be told everyday what new things it has learnt unless it is relevant to you? How can we get this sense of relevance?
Through interaction over time. If you work with someone over a period of time, there is a set of shared knowledge between us and our conversations further go with such assumption. We don’t repeat ‘The soar workshop that was conducted at the University of Michigan” every time we talk about it. Similarly, if we are referring to an event that happened at the Soar workshop, just specifying that is enough for both people to understand the reference. But how do we go about storing these things that we remember?
Given that we want all our conversations to be in words, when Rosie cannot go about performing the goal, currently, it would just say it is stuck as opposed to knowing where it went wrong? Was it in the perception, was it because of the rules? There is an additional step involved here of you having to sit and debug it, how can we prevent that?
We plan to address these focused issues using Rosie, the robot in our lab has been for the past years, been learning games and mobile-tasks on the table-top world and the mobile-world.

3. Solutions and Further Interaction capabilities Dialogue management
Answer questions regarding learnt tasks and games
Describe existing capabilities
Reason over internal representations
Dialogue management
Learning over time
Customized and improved interactions
<new>“The solution is a two-step process.
current work involves the first step, the second step is future work which I will briefly talk about at the end of this talk. The overall goal here is that verbal communication can be made most efficient leading to more natural conversations that help the robot help you achieve your goal”</new>
But this has mostly been an one-sided interaction, with instructions from the instructor. The overall goal here
For you to know this, Rosie should be able to describe hwat it already knows and be able to reason over these structures.
This would require Rosie needs to have access to its internal representations in order to be able to answer questions about already learnt games and tasks and situations that it would come across given this existing knowledge
This project is based on the premise that being able to converse with the robot makes it easier, this means the interactions must be efficient and improve over time. Using mental models, the more we learn about the user, we hope to build the common knowledge store between Rosie and the user
No matter who interacts with the robot, we want the person to be able to get going and not have to have a learning curve to interact with Rosie. And this can happen if 3

4. Answering instructor questions
Leverages existing declarative structures
Describes
concepts, failures, goals and actions in games
The world attributes in the mobile world
Rosie now has the capability of being able to answer questions about its current knowledge. It leverages the existing declarative structure that it has built based on the instructions while being taught and responds to you in English
As you can see in these examples, in a game like frogs and toads, when asked what is the goal of Frogs and toads, this answer seems reasonable.
However, let’s take the example tic-tac-toe. This is the goal as taught by the instructor. But when Rosie is asked what the goal is, after having learnt is, this is what it says. It does convey the information, but this is not probably how you or I would frame the goal, so yes, there are some steps that need to be taken in order to improve how it looks.

5. Answering instructor questions
What is the goal of frogs-and-toads?
Rosie: “The goal is that the red blocks are on the red locations and the blue blocks are on the blue locations.”
Instructor: “The goal is that all the red blocks are on the red locations and all the blue blocks are on the blue locations.”
What is the goal of tic-tac-toe?
Rosie: “The goal is that the count of the captured linear locations is 3.”
Instructor: “The goal is that three linear locations are captured.”

6. Using Referring Expressions
A referring expression is used to identify an individual object
Describe slide-block
If a block is on a location and a location is adjacent a clear location, then move a block onto a clear location. (23 words)
If a block is on a location that is adjacent a clear location, then move a block onto a clear location.(21 words)
If a block is on a location that is adjacent a clear location, then move the block onto the clear location.(21 words)
This is one way in which your conversation could get more efficient. The first sentence is the kind that would annoy oyu. And then you use something known as a referring expression.
A referring expression is used to identify an object and by that I mean, I could say “This” Laptop, “That” chair and all of these have specific reasons how they fit into a conversation. The reason I bring this up, is because in quite a few papers I read, the ease of conversation was dependent on how you used these referring expressions.
The order in which they get more relevant has no process to it, I just did it as it seemed fit, so first merging them together by specifying that is on, then using appropriate auxiliary verbs, and the next one, I changed it to near because adding a “to” was getting difficult but By the time you reach the last line, you use “the” to refer to the red block and clear location because it indicates that you are talking about a specific object and not a third object you have not mentioned yet.
(add transparency box)
clear

7. Support for interactive debugging
Previous: “I cannot detect the goal.”
Now: “I do not see a small block on a medium block.”
Goal
????
small
This access to internal representations can be used in an additionally useful way. Let us assume that Rosie is supposed to ground this goal of towers-of-Hanoi in its perception. However, this is the real world situation that is visible to Rosie. This can happen due to lighting In this case, Rosie will be able to ground the large and medium block but fails to recognize the small block, thus not being able to detect a goal. Such situations usually result in Rosie specifying that it cannot detect the goal, but not why it cannot. James and I worked together to address such non-matching criteria, and now when being encountered with this situation, Rosie will tell you that “it cannot see the small block”. Or that “It cannot see the small block on a medium block.” Given the limited understanding on the part of Rosie to understand the ambiguities that may be present in an instructor’s instruction and given that the instructor does not have access to Rosie’s nternal representations, such conversations can help the instructor and the robot reach common ground through back and forth interactions
This functionality could be further extended to understand if the instructions were not parsed correctly,
This functionality could be further used in order to debug step by step by asking what actions can you see? What is the most preferred action? Why do you prefer this action over other actions? Yes, in certain situations, it might be easier to write code for it, but when encountering issues while performing a task, this kind of interaction can go a long way in improving how we work with the robot.
medium
large 7

8. Next steps Explaining its actions and steps Hypothetical questions
What action can you perform right now? What other game has this goal?
Hypothetical questions
Can you move the red block on to the yellow location?
Being able to understand the internal workings of the robot will help us as users and instructors understand how to improve working as well as a step towards us building a model of the robot to aid interaction in the future. Additionally, if the robot is able to tell you what exactly the issue is, human intervention can help continue with the flow as opposed to being stuck
Understanding what Rosie can do next be it in a given task context or in general would be an important step for the instructor’s nderstanding of Rosie. In the context of a mid-way task or game would be very helpful in getting a step by step understanding of how Rosie works.
Similarly, this access to its internal representations, lends itself the possibility of being able to answer hypothetical questions that we can use as test cases to simulate situations before implementing it in the real world. Right now we teach simple games and tasks where you may not find such issues but in the future, definitely a more accessible way to handle corner cases
Since we did talk about back and forth interactions
clear

9. Future work Dialogue management
Answering questions based on interaction history
Building mental models of instructors
Customized interactions
Given that the overall goal was more natural conversations, it is important
Once this foundation has been built, we can proceed to build mental models of the user. This would require us to learn about the person through multiple interactions. We need to use semantic memory and episodic memory together in order to build this structure. Semantic memory to learn keywords, and tasks that this instructor would have taught Rosie and episodic memory in order to be able to answer questions based on time as well as learn preferences over time.
2. Given the nature of mental models, it is obvious that episodic memory plays a very important role. It is something that I have already begun exploring, in order to answer questions about the instructor and take that further to build the mental model structure
3. Learning from multiple people and from the same person over time, Rosie needs to learn what information can be transferred, at what point can you make inferences about the person. In the beginning, there might be too much detail in conversation but over a period of time, the conversations should be become shorter and improved based on knowledge acquired.

10. Nuggets Coal Leverages existing knowledge in Rosie
Answers questions about existing and learnt knowledge
Base for conversation capabilities that did not exist before
Does not use Natural language production rules to produce answers.
Restricted by existing knowledge structures for now
Coal
Instructor can now know what R
This ad-hoc method may restrict the type of sentences we would like to use. So, the sentences are still not as natural

11. Questions