1. How I Spent My Summer Vacation Grace D. Robot

2. The AAAI Robot Challenge  What  Conference attendee  Graduate Robot Attending ConferencE (GRACE)  Why  Push State-of-the-Art in Mobile Robotics and Human-Robot Interaction  Our tack  Collaboration

3. The Challenge Team 3 NRL

4. My Body  iRobot B21 base  Two Linux computers  Laser range-finder  Sonar  TRACLabs active stereo head  Monocular active camera  Flat-panel monitor

5. My Mind  Long legacy of components  Multiple, communicating processes  Simple, well-defined interfaces Get To Reg. Area Servo To Reg. Booth Stand in Line Register Give Talk Animated Face Speech Recognition Speech Under- standing Mobility Map-Based Navigation Stereo & Gesture Color Vision & Tracking Navigate To Talk

6. Architecture for Language and Gesture Processing Commands Gestures Command Interpreter Gesture Interpreter Goal Tracking Direction Processing (request, interpret, execute) Speech Output (requests for clarification, etc.) Robot Mobility CMU Detect Registration Swarthmore Detect people Metrica

7. “Find Registration” Task Implementation Given a destination to be reached, the module interleaves information gathering with direction execution. Examples of Spoken Directions: “Grace, Turn right” (opt. Gesture) “Grace, Go over there” + Gesture “Grace, Go forward five meters” “Grace, It is this way” + Gesture “Grace, Take the elevator” (opt. Dir) “Grace, It’s to your left & turn right” Features: Vision or Palm Pilot Gestures Requesting information “Grace, take the elevator” “Which floor?”

8. “Find Registration” Task Implementation - cont. All directions are associated with a destination. For each destination, directions are executed sequentially in given order. If an intermediate destination is specified, the module will be invoked recursively with that destination. Simple action commands are executed immediately or queued up. To get to the registration desk, go over there (with a gesture), take the elevator to the second floor, turn right, go forward 50 meters. Find Registration Desk Find Elevator (2 nd floor) Go over there (with a gesture). Turn right. Go forward 50 meters.

9. Human Gesture Recognition Kin / Dyn Constraint Layer: Spatial / Temporal Measurement Layer: Correlation Measurement Layer: Track Human Pose in 3 D Locate Spatially Distinct Object Track Surface Acquire Moving Object Find Strong CorrelationDetermine Motion Vector

10. Servoing to the Registration Desk  Two modes of operation:  Search for registration desk sign  Move towards registration desk  Requires coordinated movement to keep camera aligned as robot turns  Programmed using TDL AAAI Registration robot turns camera compensates

11. Standing in Line  Clustering and line fitting of laser data used to find people & walls  Personal space used to determine who is in line  Lines may bend, but not more than 90º Green cylinders are people in line, Blue are people not in line

12. Map-Based Navigation  Maps built from laser range-finder  Hähnel, Schulz & Burgard 2002  Monte Carlo Localization  Thrun, Fox, Burgard & Dellaert 2001  Navigation uses Markov Decision Process models

13. Animated Face  Festival speech generation  Facial expressions  Hierarchical Finite-State Machine Control

14. Giving the talk  Show a PowerPoint presentation  Read bullets from PowerPoint  Match keywords to semantic net node  Generate NL to describe KB subtree  Show demos where appropriate  “Behavior-based”  Implemented using parallel FSMs and circuit semantics

15. Knowledge-base design  Marker-passing semantic net  Nodes linked to behaviors  Ontology  Tasks  Techniques  Features  Bugs  Demos follow freespace navigation polly algorithm reactive steering demo freespace obstacle detection obstacle avoidance demo freespace follow freespace image heights edge detector

16. Reading a slide  Text obtained by COM interface  Each line keyword-matched to a KB node  Nodes labeled with discourse stack depth follow freespace navigation polly algorithm reactive steering demo freespace obstacle detection obstacle avoidance demo freespace follow freespace image heights edge detector 3 1 2

17. Explaining a topic  KB topologically sorted off line  Spreading activation to find relevant nodes  Each cycle, speak minimal relevant, unexplained node follow freespace navigation polly algorithm reactive steering demo freespace obstacle detection obstacle avoidance demo freespace follow freespace image heights edge detector 3 1 2 4 65 X X the-topic

18. Explaining a node  Incremental text generation  Deictic markers for tracking topic, parent task, etc.  Pattern-directed activation of speech and transition behaviors (define-speech-pattern explain-technique (precondition (and the-topic-is-a-technique? (exists? the-topic-task))) (pattern "cerebus" (vp the-topic) (infinitive the-topic-task)))) (define-transition-pattern start-parent-task (precondition …) (pattern (infinitive the-topic-parent-task) 'comma "first cerebus must“ (vp the-topic-task)))

19. Improvements this Year  Standing in line  Ability to approach line from any direction  More robustness  Autonomous crowd control  Critical to performance  Crowds interfere with functionality (e.g., navigation)  The humans did much of this last summer

20. Improvements (2)  People detection  Fusing (stereo) vision, laser, sonar, etc.  Used in many parts of the challenge  Standing in line, crowd control, schmoozing, volunteering  Elevator riding  More robustness

21. Improvements (3)  Schmoozing  Go up to people, read their badges, greet them  Look up information about them on the web and say about it (harder)  Converse (hard!)  During standing-in-line, or after registration  Volunteering  Give directions, highlight route, lead people  Door guard: check badges  Address people who don’t have badges in some manner  Be “mean” when necessary

22. Fun Stuff  Personality / looks  Wide range of expressions  Autonomous bantering  George & Gracie

23. Thank you  Ask humans for the details  Ask humans for related work  Have a nice day

24. Contact  Dani Goldbergdanig@cs(NSH 3216)  Brennan Sellner brennan+@cs  Reid Simmonsreids@csreids@cs  Grace (NSH A504)