1. Probabilistic Control of Human Robot Interaction: Experiments with a Robotic Assistant for Nursing Homes Joelle Pineau Michael Montemerlo Martha Pollack * Nicholas Roy Sebastian Thrun Carnegie Mellon University * University of Michigan

2. Joelle Pineau The Nursebot Project Introducing Pearl – A mobile robotic assistant for elderly people and nurses cameras sonars handle bars mobile base carrying tray LCD mouth touchscreen microphone & speakers laser ROLE: Moving things around Moving things around Management support of ADLs Management support of ADLs Providing physical assistance Providing physical assistance Remote health services Remote health services Supporting communication Supporting communication Calling for help in emergencies Calling for help in emergencies Monitoring Rx adherence & safety Monitoring Rx adherence & safety Providing info (TV, weather) Providing info (TV, weather) Reminding to eat, drink, take meds Reminding to eat, drink, take meds Linking caregiver and resources Linking caregiver and resources

3. Joelle Pineau The Nursebot Project The Nursebot project in its early days

4. Joelle Pineau The Nursebot Project Architecture Cognitive supportNavigationCommunication High-level controller

5. Joelle Pineau The Nursebot Project Localization and map building (Burgard et al., 1999) People detection and tracking (Montemerlo et al., 2002) Architecture Cognitive supportNavigationCommunication High-level controller

6. Joelle Pineau The Nursebot Project Autominder system (Pollack et al., 2002) Architecture Cognitive supportNavigationCommunication High-level controller

7. Joelle Pineau The Nursebot Project Speech recognition: Sphinx system (Ravishankar, 1996) Speech synthesis: Festival system (Black et al., 1999) Architecture Cognitive supportNavigationCommunication High-level controller

8. Joelle Pineau The Nursebot Project The role of the top-level controller Établir les priorités parmi les objectifs des différents modules Négocier entre plusieurs objectifs ayant des coûts/gains variés Négocier entre l’acquisition d’information et la rencontre des objectifs Passer d’une tâche à l’autre en partageant l’information sensorielle Planifier malgré la présence d’incertitude Cognitive supportNavigationCommunication ACTION SELECTION - based on the trade-off between: - goals from different modules; - goals with varying costs / rewards; - reducing uncertainty versus accomplishing goals. High-level controller

9. Joelle Pineau The Nursebot Project Speech recognition with Sphinx

10. Joelle Pineau The Nursebot Project Robot control under uncertainty Belief State P(s t =weather-today)=0.5 P(s t =appointment-today )=0.5 USER Action={say-weather, update-appointment, clarify-query} Speech=“today” State weather-today

11. Joelle Pineau The Nursebot Project Partially Observable Markov Decision Processes Robot control using Partially Observable Markov Decision Processes (POMDPs) Belief state USER + ENVIRONMENT + WORLD Actions Observations Costs / Rewards State Problem: Which action allows the robot to maximize its reward? P(s 1 ) P(s 2 )

12. Joelle Pineau The Nursebot Project Methods to solve POMDPs Objective: Find a policy,  (b), which maximizes reward. Complexity Performance QMDP MDP FIB UMDP AMDP O(S 2 A) O(S 2 A T )O(S 2 A O ) O(S 2 AB) T POMDP New methods?

13. Joelle Pineau The Nursebot Project New approach: A hierarchy of POMDPs Idea: Exploit domain knowledge to divide one POMDP into many smaller ones. Motivation: Complexity of POMDP solving grows exponentially with # of actions. Assumption: We are given POMDP M = {S,A, ,b,T,O,R} and hierarchy H Act ExamineHealth Navigate Move VerifyPulse ClarifyGoal NorthSouthEastWest VerifyMeds subtask abstract action primitive action

14. Joelle Pineau The Nursebot Project PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds

15. Joelle Pineau The Nursebot Project PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Step 2: Minimize the state set STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds

16. Joelle Pineau The Nursebot Project PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds PARAMETERS {b h,T h,O h,R h } PARAMETERS {b h,T h,O h,R h }

17. Joelle Pineau The Nursebot Project PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters Step 4: Plan task h STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds PLAN  h PLAN  h PARAMETERS {b h,T h,O h,R h } PARAMETERS {b h,T h,O h,R h }

18. Joelle Pineau The Nursebot Project PolCA in the Nursebot domain Goal: A robot is deployed in a nursing home, where it provides reminders to elderly users and accompanies them to appointments. Domain : |S|=512, |A|=20, |O|=19 Hierarchy:

19. Joelle Pineau The Nursebot Project Sample scenario

20. Joelle Pineau The Nursebot Project Results for dialogue system 0.1 0.18 POMDP policy MDP policy

21. Joelle Pineau The Nursebot Project Summary We have developed a first prototype robot able to serve as a mobile nursing assistant for elderly people. The top-level controller uses a hierarchical variant of POMDPs to select actions. This allows it to acquire necessary information and successfully complete assigned tasks. Probabilistic techniques have been found to be very useful to flexibly model and track individuals.

22. Joelle Pineau The Nursebot Project For more details: www.cs.cmu.edu/~nursebot The Nursebot team CMU - Robotics: Greg Armstrong Michael Montemerlo Joelle Pineau Nicholas Roy Jamie Schulte Sebastian Thrun CMU - HCI/Design: Francine Gemperle Jennifer Goetz Sarah Kiesler Aaron Powers U. of Pittsburgh - Nursing: Jacqueline Dunbar-Jacobs Sandra Engberg Judith Matthews U. of Pittsburgh - CS: Don Chiarulli Colleen McCarthy U. of Freiburg - CS: Maren Bennewitz Wolfram Burgard Dirk Schulz U. of Michigan - CS: Laura Brown Dirk Colbry Cheryl Orosz Bart Peintner Martha Pollack Sailesh Ramakrishnan Standard Robotics: Greg Baltus