Towards robotic assistants in nursing homes: challenges and results Joelle Pineau Michael Montemerlo Martha Pollack * Nicholas Roy Sebastian Thrun Carnegie Mellon University * University of Michigan

Joelle Pineau The Nursebot Project Introducing Pearl – A mobile robotic assistant for elderly people and nurses cameras sonars handle bars mobile base carrying tray LCD mouth touchscreen microphone & speakers laser ROLE: Moving things around Moving things around Management support of ADLs Management support of ADLs Providing physical assistance Providing physical assistance Remote health services Remote health services Supporting communication Supporting communication Calling for help in emergencies Calling for help in emergencies Monitoring Rx adherence & safety Monitoring Rx adherence & safety Providing info (TV, weather) Providing info (TV, weather) Reminding to eat, drink, take meds Reminding to eat, drink, take meds Linking caregiver and resources Linking caregiver and resources

Joelle Pineau The Nursebot Project The Nursebot project in its early days

Joelle Pineau The Nursebot Project Architecture Cognitive supportNavigationCommunication High-level controller

Joelle Pineau The Nursebot Project Localization and map building (Burgard et al., 1999) People detection and tracking (Montemerlo et al., 2002) Architecture Cognitive supportNavigationCommunication High-level controller

Joelle Pineau The Nursebot Project Autominder system (Pollack et al., 2002) Architecture Cognitive supportNavigationCommunication High-level controller

Joelle Pineau The Nursebot Project Speech recognition: Sphinx system (Ravishankar, 1996) Speech synthesis: Festival system (Black et al., 1999) Architecture Cognitive supportNavigationCommunication High-level controller

Joelle Pineau The Nursebot Project The role of the top-level controller Établir les priorités parmi les objectifs des différents modules Négocier entre plusieurs objectifs ayant des coûts/gains variés Négocier entre l’acquisition d’information et la rencontre des objectifs Passer d’une tâche à l’autre en partageant l’information sensorielle Planifier malgré la présence d’incertitude Cognitive supportNavigationCommunication ACTION SELECTION - based on the trade-off between: - goals from different modules; - goals with varying costs / rewards; - reducing uncertainty versus accomplishing goals. High-level controller

Joelle Pineau The Nursebot Project Speech recognition with Sphinx

Joelle Pineau The Nursebot Project Robot control under uncertainty Belief State P(s t =weather-today)=0.5 P(s t =appointment-today )=0.5 USER Action={say-weather, update-appointment, clarify-query} Speech=“today” State weather-today

Joelle Pineau The Nursebot Project Partially Observable Markov Decision Processes Robot control using Partially Observable Markov Decision Processes (POMDPs) Belief state USER + ENVIRONMENT + WORLD Actions Observations Costs / Rewards State Problem: Which action allows the robot to maximize its reward? P(s 1 ) P(s 2 )

Joelle Pineau The Nursebot Project Methods to solve POMDPs Objective: Find a policy,  (b), which maximizes reward. Complexity Performance QMDP MDP FIB UMDP AMDP O(S 2 A) O(S 2 A T )O(S 2 A O ) O(S 2 AB) T POMDP New methods?

Joelle Pineau The Nursebot Project New approach: A hierarchy of POMDPs Idea: Exploit domain knowledge to divide one POMDP into many smaller ones. Motivation: Complexity of POMDP solving grows exponentially with # of actions. Assumption: We are given POMDP M = {S,A, ,b,T,O,R} and hierarchy H Act ExamineHealth Navigate Move VerifyPulse ClarifyGoal NorthSouthEastWest VerifyMeds subtask abstract action primitive action

Joelle Pineau The Nursebot Project PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds

Joelle Pineau The Nursebot Project PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Step 2: Minimize the state set STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds

Joelle Pineau The Nursebot Project PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds PARAMETERS {b h,T h,O h,R h } PARAMETERS {b h,T h,O h,R h }

Joelle Pineau The Nursebot Project PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters Step 4: Plan task h STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds PLAN  h PLAN  h PARAMETERS {b h,T h,O h,R h } PARAMETERS {b h,T h,O h,R h }

Joelle Pineau The Nursebot Project First study in simulation 20-questions domain: Agent: “Is it an animal?” User: “No.” Agent: “Is it a vegetable?” User: “Yes.” Agent: “Is it green?” User: “No.” Agent: “...?” Agent: “Is it an animal?” User: “No.” Agent: “Is it a vegetable?” User: “Yes.” Agent: “Is it green?” User: “No.” Agent: “...?” Actions Objective : Plan a sequence of questions allowing the agent to identify the chosen object. Small complication…. the user can change objects at any time ( Pr = 0.1 ), without telling the agent. Observations

Joelle Pineau The Nursebot Project The hierarchy Animals Begin Vegetable? VegetablesMinerals Mineral?Animal? Mammal?RABBITHerbivore?TURTLE…? ……

Joelle Pineau The Nursebot Project Methods to solve POMDPs Complexity Performance POMDP QMDP MDP FIB UMDP AMDP PolCA

Joelle Pineau The Nursebot Project Results Domain: |S|=1 2, |A|=20, |O|=3

Joelle Pineau The Nursebot Project A new decomposition Animals Begin Vegetable? VegetablesMinerals Mineral?Animal? …… …

Joelle Pineau The Nursebot Project A new decomposition FruitsPlants …… Animals Begin Vegetable? VegetablesMinerals Mineral?Animal? … Fruit? …

Joelle Pineau The Nursebot Project Results Domain: |S|=1 2, |A|=20, |O|=3

Joelle Pineau The Nursebot Project Methods to solve POMDPs Complexity Performance POMDP QMDP MDP FIB UMDP AMDP PolCA

Joelle Pineau The Nursebot Project PolCA in the Nursebot domain Goal: A robot is deployed in a nursing home, where it provides reminders to elderly users and accompanies them to appointments. Domain : |S|=512, |A|=20, |O|=19 Hierarchy:

Joelle Pineau The Nursebot Project Sample scenario

Joelle Pineau The Nursebot Project Results for dialogue system POMDP policy MDP policy

Joelle Pineau The Nursebot Project Summary We have developed a first prototype robot able to serve as a mobile nursing assistant for elderly people. The top-level controller uses a hierarchical variant of POMDPs to select actions. This allows it to acquire necessary information and successfully complete assigned tasks. Probabilistic techniques have been found to be very useful to flexibly model and track individuals.

Joelle Pineau The Nursebot Project For more details: The Nursebot team CMU - Robotics: Greg Armstrong Michael Montemerlo Joelle Pineau Nicholas Roy Jamie Schulte Sebastian Thrun CMU - HCI/Design: Francine Gemperle Jennifer Goetz Sarah Kiesler Aaron Powers U. of Pittsburgh - Nursing: Jacqueline Dunbar-Jacobs Sandra Engberg Judith Matthews U. of Pittsburgh - CS: Don Chiarulli Colleen McCarthy U. of Freiburg - CS: Maren Bennewitz Wolfram Burgard Dirk Schulz U. of Michigan - CS: Laura Brown Dirk Colbry Cheryl Orosz Bart Peintner Martha Pollack Sailesh Ramakrishnan Standard Robotics: Greg Baltus

Joelle Pineau The Nursebot Project Our vision of robotic healthcare Moving things around Moving things around Enabling use of remote health services Enabling use of remote health services Supporting inter-personal communication Supporting inter-personal communication Calling for help in emergencies Calling for help in emergencies Monitoring Rx adherence & safety Monitoring Rx adherence & safety Providing information (TV, weather) Providing information (TV, weather) Management support of ADLs Management support of ADLs Reminding to eat, drink, & take meds Reminding to eat, drink, & take meds Providing physical assistance Providing physical assistance Linking the caregiver to resources Linking the caregiver to resources

Joelle Pineau The Nursebot Project Profile of an aging population  450,000 nurses to recruit before 2008 (USA)

Joelle Pineau The Nursebot Project Localization and map building

Joelle Pineau The Nursebot Project People tracking

Joelle Pineau The Nursebot Project Analysis of movements

Joelle Pineau The Nursebot Project Autominder System

Joelle Pineau The Nursebot Project The family of Markov models Markov Chain Hidden Markov Model (HMM) Markov Decision Process (MDP) Partially Observable Markov Decision Process (POMDP) State ambiguity? noyes Choice of action? yes no

Joelle Pineau The Nursebot Project The POMDP is a septuple { S, A, , b, T, O, R } Problem: Which action allows the robot to maximize its reward? b t-1 btbt a t-1 otot s t-1 stst... ?? o t-1... r t-1 rtrt The POMDP model Belief: State: Action:

Joelle Pineau The Nursebot Project Execution with a hierarchy of POMDPs At each time step, traverse the hierarchy from top to bottom. For each subtask, consult the policy a t   h (b t ) If a t is an internal node > move to that subtask. Act ExamineHealth Navigate Move VerifyPulse ClarifyGoal NorthSouthEastWest VerifyMeds

Joelle Pineau The Nursebot Project How to choose parameters? Consider for example T Navigate (s,a,s’) Case #1: a is a primitive action. T Navigate (s,ClarifyGoal,s’)  T(s,ClarifyGoal,s’) Navigate Move ClarifyGoal SouthEast West North S S’ South S’ North S’ ClarifyGoal

Joelle Pineau The Nursebot Project How to choose parameters? Consider for example T Navigate (s,a,s’) Case #1: a is a primitive action. T Navigate (s,ClarifyGoal,s’)  T(s,ClarifyGoal,s’) Case #2: a is an abstract action. T Navigate (s,Move,s’)  T(s,  Move (s),s’) Navigate Move ClarifyGoal SouthEast West North S S’ South S’ North S’ ClarifyGoal

Joelle Pineau The Nursebot Project States:1 per object (e.g. tomato, cucumber, rabbit, turtle, tulip, ….) Observations:“yes”, “no”, “  ” Actions:1 guess per objet Rewards:question= -1 + many questionscorrect guess = +5 incorrect guess= -20 The details Animals Begin Vegetable? VegetablesMinerals Mineral?Animal? Mammal?RABBITHerbivore?TURTLE…? ……