1. High-level robot behavior control using POMDPs Joelle Pineau and Sebastian Thrun Carnegie Mellon University

2. Joelle PineauCogRob2002 workshop Introducing Pearl Pearl is a prototype nursing robot, providing assistance to both nurses and elderly people. “thinkers” eyes with cameras sonar sensors handlebars wheeled base carrying tray LCD smile/frown

3. Joelle PineauCogRob2002 workshop Profile of our aging population ã 450,000 more nurses needed by 2008 ã campaign to recruit and retain nurses and other health care providers

4. Joelle PineauCogRob2002 workshop Moving things around Moving things around Enabling use of remote health services Enabling use of remote health services Supporting inter-personal communication Supporting inter-personal communication Calling for help in emergencies Calling for help in emergencies Monitoring Rx adherence & safety Monitoring Rx adherence & safety Providing information (TV, weather) Providing information (TV, weather) Management support of ADLs Management support of ADLs Reminding to eat, drink, & take meds Reminding to eat, drink, & take meds Providing physical assistance Providing physical assistance Linking the caregiver to resources Linking the caregiver to resources Our vision of robotic-assisted health-care

5. Joelle PineauCogRob2002 workshop The Nursebot project in its early days

6. Joelle PineauCogRob2002 workshop We need a high-level controller that can: prioritize goals from specialized modules trade-off goals with widely different costs/rewards trade-off between information-gathering and goal-satisfaction switch between tasks and share sensory information handle uncertainty Speech recognition&synthesis (Ravishankar, 1996; Black et al., 1999) Autonomous navigation (Burgard et al., 1999) Autominder (Pollack et al., 2002) People tracking/following (Montemerlo et al., 2002) Top controller

7. Joelle PineauCogRob2002 workshop High-level robot behavior control using Partially Observable Markov Decision Processes USER + WORLD + ROBOT ACTIONS OBSERVATIONS BELIEF STATE STATE

8. Joelle PineauCogRob2002 workshop What are POMDPs? POMDP is n-tuple { S, A, , b, T, O, R } POMDP task 1: State tracking: After an action, what is the state of the world? POMDP task 2: Computing a policy: Which action should the controller apply next? Very hard! Not so hard. Robot belief: World state: Control layer: b t-1 ?? a t-1 otot s t-1 stst... ?? o t-1... r t-1 rtrt

9. Joelle PineauCogRob2002 workshop Our approach: Hierarchical POMDPs Key Idea : Exploit hierarchical structure in the problem domain to break a problem into many “related” POMDPs. Act InvestigateHealthMove Navigate CheckPulse AskWhere NorthSouthEastWest CheckMeds subtask abstract action primitive action

10. Joelle PineauCogRob2002 workshop Planning with Hierarchical POMDPs Given POMDP model M = { S, A, , b, T, O, R } and hierarchy H For each subtask h  H: 1) Set components A h  children nodes S h  S  h   b h, T h, O h, R h 2) Minimize model S h  {z h (s 0 ), …, z h (s n )}  h  {y h (o 0 ), …, y h (o p )} 3) Solve subtask h  h  { b h, T h, O h, R h } Move Navigate AskWhere SouthEast West North A Nav ={N,S,E,W} S Nav ={X,Y}  Nav ={o 0,…,o m } A Move ={AskWhere,Navigate} S Move ={X’,Y’,Destination}  Move ={o 0,…,o p }

11. Joelle PineauCogRob2002 workshop Execution with Hierarchical POMDPs Step 1 - Update belief: Step 2 - Traversing hierarchy top-down, for each subtask: 1) Get local belief. 2) Consult local policy. 3) If a is leaf node, terminate. Else, go to that subtask. Act InvestigateHealth Move Navigate CheckPulse AskWhere NorthSouthEast West CheckMeds

12. Joelle PineauCogRob2002 workshop Experimental Setup Task: Robot provides reminders and guidance to elderly user. Action hierarchy:

13. Joelle PineauCogRob2002 workshop Sample Scenario

14. Joelle PineauCogRob2002 workshop Still unconvinced about the importance of uncertainty?

15. Joelle PineauCogRob2002 workshop numericalogical On the question of numerical vs logical representations... We haven’t tried using logical representations to control the robot… But our experience tells us that: ã Uncertainty is crucial when dealing with people. ã Probabilistic techniques are necessary to reason about uncertainty. ã Real belief tracking and planning really matters!