1. High-level robot behavior control using POMDPs
Joelle Pineau and Sebastian Thrun
Carnegie Mellon University
I - Background
III - Experimental Setup
USER + WORLD + ROBOT
ACTIONS
OBSERVATIONS
Belief
state
STATE
Abstract: This paper describes a robot controller which uses probabilistic decision-making techniques at the highest-level of behavior control. The POMDP-based robot controller has the ability to incorporate noisy and partial sensor information, and can arbitrate between information-gathering and performance-related actions. We present a hierarchical variant of the POMDP model which exploits structure in the problem domain to accelerate planning. This POMDP controller is implemented and tested onboard a mobile robot in the context of an interactive service task.
What are POMDPs? POMDPs model decision-theoretic planning problems. The goal is to find an action-selection strategy that maximizes reward, even in the presence of state uncertainty.
Introducing Pearl, the nursing assistant robot:
Problem size: |S|=576, |A|=19, |O|=18
Action hierarchy:
State features:
Robot location
Person Location
Person Status
Reminder Goal
Motion Goal
Conversation Goal
Observation features:
Words from speech recognition
Button presses from touchscreen
Laser readings
Reminder messages
We need a high-level controller that can:
select good behaviors or actions
share sensor information between modules
handle uncertainty
negotiate over goals from different specialized modules
arbitrate between information-gathering and performance actions
Formally, a POMDP is an n-tuple {S,A,,b,T,O,R }
POMDP task 1: Track state:
After an action, what is the state of the world?
POMDP task 2: Optimize policy:
Which action should the controller apply next?
S : Set of states
A : Set of actions
 : Set of observations
b(s) := Pr(s | t=0)
T(s,a,s’) := Pr(s’ | s,a)
O(s,a,o) := Pr(o | s,a)
R(s,a) 
Robot belief:
World state:
Control layer:
bt-1 ??
at-1 ot
st-1 st
...
ot-1
rt-1 rt
Top controller
Task domain: Robot provides reminders and guidance to elderly user.
Experimental scenario: Robot must go meet subjects in their apartment and take them to a physiotherapy appointment, while also engaging in appropriate social interaction.
Environment: Nursing home near Pittsburgh, PA
Test subjects: Six elderly residents in assisted living facility.
Physiotherapy
Patient
room
Robot
home
People tracking/following
Autominder
Not so hard.
Speech recognition&synthesis
Autonomous navigation
Very hard!
Our approach: High-level robot behavior control using Partially Observable Markov Decision Processes (POMDPs)
Exact policy optimization (task 2) is computationally intractable for large problems (~20+ states), therefore we need approximations.
II - Robot control using Hierarchical POMDPs
IV - Results
Assumptions
Approximating POMDPs for large domains
Act
InvestigateHealth
Move
Navigate
CheckPulse
AskWhere
North
South
East
West
CheckMeds
subtask
abstract
action
primitive action
Performance results for three contrasting users. Comparing performance of POMDP policy vs MDP policy. The MDP policy does not consider uncertainty during planning, and therefore is unable to ask clarification actions. For all users, performance is much better using the POMDP policy.
Figure 1: Example of a successful guidance experiment.
Each subtask is a separate POMDP.
Each subtask has a subset of all actions.
Primitive actions are placed in leaf nodes, and are from the original action set.
Abstract actions are introduced in internal nodes.
Each subtask has a non-trivial reward function (i.e. R(s,a) is not constant).
Key Idea: Exploit hierarchical structure in the problem domain to break a large problem into many “related” POMDPs.
What type of structure? Action set partitioning
(a) Pearl approaching subject
(b) Reminding of appointment
Planning with Hierarchical POMDPs
(c) Guidance through corridor
(d) Entering physiotherapy dept.
Given POMDP model M = { S, A, , b, T, O, R } and subtask hierarchy H
For each subtask h  H:
1) Set components
Ah  children nodes
Sh  S
h  
bh, Th, Oh, Rh
2) Minimize model
Sh  {zh(s0), …, zh(sn)}
h  {yh(o0), …, yh(op)}
3) Solve subtask
h  {bh, Th, Oh, Rh}
Execution with Hierarchical POMDPs
Move
Navigate
AskWhere
South
East
West
North
ANav ={N,S,E,W}
SNav ={X,Y}
Nav ={o0,…,om}
AMove ={AskWhere,Navigate}
SMove ={X’,Y’,Destination}
Move ={o0,…,op}
Step 1 - Update belief:
Step 2 - Traversing hierarchy top-down, for each subtask:
1) Get local belief:
2) Consult local policy:
3) If a is leaf node, terminate.
Else, go to that subtask.
(e) Asking for weather forecast
(e) Pearl leaves
Observations from nursing-home experiments:
100% task completion rate amongst test subjects
overall high-level of excitement amongst subjects
adaptive speed control for robot is necessary
improved speech recognition would be great
more verbal interaction during guidance is recommended