1. Pallet Detection and Engagement
Planning and Perception Groups
Matt Walter, Seth Teller, Nick Roy, Emilio Frazzoli, Jon How, Matt Antone, Mike Boulet,
Jeong hwan Jeon, and Brandon Luders August 8, 2008

2. Pallet Detection and Engagement
User Interface
Gestures
Range Data and
Camera Data
Inferred Pallet Geometry
Manipulation Plan

3. Pallet Detection and Engagement
User Interface
Gestures
Range Data and
Camera Data
Inferred Pallet Geometry
Manipulation Plan
Supervisor gestures provide initial estimate of pallet location
Pallet geometry (location, slot positions) inferred from sensor data
Manipulation plan coupled with inferred pallet geometry

4. Data Segmentation
Cluster laser data into different segments

5. Data Segmentation
Supervisor gestures and calibrated sensor model used to associate segmented data with correct pallet

6. Geometric Model and Pose Estimation
Given segmented pallet data, need to recover pallet geometry from different views
Two-way vs. four-way stringer pallets
Relative position, height, separation of slots
Possible non-standard geometry
May have irregular loads stacked on pallets

7. Geometric Model and Pose Estimation
Train classifier to predict laser points that are slot edges vs. non-slot-edges
Given new laser cloud for new pallet,
label slot edges
Compute initial pallet model
Compute likelihood of laser data
Use gradient of data likelihood to improve pallet model

8. Geometric Model and Pose Estimation
Camera Image
Laser data overlaid on camera
Laser data (rotated for better visibility)
Segmented laser data

9. Manipulation Planning
User Interface
Gestures
Range Data and
Camera Data
Inferred Pallet Geometry
Manipulation Plan

10. Manipulation Planning
Sample-based exploration strategy
Jointly solve for both near-pallet vehicle motion as well as the motion of the mast, carriage and tines.
Explicitly account for coupling between vehicle and mast trajectory and pallet pose estimation.

11. Manipulation Planning
Development process: various levels of sophistication and assumptions:
Zeroth-order: perfect knowledge of vehicle pose and pallet (pose, geometry and type)

12. Manipulation Planning
Development process: various levels of sophistication and assumptions:
First-order: Uncertainty in vehicle pose estimate and inferred pallet pose. ? ?

13. Manipulation Planning
Development process: various levels of sophistication and assumptions:
Second-order: Incorporate a predicted sensing model as well as account for uncertainty. i.e. “exploration”

14. Hierarchical Planning Structure
User Interface
Gestures
Range Data and
Camera Data
Navigator
Inferred Pallet Geometry
Task Observer
Task Plan
Motion Plan
Manipulation Plan
Manipulation Plan

15. Hierarchical Planning Structure
Canonical end-to-end mission.
Bot navigates to a coarse location in the SSA (e.g. reception) [a pre-defined node in a topological map representation]
Supervisor directs the bot to a desired pose (e.g. normal to truck) [arbitrary position and orientation]
The bot approaches the desired pallet
The bot acquires (manipulates) the pallet on the truck.
The bot transports the load to the desired location.
Reception
Bulk lot
Pickup

16. Task Planner Navigator Task Observer Task Plan Motion Plan
Manipulation Plan

17. Task Planner Interfaces with the UI interpretation engine
Responsible for high-level task allocation
Higher-level task queue
Task decomposition
Come to Reception
Pickup Green pallet from truck and move to Alpha Charlie
Move Red pallet from issue to truck
Achieve suitable orientation
relative to trailer
Acquire pallet
(manipulation planner)
Navigate to ASL bay Alpha Charlie
Release pallet

18. Navigator Navigator Task Observer Task Plan Motion Plan
Manipulation Plan

19. Topological Map Representation
Represent the world (SSA) as a topological map
nodes are salient points in the map (waypoints)
edges denote connectivity
Zones (reception, bulk storage, bulk issue) are analogous to submaps
Route network guides the bot’s motion
Utilize guided SSA tour to learn initial map

20. Navigator
Coarse, waypoint-level navigation relative to SSA topological map
A* search of the topological map for optimal route
Yields a “carrot” goal point for the motion planner

21. Motion Planner Navigator Task Observer Task Plan Motion Plan
Manipulation Plan

22. Motion Planner Identifies short term vehicle trajectories
Accounts for vehicle dynamics and load
Stochastic (RRT) motion planning search
Prefers lane structure, but flexible to deviations
User-modifiable paths (e.g. Google Maps)
Goal

23. Motion Planner: Vehicle Dynamics
Motion planner requires an accurate vehicle dynamics model.
Problems introduced by a range of load mass (0 lbs to 3000 lbs)
Adaptively model load dynamics
Goal

24. Task Observer Navigator Task Observer Task Plan Motion Plan
Manipulation Plan

25. Task Observer
Human operator may temporarily take control and guide the bot
The system must recognize when tasks have been completed independently of its actions
Tasks are solely dependent upon the current and historical state of the bot
The Task Completion Observer independently infers task progress from the system state

26. Navigator Motion Planner Task Observer Task Planner Manipulation
World Model
Motion Planner
Task Observer
Task Planner
Manipulation
Planner
User Interface
N810 UI
Interpreter
Tomcat
Perception WM
SLS Hub WM
Control
World Model
World Model
Actuation
Topological
Map
Object Map
Database
Vehicle
State

27. Timeline aug 08 oct 08 apr 08 jan 09 mar 09 dec 08
Core planning structure
Laser segmentation
Slot classification algorithm proposed
oct 08
Manipulation planner that accounts for uncertainty completed
apr 08
jan 09
Object geometry likelihood computation and model fitting implementation completed
mar 09
Coupled manipulation planner and perception implemented
on platform
dec 08
Planning and perception applied to an end-to-end mission on the platform