1. Agile Robotics Program Review AR Team MIT, Lincoln Lab, Draper Lab, BAE August 8, 2008

2. Agile Robotics Team Albert Huang Luke Fletcher Matthew Walter Lenny Paritsky Troy Jones Mitch Leammukda Steve Proulx Rob Truax Missy Cummings Jake Crandall Birsen Donmez Nicholas Roy Russ Tedrake Edmund Williams Ed Corbett Mike Boulet Bryt Bradley Nira Manokharan Seth Teller Matt Antone Emma Brunskill Geoff Carrigan Mofe Uku Jeong hwan Jeon Emilio Frazzoli Jon How Scott Cyphers Stephanie Seneff Tara Sainath Lee Hetherington Alex Gruenstein Howard Shrobe Andrew Correa Jim Glass Randy Davis Platform buildup Safety & intent Situational awareness Planning & control Supervisor interface Program support Matthew Walter Seth Teller Mitch Leammukda Lenny Paritsky Jim Liu Paul Pepin Brandon Luders Daniela RusNicholas Roy Whole-SSA simulation John Leonard Ryne Barry USA LIA DDRE CASCOM ARL

3. Agenda 0900 – 0930: Arrive MIT Kiva room (32G-449) 0930 – 0945: Informal introductions 0945 – 1030: Summary goals and status 1030 – 1045: Break, walk to demonstration venues 1045 – 1145: Demonstrations (Hangar, Holodeck, Kiva) 1200 – 1300: Lunch (Highlights of other MIT robotics) 1300 – 1415: Technical briefings 1415 – 1430: Break 1430 – 1530: Technical briefings 1530 – 1630: Feedback and discussion 1630: Main group adjourns 1630 – 1700: Program management discussion 1700: Program review adjourns

4. Administrative Notes Name placards Room security Kiva conference room Bathrooms Handouts / briefing charts

5. Agenda 0900 – 0930: Arrive MIT Kiva conference room (32G-449) 0930 – 0945: Informal introductions 0945 – 1030: Summary goals and status 1030 – 1045: Break, walk to demonstration venues 1045 – 1145: Demonstrations (Hangar, Holodeck, Kiva) 1200 – 1300: Lunch (Highlights of other MIT robotics) 1300 – 1415: Technical briefings 1415 – 1430: Break 1430 – 1530: Technical briefings 1530 – 1630: Feedback and discussion 1630: Main group adjourns 1630 – 1700: Program management discussion 1700: Program review adjourns

6. Project Goals Build upon foundation of DARPA Challenge –Real-time perception, planning, and control –Avoid brittle aspects (prior map, GPS) of DGC Collaborate with LIA, CASCOM –Develop “Agile Robotics” autonomous forklift Introduce new mobile manipulation aspects –Whole-pallet manipulation (Year 1) –Sub-pallet manipulation (Years 2-3) Demonstrate prototype at end of Year 1 Migrate capability to Army vehicle Years 2,3

7. Summary Status Tackling problem along four fronts, in parallel: –Requirements analysis: LIA, CASCOM, Ft. Campbell –Simulation studies for sensing, dataflow, throughput –Mockup experiments, drive-by-wire elements –Full-scale prototype development, data collection At present, substantially ahead of schedule –Several September, December milestones achieved –Demonstrated components in simulation, on mockup –(Partially) working full-scale prototype by fall 2008 Anticipated fruitful directions for Years 2, 3 –Improved manipulation, reasoning, sensing, scaling

8. 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone Bulk lot Pickup Reception 1 Scenario: A Commandable Forklift (Notional SSA Layout)

9. 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone Bulk lot Pickup Reception Forklift Supervisor 2 Demonstration Scenario

10. Bulk lot Pickup Reception ASL 3 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

11. Demonstration Scenario Bulk lot Pickup Reception 4 QA 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

12. Demonstration Scenario Bulk lot Pickup Reception 5 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

13. Demonstration Scenario Bulk lot Pickup Reception 6 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

14. Design Requirements Develop plausible prototype capability –Compatible with existing Army platforms –Safe, acceptable to Army personnel –Capable of locating, moving, placing pallets –Affordable (acceptable cost increment) –Rugged (operates outdoors on uneven terrain)

15. Principal Deliverables Prototype Demonstration (March 2009) –Single voice-, gesture-commandable forklift –Whole-pallet manipulation: truck/ground to ground/truck Documentation of effort results –Detailed drive-by-wire conversion procedure –Study of task requirements, sensor choices –Report on system architecture and algorithms Plans for continued capability development –Technology transition plan (Years Two and Three) –Extend capability along several dimensions Fine-grain manipulation; higher-level reasoning; multiple forklifts and supervisors; adverse environments; GPS-denial

16. Working Timeline Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr Y2 Year 1 Requirements analysis, system & interface design, safety, user testing Simulation development Mockup forklift Full-scale prototype Capability migration Visit to Visit to Visit to CASCOM Incursion Seamless Shouted Testing & LIA CASCOM Ft. Campbell Visit to MIT detection handoff warnings validation Port DGC Speech Gesture Pallet Situational Whole-SSA codebase integration support detection awareness simulation Drive-by-wire Sensor Planning Pallet prototyping placement and control engagement Drive-by-wire Sensor Planning Pallet Pallet Vehicle prototyping placement and control engagement mobility interaction Lincoln FMTV Laboratory studies Kickoff Meeting Program Review Year 1 Demo (Rented forklift)(Purchased forklift) Program Review Today Preliminaries: Drive-by- Wire Platform Development Prototype Autonomous Capability Development

17. Milestone Status ( = completed) Capability 0.0: 30 Jun 08 – Analyze and summarize existing SSA operational practices – Demonstrate low-fidelity simulation of terrain, pallets, and trucks – Develop prototype speech, gesture interface to command forklift Capability 0.1: 30 Sep 08 – Build a partially-actuated mockup forklift Gather data on pallet approach, pallet and slot detection / localization –Develop prototype interface, perception, planning, control algorithms –Demonstrate mockup working in a real-world scenario to identify, localize, select, engage, lift, transport, and place one pallet at a time as directed Capability 0.2: 31 Dec 08 –Convert a stock manual forklift to drive-by-wire control Identify model to purchase Lease example of target forklift Procure suitable sensor suite and compute server Instrument leased forklift with sensor suite Transfer drive-by-wire elements to prototype forklift –Team members complete required OSHA forklift training –Understand degrees of freedom, sensing and control of forklift –Develop functional interface, perception, planning, control algorithms Capability 0.3: 31 Mar 09 –Prototype demonstration, venue to be determined (tentatively Ft. Belvoir)

18. Working Timeline Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr Y2 Year 1 Requirements analysis, system & interface design, safety, user testing Simulation development Mockup forklift Full-scale prototype Capability migration Visit to Visit to Visit to CASCOM Incursion Seamless Shouted Testing & LIA CASCOM Ft. Campbell Visit to MIT detection handoff warnings validation Port DGC Speech Gesture Pallet Situational Whole-SSA codebase integration support detection awareness simulation Drive-by-wire Sensor Planning Pallet prototyping placement and control engagement Drive-by-wire Sensor Planning Pallet Pallet Vehicle prototyping placement and control engagement mobility interaction Lincoln FMTV Laboratory studies Kickoff Meeting Program Review Year 1 Demo (Rented forklift)(Purchased forklift) Program Review Today Preliminaries: Drive-by- Wire Platform Development Prototype Autonomous Capability Development

19. Understanding the Task MIT team visits LIA, CASCOM, Ft. Campbell Collect, study relevant Dept. of Army pamphlets CASCOM forklift operator visits MIT OSHA forklift operator certification for team members

20. DGC System Architecture Perception Navigator MDF Goal Trajectory Steer, gas, brake Vehicle states Drivable surface, Lane markings, Obstacles, Traffic vehicles Local map Drivable Surface, Hazards Situational Planner Vehicle Controller Landrover LR3 Vehicle State Estimator Sensors RNDF

21. AR System Architecture Existing elements (from DGC codebase) –Sensor handlers and terrain perception –Ethernet and CANbus networks Novel elements –Whole-SSA environment model –Interpretation of supervisor speech and gestures –Detection of trucks, pallets, loads, pallet slots –Forklift mast planning and control –Seamless autonomy handoff, return –Shouted warning detection –Visible and audible apparent intent

22. AR System Architecture Drive-by-Wire Modifications Mobility and Manipulation Planning and Control Sensing and Situational Awareness Supervisor Speech, Gestures Failure: Diagnostic Information Success: Confirming Information To bystanders, via annunciators Standard Forklift Forklift Operator Supervisor Interface/ Interpreter Tablet

23. AR System Architecture Perception Navigator Goal Trajectory Steer, gas, brake, transmission, parking brake, mast Vehicle state Local map Terrain, People, Trucks, Pallets Situational Planner Vehicle, mast Controller Forklift Vehicle State Estimator Sensors Supervisor Interface/ Interpreter Safety: Incursions, Shouted Warnings Speech & Gesture Interpretation Sensors Microphones Whole-SSA Environment Model Visible, Audible Apparent Intent (Novel elements shown in red boxes) (Novel elements shown in red boxes) Sensor data

24. Working Timeline Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr Y2 Year 1 Requirements analysis, system & interface design, safety, user testing Simulation development Mockup forklift Full-scale prototype Capability migration Visit to Visit to Visit to CASCOM Incursion Seamless Shouted Testing & LIA CASCOM Ft. Campbell Visit to MIT detection handoff warnings validation Port DGC Speech Gesture Pallet Situational Whole-SSA codebase integration support detection awareness simulation Drive-by-wire Sensor Planning Pallet prototyping placement and control engagement Drive-by-wire Sensor Planning Pallet Pallet Vehicle prototyping placement and control engagement mobility interaction Lincoln FMTV Laboratory studies Kickoff Meeting Program Review Year 1 Demo (Rented forklift)(Purchased forklift) Program Review Today Preliminaries: Drive-by- Wire Platform Development Prototype Autonomous Capability Development

25. Simulation Studies Experiments with: –Sensor placement –Truck, pallet, slot detection –SSA structure and layout –Planning, control algorithms –Supervisor interface & bi-directional dataflow

26. Whole-SSA Simulation Simulated whole-SSA operation –Multiple forklifts, supervisors, customers Predict task throughput, wait times for a variety of operating parameters –Speed, # trucks, # supervisors, failure rate &c.

27. Working Timeline Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr Y2 Year 1 Requirements analysis, system & interface design, safety, user testing Simulation development Mockup forklift Full-scale prototype Capability migration Visit to Visit to Visit to CASCOM Incursion Seamless Shouted Testing & LIA CASCOM Ft. Campbell Visit to MIT detection handoff warnings validation Port DGC Speech Gesture Pallet Situational Whole-SSA codebase integration support detection awareness simulation Drive-by-wire Sensor Planning Pallet prototyping placement and control engagement Drive-by-wire Sensor Planning Pallet Pallet Vehicle prototyping placement and control engagement mobility interaction Lincoln FMTV Laboratory studies Kickoff Meeting Program Review Year 1 Demo (Rented forklift)(Purchased forklift) Program Review Today Preliminaries: Drive-by- Wire Platform Development Prototype Autonomous Capability Development

28. Mockup Forklift Enables experiments with: –Forklift elements to be converted to Drive-by-Wire: Parking brake and latch release Pedal brake Steering wheel and column Mast raise, lower, tilt Mast tine spread and side-shift Rapid transfer to prototype forklift when available –Sensing, planning and control algorithms for: Rear-wheel steering Pallet detection, localization, approach Pallet engagement (tine insertion), transport, placement Unloaded, loaded mobility Incursion detection (operator override)

29. Working Timeline Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr Y2 Year 1 Requirements analysis, system & interface design, safety, user testing Simulation development Mockup forklift Full-scale prototype Capability migration Visit to Visit to Visit to CASCOM Incursion Seamless Shouted Testing & LIA CASCOM Ft. Campbell Visit to MIT detection handoff warnings validation Port DGC Speech Gesture Pallet Situational Whole-SSA codebase integration support detection awareness simulation Drive-by-wire Sensor Planning Pallet prototyping placement and control engagement Drive-by-wire Sensor Planning Pallet Pallet Vehicle prototyping placement and control engagement mobility interaction Lincoln FMTV Laboratory studies Kickoff Meeting Program Review Year 1 Demo (Rented forklift)(Purchased forklift) Program Review Today Preliminaries: Drive-by- Wire Platform Development Prototype Autonomous Capability Development

30. Rental, Purchase Forklift Enables experiments with: –Sensor placement and sensor data under pallet approach –Electrical interfaces to existing (electrical) DBW elements –Characterization of suspension dynamics when under load –Annunciator visibility/audibility Purchase forklift as base platform for eventual demonstration prototype –Toyota 8FGU-15 lift truck ordered May 19, 2008 –Toyota arranged expedited delivery: July 23, 2008 –Transfer actuated mechanisms from mockup in August –Begin integrated outdoor experiments August 2008

31. Principal Research Advances Operation in semi-structured environment Hierarchical task-level autonomy Voice/gesture interface from bot’s-eye-view Seamless autonomy handoff & return Annunciation of apparent intent Detection of shouted warnings

32. Semi-Structured Environment Idea: Give forklift a narrated, guided tour! Manually-driven forklift path Operator utterance and / or gesture

33. Hierarchical Task-Level Autonomy Not tele-operated; Not fully autonomous But: at a useful operating point in between –Summon forklift to working area –Direct it toward one truck (of several) –Direct it toward one aspect (of several) –Direct it toward one pallet (of several) –Help it localize pallet, slots (if necessary) –Direct it to destination (bulk lot, issue area etc.) Rich, incremental path to full autonomy

34. Voice and Gesture Interface PDA interface for commanding forklift Interpretation of supervisor’s speech Supervisor uses stylus gestures to: –Summon and direct forklift –Confirm or edit motion paths –Indicate which pallet, slots are to be engaged

35. Autonomy Hand-off and Return Fundamental design constraint: –Forklift pauses if human approaches –Relinquishes autonomy if s/he enters cabin –Operates indistinguishably from manual forklift –Returns to autonomous mode after human exit Perhaps after an explicit go-ahead from human Six distinct, independent safety layers –Described later in briefing Challenging implications for planning layer –Described later in briefing

36. Apparent Intent Visible/Audible mode annunciators –Words, symbols, LEDs on cabin exterior –Speakers for audible annunciation –Redundant modes for safety Robot announces when: –It is about to move –It is stuck and needs help –It is paused, awaiting supervisor’s instructions

37. Demonstrations 0900 – 0930: Arrive MIT Kiva conference room (32G-449) 0930 – 0945: Informal introductions 0945 – 1030: Summary goals and status 1030 – 1045: Break, walk to demonstration venues 1045 – 1145: Demonstrations (Hangar, Holodeck, Kiva) 1200 – 1300: Lunch [Highlights of other MIT robotics] 1300 – 1415: Technical briefings 1415 – 1430: Break 1430 – 1530: Technical briefings 1530 – 1630: Feedback and discussion 1630: Main group adjourns 1630 – 1700: Program management discussion 1700: Program review adjourns