1. Robot-Assisted Urban Search and Rescue at the WTC Disaster Prof. Robin Murphy Director, Research CRASAR also Associate Prof., Computer Science & Eng. University of South Florida murphy@csee.usf.edu John Blitch, Jenn Casper, Mark Micire, Brian Minten www.crasar.org

2. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)2 NBC Today Show Sept 20, 2001

3. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)3 Outline Why Robots? What should robots do? What did they do? What is needed?

4. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)4 The First Known Robot-Assisted USAR CRASAR stood up by John Blitch Sept. 2 to transfer military robots for USAR: deploy, train, evaluate, & research robots for SAR At WTC, CRASAR robots and personnel worked with FDNY, NYC, and FEMA teams such as INTF-1, PATF-1, VATF-1, OHTF-1 DurationTuesday, Sept 11 – Oct. 2 Victims found10+ CRASAR Teams/people Defense Advanced Projects Research Agency (1), US Dept. of Defense (2), Foster-Miller (5), iRobot (8), US Navy SPAWAR (3), University of South Florida (4) Robots on site17, 7 used on rubble pile Field excursionsthrough 9/22… 11 (8 on rubble pile with depths of 20-45ft) 1 min set up, avg. 7 minutes per run USF brought 6, 2 from NSF USF team logged most hours on The Pit in 1 st 2 weeks (except Blitch) USF was only USAR certified team

5. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)5 photos courtesy of Justin Reuter

6. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)6 Why Use Robots? Things that humans can’t do or can’t do safely –“the human use of humans” Norbert Wiener –and that applies to dogs 135 rescuers died Mexico City, 65 in confined spaces Not enough trained people –1 survivor, entombed: 10 rescuers, 4 hours –1 survivor, trapped/crushed: 10 rescuers, 10 hours photos courtesy of Justin Reuter

7. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)7 FEMA DoD Liaison Plan/OpsSafety Haz Mat Tech Search 2 Canine Search Spec. Search Team 2 Rescue Squads 12 Rescue Spec. Rescue Team 2 Medical Specialists Medical Team Logs Spec. Comm. Spec. PIO Rigging Structure Specialist Logistics Team Task Force Leader Local Jurisdiction ESF9 Tasks: Search, Assessment, Medical FEMA Task Force Organizational Chart photo courtesy of Justin Reuter

8. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)8 Haz Mat Tech Search 2 Canine Search Spec. Search TeamRescue TeamMedical Team Structure Specialist Logistics Team Task Highlights Rescue Phase 9-11 through 9-21 worked with FDNY, FEMA found 6+ victims (equivalent to FLTF-2) Rescue Phase 9-11 through 9-21 worked with FDNY, FEMA found 6+ victims (equivalent to FLTF-2) not used Recovery Phase 9-23 through 10-02 NYC DDC engineers found 3-5 victims added HazMat sensors Recovery Phase 9-23 through 10-02 NYC DDC engineers found 3-5 victims added HazMat sensors needed

9. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)9 Search Task Priorities Rescue first, recover later –first responders (always take care of your own) –civilians in rubble pile –civilians elsewhere Information for triage –where are the survivors? –where are the people likely to be? –where are the survivors likely to have survived? –which pile of rubble do I remove first? Typical resources –dogs, search cams, acoustics, sledgehammers SearchCam: camera on a pole dogs: injured by sharp metal, smell only 0.3m due to rain on 2 nd day, lack of circulating air

10. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)10 Robots Compared with SearchCams SearchCams ~$10K, Robots ~$12K SearchCams reach up to 5 meters Robots reach up to 30 meters, averaged between 6 and 13 meters Robots can put light on the object, prod it, look at it from different angles Robots can go through more twisting tunnels SearchCams and small robots take <1.5 minutes to set up and insert helped to recognize remains of a body camera with zoom, lights couldn’t do this

11. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)11 History of Robot-Assisted USAR in USA

12. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)12 Timeline of Response (Rescue and Recovery, Inspection not shown) Blitch Foster-MilleriRobot JPOSPAWAR USF 111213141516171819202122 TWThFSSuMTWThFS Moore Ciholas/ Alibozek Pratt Schein Smith Blitch Mangolds Haglund Mouru Frost Moore Ciholas Murphy Casper Micire Minten Everett Baruch Laird Levan Hudson Frost Pratt Norman Mangolds Mouru Frost Pratt Norman Mangolds Mouru Murphy Casper Micire Minten Inuktuns Solem U 114S DARPA USF Foster-Miller rubble pile (The Pit) Talon Packbot UrBot Foster-Miller iRobot SPAWAR collateral buildings

13. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)13 Where The Robots Were in the Rubble Pile New York Times 9/23/01 1 6 7 2 3 4 5 9/18/01 0-live 4-dead VATF-1 9/16/01 0-live INTF-1 9/12/01 0-live 1-dead 9/12/01 0-live 9/13/01 0-live 1-dead FDNY Not shown: PATF-1 OHTF-1 NYC

14. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)14 Environment at Ground Zero Personal Safety: -Thick dust for days - asbestos, glass -Rubble largely stable, but if fall, could be impaled - 1 injury: fell out of chair Weather: -45-70 deg F -rain 2 days, making rubble slick, slippery Personal Safety: -Thick dust for days - asbestos, glass -Rubble largely stable, but if fall, could be impaled - 1 injury: fell out of chair Weather: -45-70 deg F -rain 2 days, making rubble slick, slippery

15. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)15 Contextual View: WTC 2

16. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)16 Confined Space using Tethered Bots 2 types (chemical & sewer, HVAC inspection, TRL 9 but not for USAR) –“polymorphic”: ~60 lbs, 7 hours battery, fits in 1 backpack, $12K height of mouse to height of squirrel –“fixed geometry”: ~70 lbs., 7 hours battery, fits in 1 backpack, $10K Tethers: 100-300ft Terrain & Environment –voids ~6-12 inch diameter –dirt, rubble (but not mounds of paper) –inclines: depends up steep, go with heavier microTracks –water resistant –not intrinsically safe, can melt tracks 2 types (chemical & sewer, HVAC inspection, TRL 9 but not for USAR) –“polymorphic”: ~60 lbs, 7 hours battery, fits in 1 backpack, $12K height of mouse to height of squirrel –“fixed geometry”: ~70 lbs., 7 hours battery, fits in 1 backpack, $10K Tethers: 100-300ft Terrain & Environment –voids ~6-12 inch diameter –dirt, rubble (but not mounds of paper) –inclines: depends up steep, go with heavier microTracks –water resistant –not intrinsically safe, can melt tracks Inuktun microTrac video, 2 way audio Inuktun microVGTV, Inuktun pipe crawler video, 2 way audio www.inuktun.com

17. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)17 Needed Image Processing and Object Recognition Technologies -tracks of previous robot run -a watch -3, possibly 4 victims (covered in dust, burned) - head, arm socket?, nose, perhaps fingers WTC 2, 9/18/01

18. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)18 State of Available Information what viewpoints have already been explored –example: boot? no one rewound tape far enough back to catch earlier viewpoint which disambiguated the object!

19. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)19 Confined Space, Structural Assessment using Wireless Bots 1 type (military hostage rescue, special order, TRL 9 but not for USAR) –backpackable by 2 people, 1 for bot, 1 for OCU & batteries, 12 hours-7 days (standby) –fast, can right itself with practice –can add sensors, payloads –~$30K Wireless –depends on material Terrain & Environment –voids: mansize –dirt, rubble,inclines: depends –water resistant –not intrinsically safe 1 type (military hostage rescue, special order, TRL 9 but not for USAR) –backpackable by 2 people, 1 for bot, 1 for OCU & batteries, 12 hours-7 days (standby) –fast, can right itself with practice –can add sensors, payloads –~$30K Wireless –depends on material Terrain & Environment –voids: mansize –dirt, rubble,inclines: depends –water resistant –not intrinsically safe www.foster-miller.com FM Solem video, striper, audio

20. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)20 Need Sensing of State of the Robot, World Need state of robot –pose, size relative to the environment –health some UIs display fault detection is hard diagnostics and recovery is slow (replace or 35 minutes) State of World –topological vs. metric –3D mapping –3D interpretation video overlay structural reasoning robot projected a grid to estimate distance WTC 4, 9/16/01

21. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)21 Search and Structural Assessment of Collaterally Damaged Buildings 3 types (military hostage rescue, special order research, TRL 4-6) ~$30-45K –backpackable by 2 people, 1 for bot, 1 for OCU & batteries, 12 hours (UrBot), 20min (PackBot) –fast, stairs, grasping –self-righting or invertible –can add sensors, payloads, software Wireless Terrain & Environment –buildings, large voids, 3 story drops –dirt, rubble,inclines, stairs –Packbot is water proof –not intrinsically safe 3 types (military hostage rescue, special order research, TRL 4-6) ~$30-45K –backpackable by 2 people, 1 for bot, 1 for OCU & batteries, 12 hours (UrBot), 20min (PackBot) –fast, stairs, grasping –self-righting or invertible –can add sensors, payloads, software Wireless Terrain & Environment –buildings, large voids, 3 story drops –dirt, rubble,inclines, stairs –Packbot is water proof –not intrinsically safe www.spawar.navy.milwww.spawar.navy.mil, www.irobot.comwww.irobot.com FM Talon video, audio, gripper iRobot PackBot video, FLIR, 2 way audio SPAWAR UrBot video, audio

22. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)22 General Mobility & Hardware Problems Lost 1 Solem robot –lost wireless comms, left in hole, wasn’t there later Damage –detracked once (high heat) –speared on rubble –just wear and tear Tethers tangle –only twice not immediately recoverable 7.75 “stuck assists” per drop (or once a minute) –but tether handling is significant 9.25 “gravity assists” per drop –still have tie a rope around the wireless ‘bots Wireless and dropouts –can’t compress and do image processing –too numerous/duration to count –make it hard to do image proxy processing Fear of flipping and getting stuck

23. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)23 HRI Issues Overall Warning: camera occlusion 12.3% of a drop –teleop doesn’t work well in those cases! Operator errors (Norman ‘91) –Mistakes 2: wrong robot, had to remove and try another –microTracks bulldozes in deep dirt, can’t climb –microVGTV too light to get traction 10% of duration of Inuktun runs spent significant time adjusting lights despite auto gain –need image enhancement –Slips 0.25 collisions per drop (oversteer) 8.9% of duration, robot wheel slip (high centered, wrong configuration) Human-Robot Interfaces scared end-users off

24. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)24 iRobot: PackBot (experimental) game joystick plus laptop with video & audio robot state: battery, comms, orientation, camera, encoders scared off rescuers: too complicated, too long to train iRobot PackBot video, FLIR, 2 way audio 2 people, 35 kg

25. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)25 9:00PM 11:00PM 1:00AM 3:00AM 5:00AM 7:00AM 65min ~355min ~6.5min 35min ~5min ~105min ~30min 1.Drive to Ground Zero from Javits Center 2.Parts of 9 member group splits off while the rest waits 3.2 operators, 2 robots and VATF-2 search void; MicroTrac failure 4.2 operator, 2 robots and VATF-2 search same void; VGTV failure 5.2 members retrieve spare robot from Javits Center 6.2 members fix 2 robots using parts from spare robot 7.9 member group moves to new location to wait 8.Group returns to Javits Center from Ground Zero 12:00AM 2 operators return with robots after VATF-2 retreats In route Waiting Searching Timeline of Shift w/ VATF-2 (7:00PM 9/18/01 – 7:00AM 9/19/01) 13 minutes for entire shift (green) No drop >7 minutes <1.5 minute set up time or rescuers walk away Found 3+ victims

26. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)26 Other problems: Asynergism Lack of interoperability / “on the other robot” –Image processing, intelligent assistance –Software, sensors couldn’t be migrated between platforms (“plug and perceive”) –Only the “dumbest” robots were well-used Known capabilities “lost” or “back in the lab” robots as wireless repeaters self righting self reacquistion of comms signal general office navigation, obstacle avoidance Demo, hardware focus, rather than Systems focus

27. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)27 Problems as Point of Departure for Theory USAR robots can be generic, adapted in the field for the particular task, thought of a system Sensors and sensing need to be improved, miniaturized, $$ –State of robot: highly sensate robots, health monitoring –State of world: 3D maps plus understanding structure –State of available information Mobility and hardware need to be improved –Polymorphic and elephant trunk sensors masts –Teams of robots marsupialism for delivery, relay, proxy processing physically coupled: tether managers, in-line collaborative teleoperation distributed: hybrid tethers Human-robot Interaction needs to be improved –Perceptual user interfaces not well-defined, mapping & human models of situational awareness –Training and access is an issue; over the internet

28. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)28 Robot-Assisted USAR as IT

29. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)29 Summary of Performance Robots were successfully used at the World Trade Center –Success: quick response (luck) –effective performance 12/11 equal to manual technical search for 24/10 days user acceptance –Training in Dec. 02 for East Coast firefighters –Training in Apr. 02 at FDIC in Indianapolis –~30 copies of data sets to fire rescue teams to date USF as a university provided expertise from field research, managed data collection (and pushing data dissemination)

30. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)30 However, We Could’ve Done More… Had more robots, sensors, and people than were utilized due to credentialing, organizational issues –Lost days 3,4 –Never saw a 24/7 deployment cycle –Only 75% of the available Inuktuns were deployed at any given time, despite effectiveness –Buildings could have been surveyed, if necessary, stored on videotape Capabilities never exploited –Robots could place tubing to transport air/water/meds to survivors Dr. Eric Rasmussen, 3rd Fleet Surgeon, sent medical equipment –Added air quality monitors on larger robots after 9/25 –AI Software on larger robots not interoperable with smaller and newer robots

31. Feb 19, 2002Robot-Assisted Search and Rescue at WTC Disaster (NSF)31 more information and public video & stills: www.crasar.orgwww.crasar.org murphy@csee.usf.edu Robots Can’t Replace the Real Heroes