Georgia Tech / Mobile Intelligence 1 Multi-Level Learning in Hybrid Deliberative/Reactive Mobile Robot Architectural Software Systems DARPA MARS Kickoff Meeting - July 1999

Georgia Tech / Mobile Intelligence 2 Personnel n Georgia Tech –College of Computing < Prof. Ron Arkin < Prof. Ashwin Ram < Prof. Sven Koenig –Georgia Tech Research Institute < Dr. Tom Collins n Mobile Intelligence Inc. < Dr. Doug MacKenzie

Georgia Tech / Mobile Intelligence 3 Impact n Provide the DoD community with a platform- independent robot mission specification system, with advanced learning capabilities n Maximize utility of robotic assets in battlefield operations n Demonstrate warfighter-oriented tools in three contexts: simulation, laboratory robots, and government-furnished platforms

Georgia Tech / Mobile Intelligence 4 New Ideas n Add machine learning capability to a proven robot- independent architecture with a user-accepted human interface n Simultaneously explore five different learning approaches at appropriate levels within the same architecture n Quantify the performance of both the robot and the human interface in military-relevant scenarios

Georgia Tech / Mobile Intelligence 5 Adaptation and Learning Methods n Case-based Reasoning for: –deliberative guidance (“wizardry”) –reactive situational- dependent behavioral configuration n Reinforcement learning for: –run-time behavioral adjustment –behavioral assemblage selection n Probabilistic behavioral transitions –gentler context switching –experience-based planning guidance Available Robots and MissionLab Console

Georgia Tech / Mobile Intelligence 6 AuRA - A Hybrid Deliberative/Reactive Software Architecture n Reactive level –motor schemas –behavioral fusion via gains n Deliberative level –Plan encoded as FSA –Route planner available

Georgia Tech / Mobile Intelligence 7 1. Learning Momentum n Reactive learning via dynamic gain alteration (parametric adjustment) n Continuous adaptation based on recent experience n Situational analyses required n In a nutshell: If it works, keep doing it a bit harder; if it doesn’t, try something different

Georgia Tech / Mobile Intelligence 8 2. CBR for Behavioral Selection n Another form of reactive learning n Previous systems include: ACBARR and SINS n Discontinuous behavioral switching

Georgia Tech / Mobile Intelligence 9 3. Q-learning for Behavioral Assemblage Selection n Reinforcement learning at coarse granularity (behavioral assem- blage selection) n State space tractable n Operates at level above learning momentum (selection as opposed to adjustment)

Georgia Tech / Mobile Intelligence CBR “Wizardry” n Experience-driven assistance in mission specification n At deliberative level above existing plan representation (FSA) n Provides mission planning support in context

Georgia Tech / Mobile Intelligence Probabilistic Planning and Execution n “Softer, kinder” method for matching situations and their perceptual triggers n Expectations generated based on situational probabilities regarding behavioral performance (e.g., obstacle densities and traversability), using them at planning stages for behavioral selection n Markov Decision Process, Dempster-Shafer, and Bayesian methods to be investigated

Georgia Tech / Mobile Intelligence 12 Integration with MissionLab n Usability-tested Mission-specification software developed under DARPA funding (RTPC/UGV Demo II/TMR programs) n Incorporates proven and novel machine learning capabilities n Extends and embeds deliberative Autonomous Robot Architecture (AuRA) capabilities Architecture Subsystem Specification Mission Overlay

Georgia Tech / Mobile Intelligence 13 Development Process with Mlab Behavioral Specification MissionLab SimulationRobot

Georgia Tech / Mobile Intelligence 14 MissionLab n Example: Scout Mission

Georgia Tech / Mobile Intelligence 15 MissionLab n EXAMPLE: LAB FORMATIONS

Georgia Tech / Mobile Intelligence 16 MissionLab Example: Trashbot (AAAI Robot Competition)

Georgia Tech / Mobile Intelligence 17 MissionLab Reconnaissance Mission –Developed by University of Texas at Arlington using MissionLab as part of UGV Demo II –Coordinated sensor pointing across formations

Georgia Tech / Mobile Intelligence 18 Evaluation: Simulation Studies n Within MissionLab simulator framework n Design and selection of relevant performance criteria for MARS missions (e.g., survivability, mission completion time, mission reliability, cost) n Potential extension of DoD simulators, (e.g., JCATS)

Georgia Tech / Mobile Intelligence 19 Evaluation: Experimental Testbed n Drawn from our existing fleet of mobile robots n Annual Demonstrations

Georgia Tech / Mobile Intelligence 20 Evaluation: Formal Usability Studies n Test in usability lab n Subject pool of candidate end-users n Used for both MissionLab and team teleautonomy n Requires develop- ment of usability criteria and metrics

Georgia Tech / Mobile Intelligence 21 Schedule