CLARAty: Coupled Layer Architecture for Robotic Autonomy Issa A.D. Nesnas Ames Research Center Carnegie Mellon University Jet Propulsion Laboratory University of Minnesota (FY03) CLARAty Workshop - January 13, 2003 Sponsors: Mars Technology Program Intelligent Systems Program

January 13, 2003 CLARAty Workshop - I.A.N. 2 Presentation Overview Architecture Overview Challenges of Interoperability Software Implementation and Status CLARAty testbed Selected Examples –Motion Control –Locomotion –Navigation Summary

Background and Overview

January 13, 2003 CLARAty Workshop - I.A.N. 4 What is CLARAty? CLARAty is a unified and reusable software that provides robotic functionality and simplifies the integration of new technologies on robotic platforms A research tool for technology development and maturation

January 13, 2003 CLARAty Workshop - I.A.N. 5 CLARAty Development Team CLARAty/Rocky8/Rocky7 –Max Bajracharya (34) –Edward Barlow (34) –Caroline Chouinard (36) –Gene Chalfant (34) –Hari Das (34) –Tara Estlin (36) –Dan Gaines (36) –Mehran Gangianpour (34) –Dan Helmick (34) –Won Soo Kim (34) –Michael Mossey (31) –Issa A.D. Nesnas (34) (Task Manager) –Ashitey Trebi-Ollennu (35/34) –Richard Petras (34) –Stergios Roumeliotis (Univ. of Minnesota) –Kevin Watson (34) NASA Ames Research Center –Maria Bualat –Clay Kunz –Randy Sargent –Anne Wright (lead) Carnegie Mellon University –Kam Lasater –Reid Simmons (lead) –Chris Urmson –David Wettergreen –Andy Yang Jet Propulsion Laboratory CLARAty on FIDO Team –Hrand Aghazarian (35/34) –Terrance Huntsberger (35/34) –Chris Leger (35/34) –Matthew Robinson (35/34) Five Summer Students

January 13, 2003 CLARAty Workshop - I.A.N. 6 Working Groups CLARAty Navigation Group (Lead: Reid Simmons - CMU) CLARAty Vision Group (Lead: Max Bajracharya - JPL) CLARAty Mobility & Manipulation Group (Lead: TBD / Hari Das (former)) CLARAty Estimation Group (Lead: Daniel Gaines - JPL) CLARAty Decision Layer Group (Lead: Tara Estlin - JPL) CLARAty Test/Release Group (Lead: TDB - JPL) CLARAty Rocky Rover Group (Lead: Richard Petras - JPL) K9 CLARAty Developers (Lead: Anne Wright - ARC) CLARAty Science Working Group (Lead: Ted Roush ARC/Meemong Lee - JPL) CLARAty Simulation Group (Lead: Gene Chalfant - JPL)

THE DECISION LAYER: Declarative model-based Mission and system constraints Global planning CLARAty = Coupled Layer Architecture for Robotic Autonomy INTERFACE: Access to various levels Commanding and updates THE FUNCTIONAL LAYER: Object-oriented abstractions Autonomous behavior Basic system functionality A Two-Layered Architecture Adaptation to a system

January 13, 2003 CLARAty Workshop - I.A.N. 8 Simulation Hardware Drivers Adapting to a Rover Generic Functional Layer Rocky 8 Specialized Classes & Objects Connector Decision Layer Multi-level access Connector Rocky 8 Models/ Heuristics

January 13, 2003 CLARAty Workshop - I.A.N. 9 The Decision Layer Executives (e.g. TDL) General Planners (e.g. CASPER) Activity Database Rover Models FL Interface Plans

January 13, 2003 CLARAty Workshop - I.A.N. 10 ASPEN/CASPER System Architecture Timeline Manager Activity info Conflict status ADB modifications (e.g., add, modify, delete) PCNTCN Network Constraint Propagation Activity Database Real-time System Rover Model Act, state & resource updates Activities Activity permission updates Interface / GUI Timeline updates Search commands Activity, timeline, & conflict info TDL Executive CASPER ASPEN Activities to execute Act status; updates Goals & Init State Domain-specific information Legend System Module Socket connection Can be socket connection or embedded Pathplanning (outside module) Heuristics RepairOptimize Search

January 13, 2003 CLARAty Workshop - I.A.N. 11 The Functional Layer Transforms Motion Control VisionEstimation Input/Output Manipulation Navigation Communication Math Hardware Drivers Locomotion Rover Behaviors Path Planning Rocky 8 Rocky 7 K9 FIDO Science Simulation Sensor Adaptations

January 13, 2003 CLARAty Workshop - I.A.N. 12 Functional Layer Components Pixel MastWheeledLocLeggedLocRBLoc Locomotor CoordMotors Socket Motor Camera Instrument BBMotorControlledMotor Analog_IODigital_IO IO VisualNavigatorVisualTracker Array_2D Vector LocationPoint Matrix Image ColorImage HTrans FeatureDetectorCorrelator Behavior State Database Bit Resource Rover SocketMsg Manipulator Arm StereoGen Specialized Data Structures General Purpose Data Structures Standard Template Library String LinkedList Container Waypoint Path Camera Image Location Connector

January 13, 2003 CLARAty Workshop - I.A.N. 13 Top Level Abstractions Rover_Impl Rover Sim_RoverR7_RoverR8_Rover R8_Mast R8_Power Gestalt_Nav D* Planner Static or Object Owned R8_Arm R8_Locomotor Mast Arm Power System Camera x N Path Planner Mapper Navigator Locomotor R8_Camera Optional Pose Estimator Connector

January 13, 2003 CLARAty Workshop - I.A.N. 14 Architectural Traverse Example Stereo Camera Stereo Engine > Navigator Gestalt Navigator Grid Mapper Mapper JPL Stereo Terrain Sensor Locomotor K9_Locomotor Global Cost Func D* Path Planner Stereo Processor Camera R Camera L Synchronous/or Asynchronous e.g. Rate Set at: 10Hz used by other activities Asynchronous > Asynchronous e.g. Rate Set at: 5 Hz > Asynchronous e.g. Rate Set at: 8 Hz EKF Pose Estimator Pose Estimator Path Information Path Planner K9 Rover Rover Decision Layer Commanding and State Updates

January 13, 2003 CLARAty Workshop - I.A.N. 15 Architectural Traverse Example Stereo Camera Stereo Engine > Navigator R7/Soj Navigator Mapper JPL Stereo Terrain Sensor Locomotor K9_Locomotor Tangent Graph Stereo Processor Camera R Camera L Synchronous/or Asynchronous e.g. Rate Set at: 10Hz used by other activities Asynchronous > Asynchronous e.g. Rate Set at: 5 Hz > Asynchronous e.g. Rate Set at: 8 Hz EKF Pose Estimator Pose Estimator Path Information Path Planner K9 Rover Rover Decision Layer Commanding and State Updates Obstacle Mapper

January 13, 2003 CLARAty Workshop - I.A.N. 16 Navigation Architecture * 1 1 Navigatorinterface Navigator Callback Locomotor interface Waypoint State Estimator Vehicle ModelMap Based SelectorFuzzy Logic SelectorRover Bug Selector Action Selector …

January 13, 2003 CLARAty Workshop - I.A.N. 17 Architectural Components Data Structure Components (DSC) –Array, Vector, Matrix, Map, Container, LinkedList, Bit –Image, Message, Resource Generic Physical Components (GPC) –Locomotor, Arm, Mast, Spectrometer, … Specialized Physical Components (SPC) –K9_Locomotor, K9_Arm, R8_Mast, FIDO_Motor,… Generic Functional Components (GFC) –ObjectFinder, LayerDetector, VisualNavigator, StereoProcessor, Localizer Specialized Functional Components (SFC)

January 13, 2003 CLARAty Workshop - I.A.N. 18 Architectural Components Impl Defining base abstract classes for the generic types –DSC - abstract Data_Obj class –GPC- Device class –GFC - Behavior class Additional core classes –Template Mz for measurement –Template Es for estimated values –Template State for state handling

January 13, 2003 CLARAty Workshop - I.A.N. 19 Guiding Principles Use domain knowledge to guide design Make all assumptions explicit Use abstraction layers to help master complexity Use information hiding to protect implementation variability Use interfaces and documentation to define behavior Avoid ineffective over-generalization Encapsulate system specific runtime models –Capabilities vary considerably across platforms –Hardware and hence runtime models vary across systems –Need for both cooperative and pre-emptive scheduling Least common denominator solutions are unacceptable in hardware/firmware/software interactions

January 13, 2003 CLARAty Workshop - I.A.N. 20 On Generic Algorithms Technologies that are generic by design should not be constrained by the software architecture & implementation Non-generic technologies should be accommodated on the appropriate platforms –Example (Generic): if you are working in navigation, you would not care about H/W architecture difference among different rovers –Example (Specific): if you are doing wheel/terrain interaction research, you might require specific hardware which one of the vehicles would support

Software Implementation & Status

January 13, 2003 CLARAty Workshop - I.A.N. 22 Repository Software Development Process AFS Backbone CMU JPL CLARAty Repository 3rd Party Releases Web UW... VxWorks ARC Rocky 8 FIDO Rocky 7 Benchtops K9 Repository ATRV Some CLARAty Statistics ~170 Modules (reusable entity) ~31 Packages (module grps) ~3 rovers ~250,000 lines of C++ code ~Java/scripts/ and models Authentication K9

January 13, 2003 CLARAty Workshop - I.A.N. 23 Infrastructure & Platform Support Object-oriented Architecture and Design Patterns C++ Implementation (with some Java) AFS backbone YaM for collaborative distributed development environment ACE for OS independence Qt for Graphics Display Operating Systems & Architectures:

January 13, 2003 CLARAty Workshop - I.A.N. 24 Levels of Software Integration Level I - Encapsulated –Uses CLARAty components to interact with rover –Does not support a CLARAty API –Runs on at least one robot platform Level II - Encapsulated & Integrated –Conforms to a generic CLARAty API (or parent class) Level II - Integrated –Has no unsupported 3rd party dependencies –Runs on all applicable rover platforms Level III - Refactored and Reviewed –Software reviewed by committee to ensure internal/external consistency –Uses all applicable CLARAty classes –Internally conforms to CLARAty conventions and coding standards Level + - Reused –Re-used by other modules in CLARAty - dependent module –Provides access to all internal data products

Challenges in Interoperability

January 13, 2003 CLARAty Workshop - I.A.N. 26 Solaris x86 Ames JPL Rocky 8 x86 VxWorks Linux Rocky 7 K9 ROAMS Currently Supported Platforms FIDO CMU Linux ATRV x86 JPL FIDO VxWorks x86 JPL ppc VxWorks JPL Linux

January 13, 2003 CLARAty Workshop - I.A.N. 27 Distributed Hardware Architecture x86 Arch Wireless ethernet 1394 FireWire I2C Bus Rocky 8 Compact PCI K9 Controllers  PIC Servos  Current Sensing  Digital & Analog I/O Actuator/Encoders Potentiometers I2C / Serial Bus 1394 Bus IMU RS232 Serial Sun Sensor K9 Rocky Widgets  Single Axis Controllers  Current Sensing  Digital I/O  Analog I/O

January 13, 2003 CLARAty Workshop - I.A.N. 28 Custom Architecture/Variability m68k Arch Framegrabbers Digital I/O Analog I/O Wireless ethernet Rocky 7 VME Arch Actuator/Encoders Potentiometers Parallel Custom Interface MUX/Handshaking Video Switcher Gyros Accels AIO PID Controllers

January 13, 2003 CLARAty Workshop - I.A.N. 29 Centralized Hardware Mapped Architecture x86 Arch Framegrabbers Digital I/O Analog I/O Wireless ethernet Fido PC104 Actuator/Encoders Potentiometers PID Control in Software Video Switcher IMU RS232 Serial

Selected Examples

January 13, 2003 CLARAty Workshop - I.A.N. 31 Examples of CLARAty Reusability Controlled_Motor_Impl ControlledMotor Linear_Axis Joint Non-Resuable Layer R8_Motor Fido_Motor Sim_Motor Trajectory Trajectory_Generator Mz HCTL_Chip Widget_Board Widget_Motor PID ControllerCounter DIO Analog Out Analog In MSI P460 MSI P430 MSI P415 MSI P430 Resuable HW reusable Non-reusable R7_Motor LM629_Chip R7_MC_Board

January 13, 2003 CLARAty Workshop - I.A.N. 32 Code Reusability

January 13, 2003 CLARAty Workshop - I.A.N. 33 Capabilities of Wheel Locomotor Type of maneuvers: –Straight line motions (fwd / bkwd) –Crab maneuvers –Arc maneuvers –Arc crab maneuvers –Rotate-in-place maneuvers (arc turn r=0) Driving Operation –Non-blocking drive commands –Multi-threaded access to the Wheel_Locomotor class – e.g. one task can use Wheel_Locomotor for driving while the other for position queries –Querying capabilities during all modes of operation. Examples include position updates and state queries –Built-in rudimentary pose estimation that assumes vehicle follows commanded motion

January 13, 2003 CLARAty Workshop - I.A.N. 34 Wheeled Locomotion – works for Rocky 8, Rocky 7, Fido, K9,... Front x y z C (a) Skid Steering (no steering wheels) x y z C (b) Tricycle (one steering wheel) Front x y z C (d) Partially Steerable (e.g. Sojourner, Rocky 7) Front x y z C (e) All wheel steering (e.g. MER, Rocky8, Fido, K9) Front x y z C (c) Two –wheel steering Generic Reusable Algorithms (f) Steerable Axle (e.g.Hyperion)

January 13, 2003 CLARAty Workshop - I.A.N. 35 Example: collaborative development for locomotor JPL CMU Future ARC Designed for Rocky 7 Used Motor class Separated wheel control from locomotion Built-in pose estimation Generalized design for wheeled locomotors Full and partially steerable vehicle Used generic motor classes Implements fixed axle model Developed continuous driving Adapted to Rocky 8, Rocky 7, and Sim Version 1.0 Version 2.0 Separated model from control Add separate locomotor state Add concept of wheel and steerable wheel, Drive Cmd, Drive Sequence Adapt to ATRV, Sim, Rocky 7, Rocky 8 Version 3.0 Redesign/ mature Version 4.0 Use device and telemetry infrastructure Add adaptation to K9 Add JPL

January 13, 2003 CLARAty Workshop - I.A.N. 36 Analysis of amount of resuable code across implementations: -Reusable131Point 2D Reusable2080Controlled Motor -Reusable1083Matrix, Vector, Array Rotation Matrix, Point 2DReusable341Location, Homogeneous Transforms -Reusable435Rotation Matrices 584 (non-reusable) Non-reusable Reusable Status Widget Motor, etc... Rocky 8 Motor - Vector Motion Sequence, 1D Solver, Homogeneous Transforms Depends On ~92%Total Reusable 6995Total 334Rocky 8 Motor 250Rocky 8 Locomotor 3561D Solver 540Motion Sequence 1445Wheel Locomotor Lines of CodeModule Code Resuability Results

CLARAty Status and Future Work

January 13, 2003 CLARAty Workshop - I.A.N. 38 Base Package Developed Generic Physical Component Base classes Math and Data Structure modules –Merging array/matrix between ARC/JPL. Added support for sub-arrays & STL iterators Bits and I/O modules (JPL) –New implementation of bits, DIO, & AIO - more consistent, reentrant & efficient Telemetry module (ARC) –Added telemetry modules to represent data and parameter sets, handles general serialization/deserialization, provides multi-threaded clients, & increases efficiency Device module (ARC) –Provide base classes for all devices such as cameras, arms, masts, locomotors, etc. Uses telemetry classes and provides generic interfaces for accessing telemetry Power System module (ARC) –Provides information on devices such as power sources, batteries, and battery chargers. Implemented specializations for K9 power hardware Converge implementations and adapt for all platforms Add Mz, Es, and State logging capabilities

January 13, 2003 CLARAty Workshop - I.A.N. 39 Developed algorithms for partially- steered and fully-steered vehicles Developed continuous driving capability (JPL) Demonstrated on Rocky8 & Rocky7, and FIDO (JPL) Third generation redesign (CMU) –Separated locomotor model from control –Added wheels, drive cmds and sequences. –Adapting R8, R7, ATRV, FIDO, and K9 –Tested on ATRV, R8, R7, & ROAMS Locomotion Package Front x y z C (d) Partially Steerable (e.g. Rocky 7) Front x y z C (e) All wheel steering (e.g. Rocky8, Fido, K9) (c) Steerable Axle (e.g.Hyperion) Continuous Driving and Separated Model from Control Converge with Device and refine APIand provide models for estimation Add ATRV adaptation to repository

January 13, 2003 CLARAty Workshop - I.A.N. 40 Vision Package Developed generic infrastructure for vision –Generic Camera, Stereo Camera, Camera Models, Image, Image operations (e.g. rectification, edge and corner detection), Image I/O, and Image transport Provided wrappers for vision algorithms –JPL Stereo, CAHVOR models, ARC Stereo, SVS Stereo, SLOG tracker Adapted package to various rovers –Rocky 8, K9, FIDO, Rocky 7 are currently using the same vision infrastucture Worked with Machine Vision Group to integrate visual odometry Worked with RMSA team, Olson, Roush, M. Lee Developed generic Image and Camera infrastructure Enhance operations/functionality for target tracking and stereovision Integration of Wide-baseline Stereo (U. Washington) Integration of Bundle Adjustment Techniques (JPL & Ohio State)

January 13, 2003 CLARAty Workshop - I.A.N. 41 Completed and tested FIDO-based EKF port to CLARAty (JPL) –Two tier Kalman Filter –Estimating heading (IMU + odometry), x, and y (odometry) Created second generation design for the Estimator module (JPL & Univ. of Minnesota) –Handle non-EKF based estimators –Designed to better integrate with Locomotor and other model infrastructure –Assume filter primary functions are to propagate state and update measurements Estimation Package 3DOF FIDO EKF integrated and tested in CLARAty Implement Framework for more general estimation algorithm+(6DOF) Integrate FIDO/R7 sun sensing algorithms

January 13, 2003 CLARAty Workshop - I.A.N. 42 Navigation & Mapper Package New modular design for the Navigator (CMU) –Based on generic action-selector - can be adapted to support navigation technologies –Specialized action selector to Map-based selector –Tightly integrated local and global cost functions (D*) –Navigator interfaces with Locomotor, Location Estimator, and Terrain Map generator. Refactored Gestalt to separate terrain evaluation from action selection (JPL) Developed infrastructure to process multi-tiered panoramic images for terrain evaluation (JPL) Acquired complete 3-D map of Mars Yard Developed Generic Navigation Infrastructure Integrate two navigation algorithms to exercise navigation framework

January 13, 2003 CLARAty Workshop - I.A.N. 43 Resurrected dexter platform and made part of CLARAty testbed (JPL) Upgraded manipulation modules to support acquisition of multi-tiered panoramic images (JPL) Manipulation Package Dexter Dexter Manipulation platform now part of CLARAty testbed Converge manipulation software between JPL and ARC Adapt to all manipulators (K9, R7 arm/mast, FIDO/Rocky 8/Dexter arm/mast)

January 13, 2003 CLARAty Workshop - I.A.N. 44 Developed adaptation of the CLARAty controlled motor classes to interface with ROAMS simulation (JPL) Tested Locomotor sending commands to wheels which in, turn, send commands to controlled motors (JPL) Specialized Sim_Motors turn commands into streams for socket- based communication Drove Simulated Rocky 8 rover with new locomotor (CMU) Interface between DL/FL (rover-level and ROAMS) is also available Simulation Package CLARAty Locomotor talking to ROAMS via motor cmds Provide adaptations for inertial sensors, I/Os, cameras, and science instruments Test functionality such as pose estimation, navigation and planning using ROAMS. Provide API for integration with MSF

January 13, 2003 CLARAty Workshop - I.A.N. 45 Interactions with T. Roush’s team for integration of science analysis and spectrometer functionality Analysis code checked in the repository: –Carbonate Analysis –Edge Layer detection Interactions with Meemong Lee’s for interface with science instruments, analysis, and simulation. Code available in repository –Calibrated Reflectance Spectrum –Uncalibrated Spectrum –Spectrometer Science Package Infrastructure for Spectrometer and terrain analysis Provide adaptations to integrate with science instrument simulator and possibly real spectrometers Continue integration of science analysis

January 13, 2003 CLARAty Workshop - I.A.N. 46 Communication Package Decision Layer/Functional Layer communication module K9 (ITAR-free) communication interface Communication mechanism between adaptations of CLARAty classes and ROAMS Uses CLARAty socket module Interface between DLs and FL Provide easy and automated means for transporting CLARAty data structures (Data_Obj serialization and deserialization) Modularize communication components and improve reuse Complete study regarding use of TAO for DL/FL interaction

January 13, 2003 CLARAty Workshop - I.A.N. 47 Decision Layer Package Integrated D* Path Planner with Decision Layer Developed infrastructure for sharing terrain map with FL Migrating a version of CLEaR (CASPER/TDL - Linux) into CLARAty repository. Now available are: –TCM - Task Control Management used by TDL –Tangent Graph Path Planner (as used by DL) –D* Path Planner (as used by DL) –TDL and Java utilities Unifying DL dependencies to use the CLARAty modules (e.g. path planners - tangent graph, D*), etc. Decision Layer integrated D* Path Planner Converge more models between DL and FL Interface with WITS and enhance interface with FL Integrate another Decision Layer with Functional Layer (pending)

CLARAty Testbed

January 13, 2003 CLARAty Workshop - I.A.N. 49 FIDO Benchtop Rocky 8 Benchtop Dexter Manipulators Rocky 7 Benchtop CLARAty Testbed (1/2)

January 13, 2003 CLARAty Workshop - I.A.N. 50 CLARAty Testbed (2/2) 5 cPCI embedded targets (x86, ppc) 2 Linux and 5 Solaris hosts/targets AFS VxWorks installations (Tornado I, II, II.2, x86, ppc) Mockups for Rocky 8, FIDO, and Rocky 7 Remotely accessible to CLARAty developers & users Web-based target status and control Small lab sandbox built for indoor testing Several remote and local users exercising testbed (MDS, CMU, ARC, JPL) Online sign up for target/rover usage

January 13, 2003 CLARAty Workshop - I.A.N. 51 Summary CLARAty provides a repository of reusable software components at various abstraction levels It attempts at capturing well-known robot technologies in a basic framework for researchers It publishes the behavior and interfaces of its components It allows researchers to integrate novel technologies at different levels of the architecture It is a collaborative effort within the robotics community It will run on multiple heterogeneous robots

January 13, 2003 CLARAty Workshop - I.A.N. 52 Acknowledgements CLARAty Team (multi-center) Jet Propulsion Laboratory ROAMS/Darts Team CLEaR Team Instrument Simulation Team Machine Vision Team FIDO Team Ames Research Center K9 Team Carnegie Mellon University

Thank you for your Attention

CLARAty Software Packages

January 13, 2003 CLARAty Workshop - I.A.N. 55 Decision Layer Package General Planners  Basic planner infrastructure... Activity Database  Generic Activity database structure Planner Implementations  CASPER  ASPEN Executives –TCM - Task Control Management used by TDL –TDL - Task Description Language Interfaces  Functional Layer Interface ...

January 13, 2003 CLARAty Workshop - I.A.N. 56 Base Package Share  Common Definitions  Math Constants  Function Operators  Display Functions Array  1D Arrays  2D Arrays  Sub Arrays Matrix  1D matrices (math)  2D matrices (math)  Fixed size matrices  Rotation matrices Transforms  Homogeneous Transforms  Quaternions Frame  Coordinate Frames  Locations (Poses) Point  2-D Points  3-D Points Numeric Solvers  Newton Rhapson  Nedler-Mede Resource  Timers  Tasks... State  State data logging  Database Greyed Out – in progress Orange – non-JPL/non-CLARAty development Green – JPL-non-CLARAty development GPC & GFC  Device  Telemetry  Periodic Objects

January 13, 2003 CLARAty Workshop - I.A.N. 57 Input Output Package I/O  I/O  Digital I/O  Composite Digital I/O  Analog Input  Analog Output Bits  Bits data structure  Bit manipulation/masking

January 13, 2003 CLARAty Workshop - I.A.N. 58 Motion Control Package Motor  Open Loop Motors  Controlled Motors Trajectory  Trajectories  Trajectory Generators  Continuous Profiling... Coordinated Systems  Motor Coordinators

January 13, 2003 CLARAty Workshop - I.A.N. 59 Locomotion Package Wheel Locomotion  Wheels/Steerable Wheels  Locomotor models  Locomotor State  Drive Commands  Motion Sequences  2-D Wheel Locomotion algorithms  3-D Rocker Bogie Locomotion  Continuous driving modules Legged Locomotion  Legged Locomotors Hybrid Locomotion  Wheel/Legged Locomotors...

January 13, 2003 CLARAty Workshop - I.A.N. 60 Vision Package Camera  Framegrabbers  Cameras (single, multi)  Stereo Pairs  Camera models Data Structures  Images  Image Pyramids  Image I/O  Color Images  Camera Images  Point Clouds  Meshes Stereo Implementations  JPL Stereo engine  SRI Stereo engine (commercial)  Ames Stereo engine... Stereo Vision  StereoProcessor  StereoEngine Visual Tracking  2-D Feature extraction  3-D Feature extraction  Affine Tracking  2-D Tracking algorithms  Shaped based tracking  Combined 2-D & 3-D tracking  Visual odometry Basic Image Processing  2-D Image processing -Filtering & smoothing -Edge detection -Blob detection -Corner feature detection

January 13, 2003 CLARAty Workshop - I.A.N. 61 Estimation Package Kalman Filter Algorithms  3 DOF EKF pose estimation  6 DOF EKF pose estimation Pose Estimators  Wheel-based Pose Estimators  Legged-based Pose Estimators... Data Structure  Measurements  System models  Generic Kalman Filter

January 13, 2003 CLARAty Workshop - I.A.N. 62 Navigation Package Local Navigators  Action Selectors  Mapped-base Action Selector  Waypoints  Generic local navigator structure Path Planner  Generic path planner structure... Navigation Implementations  MER/GESTALT Navigator  Morphin (CMU) Navigator  Fuzzy Navigator

January 13, 2003 CLARAty Workshop - I.A.N. 63 Manipulation Package Manipulators  Joints  Links  Generic Manipulators  Generalized Fwd Kinematics  Generalized Inverse Kinematics  Limited DOF manipulators  Redundant manipulators  End effectors... Sensor-based Manipulation  Stereovision based manipulation  Vision/tactile manipulation

January 13, 2003 CLARAty Workshop - I.A.N. 64 Simulation Package Simulator Connectors  Interfaces to ROAMS simulation  Rover Sim  Navigator Sim  Locomotor Sim  Motor Sim  I/O Sim  IMU Sim... Simulation Implementations  ROAMS Rover Simulation (JPL)

January 13, 2003 CLARAty Workshop - I.A.N. 65 Sensor Package... IMUs  Generic inertial measurement unit infrastructure Tilt Sensors  Generic Electronic Tilt sensors  Generic Accelerometers Rate Sensors  Generic Gyros structures Temperature Sensors Voltage & Current Sensors

January 13, 2003 CLARAty Workshop - I.A.N. 66 Science Package Spectrometers  Generic Spectra classes –Calibrated Reflectance Spectrum –Uncalibrated Spectrum  Generic Spectrometer classes... Science Cameras  Special science cameras (inherits for Cameras of Vision Package) Analysis Algorithms  Carbonate Analyzer  Edge Layer detector Science Database

January 13, 2003 CLARAty Workshop - I.A.N. 67 Communication Package Ethernet  Socket Communication  TCP Comm  UDP Comm Messages  Object serialization  Socket messages Connector  Client/server comm.  Executives... Serial I/O  Serial Comm

January 13, 2003 CLARAty Workshop - I.A.N. 68 Path Planning Package Global Path Planners  Generic path planner structure... Path Planner Implementations  D* Path Planner (CMU)  Tangent Graph Path Planner (JPL)  ISE Path Planner (CMU)

January 13, 2003 CLARAty Workshop - I.A.N. 69 Behavior Package Behaviors  Basic rover behaviors... Behavior Coordination  Arbiters  Behavior combinators

January 13, 2003 CLARAty Workshop - I.A.N. 70 Hardware Drivers Motor Controllers HCTL 1100 Motor Controller LM 629 Motor Controller... I/O Boards/Chips VPAR10 – Digital I/O Board VADC20 – Analog Input Board Framegrabbers/Vision MAX186 Analog to Digital Conv. MAX528 Digital to Analog Conv. Imagenation PX610 Framegrabber Imagenation CX100 Framegrabber Custom Boards Widget Board – Motion Control Other I2C Master Controller Crossbow DMU – Interial Measurement Unit Sensoray S720 Digital I/O Board 1394 Camera Driver... VISA – Vme to ISA Converter Widget Board Firmware PCI Controller...

January 13, 2003 CLARAty Workshop - I.A.N. 71 Rocky 8 Package Rocky 8 Specialized Physical Components  Rocky 8 Motors  Rocky 8 Cameras  Rocky 8 IMUs  Rocky 8 Hardware Map  Rocky 8 Digital I/O  Rocky 8 Analog I/O... Rocky 8 Specialized Functional Components  R8 Locomotor  R8 Contrained Locomotor  R8 Navigator  R8 Rover

January 13, 2003 CLARAty Workshop - I.A.N. 72 K9 Package K9 Specialized Physical Components  K9 Motors  K9 Cameras  K9 IMUs  K9 Hardware Map  K9 Digital I/O  K9 Analog I/O  K9 Locomotor  K9 Contrained Locomotor  K9 Rover... K9 Specialized Functional Components  K9 Navigator

January 13, 2003 CLARAty Workshop - I.A.N. 73 Rocky 7 Package Rocky 7 Specialized Physical Components  Rocky 7 Motors  Rocky 7 Cameras  Rocky 7 Accels  Rocky 7 Gyros  Rocky 7 Hardware Map  Rocky 7 Digital I/O  Rocky 7 Analog I/O... Rocky 7 Specialized Functional Components  R7 Locomotor  R7 Contrained Locomotor  R7 Navigator  R7 Rover

January 13, 2003 CLARAty Workshop - I.A.N. 74 FIDO Package FIDO Specialized Physical Components  FIDO Motors  FIDO Cameras  FIDO Accels  FIDO Gyros  FIDO Hardware Map  FIDO Digital I/O  FIDO Analog I/O... FIDO Specialized Functional Components  FIDO Locomotor  FIDO Contrained Locomotor  FIDO Navigator  FIDO Rover

January 13, 2003 CLARAty Workshop - I.A.N. 75 Map Based Selector Implements Morphin/D* & Gestalt type algorithms Enables easy exchange of terrain evaluation and cost functions Combine cost functions through arbitration or summation

January 13, 2003 CLARAty Workshop - I.A.N. 76 GESTALT Navigator on Rocky 8

January 13, 2003 CLARAty Workshop - I.A.N. 77 Background & Objectives Why are we doing this work? To capture and preserve robotics expertise from JPL and other centers To provide a framework for future NASA rover technology development and integration To reduce the cost of integrating new technologies To operate various robots from a unified framework To eliminate cost of redeveloping basic robotic infrastructure

January 13, 2003 CLARAty Workshop - I.A.N. 78 Goals Capture and integrate a wide range of technologies Leverage existing tools Leverage experience and tools of the larger software development community Apply appropriate design patterns to the domain Provide an infrastructure that enables rapid robotic development Capture experience of technologists implementations

January 13, 2003 CLARAty Workshop - I.A.N. 79 How? Study software implementation of various robots Study interactions of elements in various systems Identify reusable/recurring elements in robotic systems Identify implicit assumptions made Project potential advances to these elements Design a generic/flexible implementation of these elements Adapt to a number of robotic systems Test and study the limitations of the design Go back to design and iterate Modify/extend/redesign to address limitations and variability across systems Your generic base is reusable

January 13, 2003 CLARAty Workshop - I.A.N. 80 Approach Domain knowledge guides design Layers of abstraction help master complexity Abstractions also provide a classification of various technology elements Information hiding protects implementation variability Small modular components are more reusable than monolithic blocks Interfaces define behavior of various elements

January 13, 2003 CLARAty Workshop - I.A.N. 81 Things to be aware of Over-generalizing leads to ineffectiveness –More general -> less functionality -> more work for results –Number of abstractions vs. complex hierarchies –Modular elements with strongly typed interfaces –Algorithm generality influences abstraction design Runtime models vary across systems –Challenges in combining hardware/firmware/software architectures in most effective manner –Need for both cooperative and pre-emptive scheduling

January 13, 2003 CLARAty Workshop - I.A.N. 82 Component Analysis State Objects Members Generic Physical Comp Estimator State Machines Object Services Creates Private Public Links to State Handler Sub-object Internal Implementation Local Estimation State 1 Queries HW Object - optional link

January 13, 2003 CLARAty Workshop - I.A.N. 83 Relationships with Other Components Locomotor change_position(Location& loc) change_velocity(double vel) set_position(Location& loc) set_velocity(double vel) get_position() get_velocity() _compute_angles() CoordMotionSystem _position: Vector _velocity: double _acceleration: double Provides trajectory planning based on a selection criteria possibly by the Decision Layer Functionality encoded within the objects Wheeled_Locomotor Legged_Locomotor Manipulator Leg * Wheel Steerable Wheel **

January 13, 2003 CLARAty Workshop - I.A.N. 84 R8 Specific Rover Implementation Mast LeggedLoc Wheeled Locomotor Locomotor CoordMotionSystem Motor BBMotor ControlledMotor Analog_IO Digital_IO IO Manipulator Arm R8_ArmR8_Mast Implements general fwd & inv. kinematics & joint ctrl Specialized inv. Kinematics (overrides default) Attaches proper motors Attaches proper cameras for mast Adds filter wheel R8_Locomotor Attaches proper motors Restricts Steering to 2 wheels R8_Motor R8_Rover R8_Arm R8_MastR8_Locomotor R8 Widget Board Widget AIOWidget DIO Widget Motor HCTL 1100 Chip Non reusable Code Reusable Code Trajectory Trajectory_Generator Timers

January 13, 2003 CLARAty Workshop - I.A.N. 85 R7 Specific Rover Implementation Mast WheeledLocLeggedLoc RBLoc Locomotor CoordMotionSystem Motor BBMotor ControlledMotor Analog_IO Digital_IO IO Manipulator Arm R7_ArmR7_Mast Implements general fwd & inv. kinematics & joint ctrl Specialized inv. Kinematics (overrides default) Attaches proper motors Attaches proper cameras for mast Adds filter wheel R7_Locomotor Attaches proper motors Restricts Steering to 2 wheels LM629Chip LM629Motor R7_Rover R7_Arm R7_MastR7_Locomotor R7 Device Drivers VPAR10Board Non reusable Code Reusable Code

January 13, 2003 CLARAty Workshop - I.A.N. 86 Example: Generic Controlled Motor Define generic capabilities independent of hardware Provide implementation for generic interfaces to the best capabilities of hardware Provide software simulation where hardware support is lacking Adapt functionality and interface to particular hardware by specialization inheritance Motor Example: public interface command groups: –Initialization and Setup –Motion and Trajectory –Queries –Monitors & Diagnostics Time Velocity S- Profile Instantaneous Profile Change

January 13, 2003 CLARAty Workshop - I.A.N. 87 Locomotor Design Original DesignNew Design

January 13, 2003 CLARAty Workshop - I.A.N. 88 Vehicle State Contains 6DOF pose Mechanism state contains variable vehicle parameters Mechanism information encapsulated in a separate class (bridging) Potential for dynamical state or other descendant state classes

January 13, 2003 CLARAty Workshop - I.A.N. 89 Estimation Package

January 13, 2003 CLARAty Workshop - I.A.N. 90 Navigator Public interface for navigation capability Provides a bridge between evaluation and execution Manages waypoints Multi-threaded –Interface available while executing a task Callback provides hook to trigger non- navigation events –Periodic science on the fly

January 13, 2003 CLARAty Workshop - I.A.N. 91 Action Selector Navigation algorithms provide this interface Used to determine the next motion the vehicle should take

January 13, 2003 CLARAty Workshop - I.A.N. 92 Maps Generic base class extends matrix Iterators allow movement through maps in arbitrary directions

January 13, 2003 CLARAty Workshop - I.A.N. 93 Nesnas/JPL DECISION LAYER FUNCTIONAL LAYER Terrain Map from Laser D* Path Planner Terrain Map from Laser Path Rover Navigator Position Estimator Estimator Kalman Filter Locomotor IMU R7_Rover R8_Rover R7_Locomotor R8_Locomotor Motor R7_Motor R8_Motor Widget Board Stereo Engine Camera CameraPX610 Camera1394 FUNCTIONAL LAYER Wheel Locomotor Model Camera_Image D* Star JPL Stereo Connector CLARAty Level I Milestone

January 13, 2003 CLARAty Workshop - I.A.N. 94 Publications & Website T. Estlin, F. Fisher, D. Gaines, C. Chouinard, S. Schaffer, and I. Nesnas, “Continuous Planning and Execution for an Autonomous Rover,” To appear in the Proceedings of the Third International Workshop on Planning and Scheduling for Space, Houston, TX, Oct 2002 I.A.D. Nesnas, “CLARAty - An Architecture for Reusable Robotic Software,” CSMISS IT Spotlight Series, June 19, 2002 I.A.D. Nesnas, R. Volpe, T. Estlin, H. Das, R. Petras D. Mutz, "Toward Developing Reusable Software Components for Robotic Applications" Proceedings of the International Conference on Intelligent Robots and Systems (IROS), Maui Hawaii, Oct Nov pdf (8 pages, 2MB) T. Estlin, R. Volpe, I.A.D. Nesnas, D. Mutz, F. Fisher, B. Engelhardt, S. Chien, "Decision-Making in a Robotic Architecture for Autonomy." Proceedings of 6 th International Symposium on Artificial Intelligence, Robotics, and Automation in Space (i-SAIRAS), Montreal Canada, June pdf (8 pages, 72KB) R. Volpe, I.A.D. Nesnas, T. Estlin, D. Mutz, R. Petras, H. Das, "The CLARAty Architecture for Robotic Autonomy." Proceedings of the 2001 IEEE Aerospace Conference, Big Sky Montana, March pdf (12 pages, 470 KB) R. Volpe, I.A.D. Nesnas, T. Estlin, D. Mutz, R. Petras, H. Das, "CLARAty: Coupled Layer Architecture for Robotic Autonomy." JPL Technical Report D-19975, Dec pdf (116 pages, 904 KB)

January 13, 2003 CLARAty Workshop - I.A.N. 95 FY03 Plans

January 13, 2003 CLARAty Workshop - I.A.N. 96 CLARAty Collaborations (1/2) JPL - MDS Team –Participated with MDS team on CLARAty/MDS infusion task –Supported MDS by upgrading Rocky 7 to MSL-like hardware and provided working version of CLARAty software for migration to MDS JPL - ROAMS Team (A. Jain) –Close collaborations for the integration of the Functional and Decision Layers to the ROAM rover simulation JPL - CLEaR Team (F. Fisher) –Provided an instantiation of the CLARAty Decision Layer, integrated, tested and debugged on Rocky 7 and Rocky 8 rovers JPL - Instrument Simulation Team (M. Lee) –Worked closely with instrument simulation team to develop spectrometer and spectra infrastructure for CLARAty.

January 13, 2003 CLARAty Workshop - I.A.N. 97 CLARAty Collaborations (2/2) RMSA Teams: –ARC - Science Analysis Team (T. Roush) Close interactions to support integration of science analysis into the CLARAty environment. –JPL - Slope Navigation (L. Matthies) Collaboration for integration of visual odometry on rover platforms –University of Washington (C. Olson & R. Li) Close collaboration for integration of bundle adjustment and wide baseline stereo –Carnegie Mellon University (A. Stentz) Integration of D* path planner for long range traverse –Ames Research Center (E. Bandari) Integration of visual tracking ALERT Team (former REE) –Providing CLARAty communication and rover control infra-structure to REE for a navigation demonstration on Rocky 8

January 13, 2003 CLARAty Workshop - I.A.N. 98 CLARAty Collaborations (1/2) JPL - MDS Team –Participated with MDS team on CLARAty/MDS infusion task –Supported MDS by upgrading Rocky 7 to MSL-like hardware and provided working version of CLARAty software for migration to MDS JPL - ROAMS Team (A. Jain) –Close collaborations for the integration of the Functional and Decision Layers to the ROAM rover simulation JPL - CLEaR Team (F. Fisher) –Provided an instantiation of the CLARAty Decision Layer, integrated, tested and debugged on Rocky 7 and Rocky 8 rovers JPL - Instrument Simulation Team (M. Lee) –Worked closely with instrument simulation team to develop spectrometer and spectra infrastructure for CLARAty.

January 13, 2003 CLARAty Workshop - I.A.N. 99 CLARAty Collaborations (2/2) RMSA Teams: –ARC - Science Analysis Team (T. Roush) Close interactions to support integration of science analysis into the CLARAty environment. –JPL - Slope Navigation (L. Matthies) Collaboration for integration of visual odometry on rover platforms –University of Washington (C. Olson & R. Li) Close collaboration for integration of bundle adjustment and wide baseline stereo –Carnegie Mellon University (A. Stentz) Integration of D* path planner for long range traverse –Ames Research Center (E. Bandari) Integration of visual tracking ALERT Team (former REE) –Providing CLARAty communication and rover control infra-structure to REE for a navigation demonstration on Rocky 8