Unified Robotic Software Development using CLARAty Issa A.D. Nesnas Mobility and Robotic Systems Section Autonomous Systems Division July 20, Briefing to the Office of the Chief Technologist Sponsors: Mars Technology Program Intelligent Systems Program Theme: The Long Road to Technology Infusion

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 2 Presentation Overview Historical Antecedents Problem and Objectives What is CLARAty? Interoperability Success Stories –Navigation –Single-cycle Instrument Placement Challenges of Software Interoperability Challenges in Technology Infusion Architecture Overview Challenges Ahead

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 3 Some JPL Robots

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 4 Historical Antecedents Late 80’s - Early 90’s: parallel robotic developments –RSI, MOTES, Satellite Servicing, Robby, Mircorover –No shared hardware or software Mid 90’s: Mars rover research centralized with Rocky 7 –First flight rover Late 90’s: Expansion and diversification of rover work –No software interoperability (Rocky 7, FIDO, Athena, DARPA) –Autonomy demonstration of Remote Agent Experiment (ARC and JPL) –MDS investigates reusable software for spacecraft control. ‘99-Early 00: Exploration Technology Program develops concept for a unifying autonomy architecture –Unifying autonomy and robotic control –Started the CLARAty task Today: –Unification of several robotic developments at JPL, ARC, and CMU –Two flight rovers with several new robotic capabilities

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 5 Problem and Approach Problem: –Difficult to share software/algorithms across systems –Different hardware/software infrastructure –No standard protocols and APIs –No flexible code base of robotic capabilities Objectives –Unify robotic infrastructure and framework –Capture and integrate legacy algorithms –Simplify integration of new technology –Operate heterogeneous robots

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 6 What is CLARAty? CLARAty is a unified and reusable software that provides robotic functionality and simplifies the integration of new technologies on robotic platforms A tool for technology development and maturation

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 7 Stereovision JPL_STEREO Interoperability: Software & Hardware VxWorks Linux QNX CLARAty Reusable Software Robot Adaptation Pose Estimation MER_SAPP Obstacle Avoidance MORPHIN Obstacle Avoidance MORPHIN Obstacle Avoidance MORPHIN GESTALT Drivemaps Stereovision JPL_STEREO Stereovision JPL_STEREO Pose Estimation MER_SAPP Pose Estimation MER_SAPP Pose Estimation MER_SAPP SRI Stereo ARC Stereo Sojouner Pose FIDO 3DEKF 6D EKF CAPABILITY: Navigation

Interoperability Success Stories  Navigation  Single-cycle Instrument Placement

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 9 Capturing Flight Algorithms: MER GESTALT on FIDO

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 10 Navigation Interoperability on FIDO and Rocky 8 Complex Algorithms on different Platforms I/O, motion control Trajectory Generation Rough Terrain Locomotion Odometry Pose Estimation Stereo Processing Visual Odometry Navigation (Morphin) –Obstacle avoidance –Path Planning

Interoperability Success Stories  Navigation  Single-cycle Instrument Placement

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 12 Single-cycle Instrument Placement Changes in FOV (a) Target (b) Designated Target Target Tracking time = t2 (avoiding an obstacle) time = t1 1 st Frame 37 th Frame after 10 m

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 13 FALCON Visual Target Tracker on Rocky 8 Integration of Complex Algorithms I/O, motion control Trajectory Generation Rough Terrain Locomotion Odometry Pose Estimation Stereo Processing Visual Odometry Obstacle avoidance Mast Control Visual Tracking

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 14 Integrated Single-Cycle Instrument Placement

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 15 CLARAty Framework for Single-cycle Instrument Placement 2D/3D Visual Tracking R8 Mast Pointing Morphin Navigator KIM Hand-off MSL Base Placement HIPS Visual Odometry Maestro Ground System Visual Tracker Visual Odometry SIFT Tracker Gaze Pointing K9 Mast Pointing Locomotor R8 Locomotor Wheel Odometry Pose Estimator GESTALT Drivemaps FIDO EKF 6DOF EKF Visual Odometry Camera Hand-off Mesh Registration Bundle Adjustment Base Placement Haz Camera Tracking 2D/3D Visual Tracking SIFT Tracker Obstacle Avoider Possible Alternatives Vision-guided Manip On -board Rover Software Infrastructure

Challenges in Interoperability  Mechanisms and Sensors  Hardware Architecture  Software Algorithms

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 17 Different Mobility Mechanisms Front x y z C (a) Skid Steering (no steering wheels) Frontx y z C (d) Partially Steerable (e.g. MER, Sojourner, Rocky 7) Front x y z C (e) All wheel steering (e.g. Rocky8, FIDO, K9) Front x y z C (b) Two –wheel steering

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 18 Different Sensors and Appendages Custom Analog Sun Sensor 3 Accels z-axis gryo 6 DOF IMU 4 DOF Arm 4 DOF Mast 2 DOF Arm 3 DOF Mast Camera Sun Sensor

Challenges in Interoperability  Mechanisms and Sensors  Hardware Architecture  Software Algorithms

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 20 Centralized Hardware Architecture FIDO Actuator/Encoders Potentiometers PID Control in Software Video Switcher IMU RS232 Serial PC104+ x86 Arch Framegrabbers Digital I/O Analog I/O Wireless Ethernet

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 21 Distributed Hardware Architecture Compact PCI - x86 Arch - Wireless E/net FireWire - I2C Bus Actuator/Encoders Potentio- meters I2C 1394 Bus IMU Sun Sensor Rocky Widgets Single-axis controllers Current sensing Digital I/O Analog I/O RS232 Distributed Motion Control and Vision Rocky 8

Challenges in Interoperability  Mechanisms and Sensors  Hardware Architecture  Software Algorithms

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 23 Software Challenges for Algorithm Infusion The new algorithms to be integrated may: Have architectural mismatches with the framework Include multiple orthogonal functionalities Make implicit assumptions about the platform Duplicate functionality in the framework Use incompatible data structures Are complex and hard to tune Require highly specialized domain expertise Are poorly implemented

Architecture and Process

THE DECISION LAYER: Declarative model-based Global planning CLARAty = Coupled Layer Architecture for Robotic Autonomy INTERFACE: Access to various levels Commanding and updates THE FUNCTIONAL LAYER: Object-oriented abstractions Autonomous behavior Basic system functionality A Two-Layered Architecture Adaptation to a system

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 26 Multi-level Abstraction Model Use abstractions Interface at different levels Input / Output (AI, AO, DIO) Motor Motor Group Wheeled Locomotor Rover Locomotor Example of Levels of Abstraction for locomotors FIDO HW Rocky 8 Firmware/HW ATRV SW/HW Pluto SW/HW

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 27 Technology Tasks Collaborations & Interactions CLARAty Jet Propulsion Lab CMU NASA ARC U. Minnesota R&TD, MDS, DRDF Competed Mars Technology Program Other NASA Programs Rover Simulation ROAMS Rover Hardware JPL Internal Programs Flight Focused Technology Programs Science Instruments Simulation Operator Interface Legacy Algorithms Flight Algorithms NASA Centers and Universities Technology Tasks NASA Centers and Universities Technology Tasks NASA Centers and Universities Technology Tasks NASA Centers and Universities Technology Tasks Technology Validation Tasks Technology Tasks Technology Tasks

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 28 CLARAty Test Bed for Regression Testing ATRV Jr. FIDO2 Stack Dexter Manipulator Bench top Rocky 8 PPC Bench top

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 29 Challenges Ahead Mature framework and robotic capabilities Investigate relevance to flight and define migration path Develop regression tests for long-term maintainability of robotic capabilities - very hard and open research topic Maintain current capabilities on existing platforms Develop new capabilities (e.g. continuous motion) Integrate new technologies from competed programs Develop a releasable version Develop formal documentation and tutorials Identify and deploy on low-cost rover platform Open source

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 30 Summary Developed a unified and reusable software framework Deployed at multiple institutions Deployed on multiple heterogeneous robots Integrated multiple technologies from different institutions Delivered algorithms for formal validation Enabled new technology developments on multiple platforms Integrated flight algorithms for detailed performance characterization and operation on research rovers. Taking a technology from inception, to development in CLARAty, to validation, and now to integration into flight

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 31 Current CLARAty Core Team NASA Ames Research Center –Clay Kunz –Eric Park –Susan Lee Carnegie Mellon University –David Apelfaum –Nick Melchior –Reid Simmons University of Minnesota –Stergios Roumeliotis Jet Propulsion Laboratory –Antonio Diaz Calderon –Tara Estlin –John Guineau –Won Soo Kim –Richard Madison –Michael McHenry –Mihail Pivtoraiko –Issa A.D. Nesnas –Babak Sapir –I-hsiang Shu OphirTech –Hari Das Nayar Full Credits for all Developers and Contributors at:

Questions?

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 33 Rocky 7 Distributed Software Development AFS Backbone CMU JPL CLARAty 3rd Party Web UW... VxWorks ARC FIDO K9 Authentication U. Minnesota Rocky 8 Number of employees and not FTEs

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 34 Unify Mechanism Model Ground_Body Body 4 Body 5 Body Tree Body 0 Body 2 Body 3 Body 1 Mechanism Tree Body Reference Frame Sensor Mount Frame Arm mount Frame Center of mass Camera Mount Frame Body i B1B1 Joint i Bounding Shape Tree B2B2 B3B3 B3B3 B4B4 B5B5 C1C1 B1B1 B2B2 B3B3 B4B4 C1C1 B5B5 Coarse Shape Finer Shape Finest Shape Leaves of tree define finest shape Relative to body reference frame Bounding Shapes Resolution Levels Bodies and Joints Body 1 Reference Frame Body 1 CG Body 1 Upper arm link Rover Reference Frame Rover CG Camera Frame Body 0 Rover Joint 1 Body 2 Lower arm link Sensor Mount Frame Camera Mount Frame Body 2 CG Articulated Rotation Body 2 Reference Frame Fixed Transform Arm Mount Frame Shoulder Yaw Joint 2 Articulated Translation

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 35 Some CLARAty Statistics ~320 modules in repository (increase of 6% from FY03) – goal is to limit modules ~60 modules are researched technology algorithms (~20%) About 500,000 lines of C++ code – revise and reduce Five adaptations: Rocky 8, FIDO, Rocky 7, ATRV, K9 Most technology modules are at Level III None are at Level IV or Level V (formally reviewed, documented, and open source) CLARAty Integration Levels Level I – Deposited Level II – Encapsulated Level III – Refactored Level IV – Formally reviewed Level V – Open source and fully documented

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 36 Relevance to the Missions Why is this work relevant to the missions? Provides a common environment for development, test, and comparison of advanced robotic technologies Provides an infusion path for robotics technologies into flight missions Demonstrates technologies on relevant robotic systems Makes research rovers viable test platforms for flight algorithms (e.g. navigation) Is robust to changes in rover hardware designs Can be easily adapted to flight and new research rovers

July 20, 2005 Briefing to the Chief Technologist - I.A.N. 37 Measuring Success or Failure We succeed IF we: Significantly reduce integration time of new technology software onto real robotic systems Support multiple platforms with different hardware architectures Provide a service that is enabling for technologists Simplify the development/integrate/debug/test cycle for current and next generation NASA rovers Have people other than the developers using and “like” the system