Contingency Software in Autonomous Systems Stacy Nelson, Nelson Consulting/QSS Robyn Lutz, JPL/Caltech & ISU SAFE Terminate Flight This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, and at NASA Ames Research Center, under a contract with the National Aeronautics and Space Administration. The work was sponsored by the NASA Office of Safety and Mission Assurance under the Software Assurance Research Program led by the NASA Software IV&V Facility. This activity is managed locally at JPL through the Assurance and Technology Program Office OSMA Software Assurance Symposium July 20-July 22, 2004

Overview –Goals –Technology Readiness Level –Availability of Data Approach Preliminary Results Work-in-progress Benefits –Potential Applications –Barriers to Research or Application Future Work Topics Contingency Software in Autonomous Systems

Video from Camcorder Video from Color Camera Video from tracking camera on trailer Virtual rotorcraft following APEX plan (green bar) Apex plan DART DEMO

Adding intelligent diagnostic capabilities by supporting incremental autonomy Responding to anomalous situations currently beyond the scope of the nominal fault protection Contingency planning using the SAFE (Software Adjusts Failed Equipment) approach Unique Research Relevant to NASA Contingency Software in Autonomous Systems

Mitigate failures via software contingencies resulting in safer, more reliable autonomous vehicles in space and in FAA national airspace –Enhance diagnostic techniques to identify failures –Provide software contingencies to mitigate failures –Perform tool-based verification of contingency software –Apply results to ARP (Years 1 & 2) and MSL (Years 2 & 3) Status: Year 1 of planned 3-year study (1/04 start) Overview Contingency Software in Autonomous Systems Current Practice SW Contingency Planning Full Autonomy

Current technology readiness level = 2+ –2: “Technology concept and/or application formulated” – completed 6/04 –3: “Analytical and experimental critical function and/or characteristic proof-of-concept” – in-progress (12/04 completion) Current penetration factor = 8 –Data passed back to project Contingency Software in Autonomous Systems Technology Readiness Level

Contingency Software in Autonomous Systems Availability of Data: High

Contingency Software in Autonomous Systems Problem Failure WHAT FAILED? Autonomous vehicles have limited capacity to identify/mitigate failures

Contingency Software in Autonomous Systems Enhance diagnostic techniques to identify failures Provide software contingencies to mitigate failures Perform tool-based verification of contingency software and Apply results to ARP (and MSL) to pave the way to more resilient, adaptive unmanned systems Approach SAFE Vehicle (Software Adjusts Failed Equipment) Flight Critical Parameters Failure Diagnosis Failure 1 2 3

ARP Functional Requirements: Current Planned Contingency Analysis: SFMECA SFTA Contingency Planning: Available indicators Contingency triggers Contingency responses 2-Level (recover/predict) CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Contingency Process Overview Customized the IEEE/EIA Annex I Evolutionary/Spiral Methodology 1. Brainstorm with UAV team to uncover candidates for software contingencies Review UAV literature and project reports Lead brainstorming sessions with domain experts Work with team to identify and prioritize high-concern candidates Select top priority candidates 2. Model unit of interest (i.e. cameras, communications systems…) Model system including: Architecture & State diagram Verify models with UAV team 3. Contingency requirements verification Perform SFMECA 4. Analyze testability Identify how each contingency can be detected Perform SFTA Experiment with assignment of measure of uncertainty 5. Develop recovery strategy Determine candidate strategies for contingency responses (prevent/respond/safe) Determine availability of data needed to determine/execute appropriate contingency 6. Prototype contingency in progressively higher fidelity testbeds 7. Monitor contingency performance

Design of Hybrid Mobile Communication Networks for Planetary Exploration Richard Alena, John Ossenfort, Charles Lee, Edward Walker, Thom Stone CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Related WLAN Work RF signal strength measurements can be normalized to theoretical values and used to predict range ( Good correlation and repeatability of signal strength measurements using different antenna configurations and test distances) Network throughput is reasonably predictable for single hop links at short distances (WLAN link runs under nominal conditions with no packet loss) However, network throughput is not predictable for complex WLANs consisting of multiple repeater hops or long distances. WLAN links run under conditions of varying packet loss. Packet loss significantly reduces data pipelining by introducing highly variable packet transfer latencies due to packet re-transmission RF signal strength measurements can be normalized to theoretical values and used to predict range ( Good correlation and repeatability of signal strength measurements using different antenna configurations and test distances) Network throughput is reasonably predictable for single hop links at short distances (WLAN link runs under nominal conditions with no packet loss) However, network throughput is not predictable for complex WLANs consisting of multiple repeater hops or long distances. WLAN links run under conditions of varying packet loss. Packet loss significantly reduces data pipelining by introducing highly variable packet transfer latencies due to packet re-transmission Packet loss due to multi-path, low signal strength, interference significantly disrupt the timing of packet transfers due to packet re-transmission. MAC layer uses packets for many purposes such as node authentication, data flow management and data transfer. Packet loss can affect any of these functions resulting in a wide variety of failures.

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Perception (Cameras) Perception is a critical function in systems requiring obstacle avoidance, threat detection, science missions and “opportunistic” discovery. Optical flow systems use contrasts in the surrounding imagery to determine position. If a vehicle using optical flow flies, for instance, over a very regular terrain such as a grassy field or an empty parking lot, it may crash.

Rotorcraft Control Center (“Trailer”) Rotorcraft Comm. Range (varies) Not to Scale CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Radio Modem b PCMCIA card Onboard Antenna GPS Autonomous flight (Nominal Case) (RC pilot standing by in case of emergency) Equipment New: Critical communications over radio modem and other communications via WiFi. Reason: Security and bandwidth

CLAW Flight Control Laws DOMS Distributed Messaging System GPS APEX Reactive Planner Telemetry CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Partial Onboard Architecture *domsD – DOMS transport daemon * Yamaha System

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Perception (Cameras) Perception is a critical function in systems requiring obstacle avoidance, threat detection, science missions and “opportunistic” discovery. Optical flow systems use contrasts in the surrounding imagery to determine position. If a vehicle using optical flow flies, for instance, over a very regular terrain such as a grassy field or an empty parking lot, it may crash.

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Cameras Onboard Rotorcraft Gray scale wing tip (stereo vision) Color Camcorder Color Camera for situational awareness Firewire Hub Image Processing System Firewire Left Wing Right Wing

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Other Perception Components Onboard Rotorcraft SIC (K) – Fast & accurate scanning laser Laser range finder – returns single point used for precision autonomous landing if GPS signal is lost Sonar (or Ultrasonic) range finder to determine distance to ground Sonar Range Finder Laser Range Finder (coming soon) GPS Scanning Laser Range Finder (SICK) (coming soon) Cameras

Cases in which the cameras are a critical system: 1.Cameras assigned responsibility during nominal ops No line of sight -> Camera provides position info 2.Cameras are backup when other subsystems fail Failed/degraded GPS -> Camera provides position info Failed/degraded ARP -> Camera provides landing-site data 3.Images as mission objective (surveillance) Failure of cameras can jeopardize success CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Camera Criticality

Collaborating with Autonomous Rotorcraft Project to experimentally apply approach Project provides feedback on our models, guidance on future plans –Feasibility check –Reviewed ARP architecture including communications & perception –Proposed initial SW contingencies for communication and perception failures ARP team including us in team meetings PM has agreed to try contingencies appearing viable Finalized SW contingencies for communications & perception with ARP team –ARP team considers further investigation & simulation high priority for 4 identified SW Contingencies CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Preliminary Results

Loss of Communication: Detect loss of communication revise mission plan: –Reroute –Fly to rally point Interference with Communication: WiFi Security Throttle back communication CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Preliminary Results Loss of Perception: Detect camera failure and reconfigure to use another camera –If color camera used for situational awareness fails, then switch to one of the gray scale cameras. –If left wing camera fails then reconfigure to use left wing color camera for stereo vision. Degradation of Perception: Change image-acquisition configuration or parameters –If need to lower resource usage, reduce image size Change image-transmission configuration or parameters –If need to lower bandwidth, drop color, drop frame rate, compress image more (trade off with CPU cycles)

Paves the way to more resilient, adaptive unmanned systems Supports spectrum of project adoption of autonomy –Flexible: p roject determines how much autonomy –Incremental requirements (evolutionary process model) Considers contingencies beyond failures: –Environmental changes that threaten mission (e.g., surveillance) –Changes in resource needs vs. availability that impact mission success (e.g., will need high-bandwidth) –Mobility capabilities that create tradeoffs with communication, imaging optimizations NASA Experience: Will demonstrate on NASA projects Anticipated cost savings for projects with evolving autonomy needs Equips us with a methodology to continue to move toward autonomy CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Benefits

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Towards MSL Risk Assessment for SW Contingencies Example Using DDP tool (fault tree Approach) to assess risk of SW Contingency Plans (collaboration between CSAS & Dr. Martin Feather) Note: example risk numbers relative not absolute – more work required

Autonomous Rotorcraft Project (ARC) Mars Science Laboratory (JPL) Other autonomous vehicles Other mobile imaging systems CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Potential Applications

Challenge 1: ARP is moving target (rapid evolution) Approach: Track planned & unplanned changes via weekly telecons Challenge 2: Planning for MSL application Approach: Demo benefits on ARP first; select ARP functionalities also important to MSL (communication, perception) Challenge 3: Tech transfer will depend on ease of reuse Approach: Provide results both in terms of (1) improved verification techniques for contingencies and (2) reusable designs for common contingency applications CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Barriers to Research or Application

Tool-based verification on NASA project Advance NASA’s information about communications and perception systems for autonomous vehicles CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Future Work