Human Performance Contributions to Safety in Commercial Aviation Presented by Jon Holbrook, PhD National Aeronautics and Space Administration February 2019

What is NASA’s role in civil aviation? NASA Aeronautics goal: Support safe, efficient, flexible, and environmentally sustainable air transportation Guidance for technology/concept feasibility and maturation Recommendations for design of systems, policies, & procedures Tools to support performance assessment, VV&C NASA maintains cooperative relationships with Aeronautics industry (e.g., manufacturers, operators) Government agencies concerned with operations and regulations (e.g., FAA) Academic community engaged in aerospace research

Aviation is a data-driven industry We (rightly) want to make data-driven decisions about safety management and system design. The data that are available to us affect how we think about problems and solutions (and vice versa). In current-day civil aviation, we collect large volumes of data on the failures and errors that result in incidents and accidents, BUT…

Aviation is a data-driven industry We rarely collect or analyze data on resilient behaviors that result in successful outcomes. We currently make safety management and system design decisions based on a small sample of non- representative data As a consequence, many in the industry are making some dubious assumptions, such as: The safest future state is the one that removes humans from the safety decision making loop.

(Some of the) Safety-II challenges in aviation Overarching challenge: Safety-I thinking is so ingrained in current practices and vocabulary that it effectively filters or excludes data from consideration that, from a Safety-II perspective, would be relevant.

(Some of the) Safety-II challenges in aviation Convincing policy makers that there is a problem with thinking about safety exclusively in terms of adverse outcomes How can we demonstrate the magnitude of the problem? How can we show that this problem is relevant to policy makers? Identifying data-driven solutions to the problem How can we characterize operators’ resilient performance? Where can we get data on operators’ resilient performance?

1. Demonstrate magnitude of the problem Human error implicated in 80% of accidents in civil aviation (Weigmann & Shappell, 2001) Pilots intervene to manage malfunctions on 20% of flights (PARC/CAST, 2013) World-wide jet data from 2007-2016 (Boeing, 2016) 244 million departures 388 accidents Outcome Not Accident Accident No ? ? ? Attributed to Human Intervention Yes 20% 80% ? ? 388 244,000,000

1. Demonstrate magnitude of the problem When we characterize safety only in terms of errors and failures, we ignore the vast majority of human impacts on the system. Outcome Not Accident Accident No 195,199,690 78 195,199,768 Attributed to Human Intervention Yes 48,799,922 310 48,800,232 243,999,612 388 244,000,000

2. Show problem is relevant to policy makers Push for machines to take on tasks and responsibilities currently performed by humans Without understanding how humans contribute to safety, any estimate of predicted safety of autonomous capabilities is incomplete and inherently suspect. Provides the necessary basis for designing machine systems that could perform safety-producing behaviors Push for development of in-time safety monitoring, prediction, and mitigation technologies Requires a more complete understanding of safety – both Protective (Safety-I) and Productive (Safety-II)

3. Characterizing resilient performance Lots of failure taxonomies, few success taxonomies “Positive” taxonomies largely focused on positive outcomes (e.g., flight canceled/delayed, rejected takeoff, proper following of radio procedures) Can we use Safety-II/Resilience Engineering approaches to identify “universally desired” behaviors, regardless of subsequent outcomes? Can we identify “language” of resilience? How do operators in YOUR domain talk about resilient performance? Behaviors are complex, and occur within a rich context How can we systematically capture “situated” performance without losing that richness?

3. Characterizing resilient performance No single data source can provide all of this information Strategy Resilience Capability: Anticipate, Monitor, Respond, Learn Actors / Interactions: Crew, ATC, Dispatch, Ground Ops, Airline… Context: External & Internal Objectives: Intentions, Goals, Pressures Resources: Tools & Knowledge (Adapted from Rankin, et al., 2014) is a function of is an action of type is an action by Observable Behavior: Direct & Indirect manifests as

4. Sources of resilient performance data: Data that are collected but not analyzed What data are currently available? Operator-, observer-, and system-generated Access challenge How and why are those data collected? Sunk cost challenge Happenstance reporting challenge How and why are those data analyzed? Implications for post-hoc coding Big-data challenge, and the need for tools to support analysis of narrative data There is no silver bullet Fusing data into a coherent picture De-identification challenge

4. Sources of resilient performance data: Data that are not collected but could be For system-generated data, what parameters could be captured that aren’t (simply because nobody thought to capture them)? For observer-generated data, what could observers be trained to record that they aren’t? For operator-generated data, what questions could be included on the data forms that aren’t? For operator-generated data, how do we overcome non- reporting bias (e.g., being resilient is part of my everyday job – why would I report that)?

Recommendations Redefine safety in terms of the presence of desired behaviors and the absence of undesired behaviors. Leverage existing data to identify strategies and behaviors that support resilient performance. Develop tools to capture new data on strategies and behaviors that support resilient performance. Observer-based, operator-based, system-based Develop a system-level framework for integrating across data types to facilitate understanding of work-as-done Develop organization-level strategies that promote recognition and reporting of behaviors that support resilient performance.

Questions or Comments? jon.holbrook@nasa.gov 757-864-9275