Management and Reduction of Human Error in ATM Validation of the JANUS Technique:causal factors of human error in operational incidents Julia Pounds Ph.D. and Anne Isaac Ph.D. CAMI/FAA DAS/HUM USA EUROCONTROL

Management and Reduction of Human Error in ATM Objective To develop a harmonised method providing a more consistent and effective approach for assessing human error causal factors in ATM incidents.

Two approaches HFACS HERA

Harmonization Lighter treatment by HFACS Addressed in both but to different degrees Lighter treatment by HERA Implied in HFACS Addressed differently Implied in HFACS

Harmonization JANUS HERA-JANUS

Error Details Perception and Vigilance Response Execution Memory yes Did the controller forget recent information/actions OR forget future action/intentions forget stored information? Did the controller mis-see or mishear information OR Was the information not detected or detected late? no yes Response Execution Memory no Did the controller immediately perform an action on the identified object. OR Did the controller make an error in performing an action, or speech? no yes Did the controller misjudge information OR Make an error in planning, problem solving or decision-making? yes Planning and Decision-making

The Contextual Conditions More than 200 options under: Pilot-ATCO Communication Pilot Actions Traffic & Airspace Weather Documentation & Procedures Training & Experience Workplace design & HMI Environment Personal Factors Team Factors Organisational Factors

Time-table

Validation Principles Reliability and Objectivity Content-Related Validity Empirical Validity Practicality and Usability Face Validity and Acceptance

Beta-testing in Europe From June 2001 to March 2002 Trained 48 investigators from Austria, France, Germany, Netherlands, Sweden, UK/Serco and Portugal JANUS assessments used paper-based decision flow diagrams Total incidents: 60 (mix of new incidents and re-analysis of previous reports)

Validation in Europe Incident investigators reached reasonable levels of agreement Error Type 78% Error Detail 72% Error Mechanism 66% Information Processing 57% Contextual Conditions 76% Overall agreement of the group with each individual Incident Investigator 88%

Validation in Europe Comparison of HERA-JANUS to previous methods 86% - More detailed, objective, structured, and precise results 86% - More useful information and prompted looking in more detail at contextual conditions 66% - More confidence in investigation process 50% - Controllers accepted HERA-JANUS as better than previous method 100% -Investigators accepted HERA-JANUS as better than previous method

Beta-testing in the USA From December 2001 to September 2002 Participation by 29 facilities with data collected from 12 ARTCCs – 6 TRACONS – 6 JANUS assessments made by CAMI researchers in parallel to but independent from current investigation process JANUS interviews used PC-based tool that automated decision flow diagrams Total incidents: 79 (new)

Validation in the USA JANUS interview data collected from 79 primary controllers and 136 other controllers, supervisors, and managers Inter-rater agreement with the primary controller ranged from 73% for other controllers (kappa=0.48), to 65% for facility managers (kappa=.47) on Perception and Vigilance factors Per incident: Primary controllers identified an average 3.6 psychological factors (Planning and Decision Making most frequently reported) All participants identified an average 9.6 contextual factors (Traffic and Airspace most frequently reported)

Validation in the USA 48% of JANUS factors not found in current FAA report form ¨causal factors¨ Results from questionnaires showed positive support Average number of causal factors with current FAA report form = 1.7 Average number of causal factors with JANUS = 13.2

Feedback “The Technique takes an intensive look behind the incident and helps to eliminate the possible causes from the probable facts” “It replaces the feeling of guessing with a structured approach” “More detail, more informative” “Pinpoints the causal factors”

Feedback Conclusions Does it work? How well? Better than current method?

Are there any questions? Thank You Are there any questions?