"A system that trains the evaluator" From NOTECHS to a New System of Observing Behavioural Markers as a Basis for Assessing Pilots' Performance "A system that trains the evaluator" Cpt. Harry Neb, Lufthansa German Airlines FRA PC/L Human Factors Operational Input

Influences and Sources LH Interpersonal Competence LH Safety Study (1997) External Information (scientific, other airlines, RAF, etc.) NOTECHS JAA LH CRM-Markers (1992) FRA PC/L Human Factors 18.09.2018

Influences and Sources JAA FRA PC/L Human Factors 18.09.2018

Frequency Distribution by Event-Configurations 9.1% 46.2% 30.8% Now we are going to take a look on the results - where the factors are left in their configurations and compared by the frequency of reported occurrence. As we can see, the number of incidents is dependent on the complexity of aspects, with increasing complexity the frequency increases as well. Although this doesn´t come as a complete surprise, it is the first time - that we are able to prove this connection. 14.5% FRA PC/L Human Factors 18.09.2018

Aggravating Social Factors contributed to 71% of events. Deficiencies in Involvement Information management 67.2% Quality of communication 52.3% Social climate 27.2% FRA PC/L Human Factors 18.09.2018

FRA PC/L Human Factors 18.09.2018

Interpersonal Competence Elements Communication Generally, communication includes information transfer and social aspects. Crew members share information, and assure reception and understanding. Suggestions of other crew members are considered, even if one does not agree. Ambiguities and uncertainties are announced. Workload Management Leadership and Teamwork Crew members clearly prioritise operational tasks and distribute them appropriately. Available external and internal resources are used to accom-plish task completion in good time. Stress and error are inherent factors of flight, and crew members aim to minimise their negative effects. Led by the commander, the crew achieves a safe and efficient performance in a climate that is rational and free of intimidation. Social interaction conflicts have to be addressed and managed. Every crew member takes initiative to be an active and constructive part of the team. Interpersonal Competence Situation Awareness and Decision Making Crew members recognise and anticipate factors affecting the flight. After these factors are evaluated, they choose the appropriate course of action. To achieve a favourable outcome, crew members actively monitor execution and development of the situation. FRA PC/L Human Factors 18.09.2018

Descriptors Communication Atmosphere Information Transfer encourage open and honest communication achieve a positive first impression listen actively consider suggestions Information Transfer share information assure reception assure understanding Information Management clearly state plans and intentions announce ambiguities announce uncertainties speak frankly about problems within the crew FRA PC/L Human Factors 18.09.2018

Descriptors Leadership & Teamwork Command- ability Team Ability take the lead of the crew as commander establish goals, control outcome, and correct consider condition of others Team Ability act as a constructive member of the team take initiative encourage others to co-operate support others seek ideas and views from others present own point of view provide appropriate feedback propose alternative ideas if appropriate Conflict Management address and manage conflicts achieve rational climate avoid intimidation adopt assertive behaviour if appropriate and persist until attention of others is gained or corrective action taken accept appropriate criticism avoid competition between crew members FRA PC/L Human Factors 18.09.2018

Descriptors Workload Management Task Time Stress and Error prioritise operational tasks distribute tasks appropriately complete tasks in good time use external and internal resources Time plan ahead allocate time to tasks appropriately Stress and Error aim to minimise negative effects of stress aim to minimise effects of error FRA PC/L Human Factors 18.09.2018

Descriptors Situation Awareness and Decision Making Preparation act with respect to time available avoid distractions anticipate factors affecting the flight recognise factors affecting the flight Processing evaluate factors affecting the flight choose appropriate course of action monitor execution monitor development of the situation apply FOR-DEC for complex decisions F-Facts, O-Options, R-Risks & Benefits, D-Decision, E-Execution, C-Check Interaction involve others in the process discuss discrepancies FRA PC/L Human Factors 18.09.2018

Evaluation A New Concept. Why?

Evaluation Prerequisites Appropriateness and Relevance Standardization of contents and execution Validity: Accuracy of correspondence with the relevant learning goals Reliability: Constant accuracy and correspondence with comparable results Objectivity: Same results with application by different evaluators Transparency and Reasonableness: No traps and tricks Fairness: Equal treatment of candidates from: K. Steininger, Handbuch der Flugpädagogik FRA PC/L Human Factors 18.09.2018

Present Status Description of performance in the words of the individual evaluator Subjective view due to different priorities Interpretation of assessment by the training management CRM criteria (behavioural markers) not integrated And, to top all of it, the JAR demand for assessment of CRM (primarily during line checking) had to be fulfilled. But nobody within LH so far had developed distinct and detailed criteria for this. FRA PC/L Human Factors 18.09.2018

Present Status contd. Quality description in two steps: proficient repeat Trend not recognizable Flaws in the system not identifiable Another point in the present system is the problem of recognizing trends. Only yes or no is simply not sufficient to define a trend curve. And what, if deficiencies in the performance of trainees was a result of deficiencies in the training system? FRA PC/L Human Factors 18.09.2018

For all assessments Goal (Highlights) Uniformity Objectivity As I mentioned before, we started with initial line training for FO or upgrading to Commander. But developing a sound concept meant also to consider all training and check events. Take calculated measures at the earliest possible stage for any performance below standard FRA PC/L Human Factors 18.09.2018

Uniformity Demand the same performance under the same circumstances Ask the same questions Observe the same defined behaviour Treat each trainee equally FRA PC/L Human Factors 18.09.2018

For all assessments Goal (Highlights) Uniformity Objectivity As I mentioned before, we started with initial line training for FO or upgrading to Commander. But developing a sound concept meant also to consider all training and check events. Take calculated measures at the earliest possible stage for any performance below standard FRA PC/L Human Factors 18.09.2018

Objectivity Application by different evaluators leads to the same result FRA PC/L Human Factors 18.09.2018

For all assessments Goal (Highlights) Uniformity Objectivity As I mentioned before, we started with initial line training for FO or upgrading to Commander. But developing a sound concept meant also to consider all training and check events. Take calculated measures at the earliest possible stage for any performance below standard FRA PC/L Human Factors 18.09.2018

Expected Outcome Assess quality to manage quality Identify trends to assure (maintain or improve) quality with Trainee Instructor System FRA PC/L Human Factors 18.09.2018

Identified Necessary Actions Define detailed observable rating criteria => Description of company standard By observing frequencies of behaviour the progress and the absolute performance to be described Lead instructors to observe largely the same Make transparent for the trainee FRA PC/L Human Factors 18.09.2018

Identified Necessary Actions Differentiate quality of performance Define behavioural markers to describe the interpersonal competence FRA PC/L Human Factors 18.09.2018

Quality Assurance for: TRAINEE by providing well aimed support for cases of performance below standard {below satisfactory} (because identifiable) FRA PC/L Human Factors 18.09.2018

Quality Assurance for: TRAINER (Evaluator) by enabling comparable assessments (for objectivity) FRA PC/L Human Factors 18.09.2018

Quality Assurance for: SYSTEM to improve the training process (by discovering weak spots) FRA PC/L Human Factors 18.09.2018

! Goal (Trainer): No substantial change of methodical and didactic means for training (The system does not intervene in the personal style of training) FRA PC/L Human Factors 18.09.2018

Definition Trainer: "A trainer is an employee trained in training who is instructed by the traineeship provider to train trainees." Hamburg Senate "Definitions" FRA PC/L Human Factors 18.09.2018

Goal (Trainer): Keep the sequence: Observe Note Evaluate FRA PC/L Human Factors 18.09.2018

Observation Prerequisite for evaluation is observation of Quantity and Quality of behaviour how often what FRA PC/L Human Factors 18.09.2018

What Defines Quality Specified OBSERVATION CRITERIA QUALITY (i.e. behavioural markers resp. observable behaviour) QUALITY FRA PC/L Human Factors 18.09.2018

Therefore observation criteria have to be: relevant extensive observable FRA PC/L Human Factors 18.09.2018

general descriptors of the areas Structure 9 defined phases of flight (from "preparation" to "leaving aeroplane") + general descriptors of the areas technical procedural interpersonal FRA PC/L Human Factors 18.09.2018

Break FRA PC/L Human Factors 18.09.2018

What Next? Principles of Behaviour Assessment in NOTECHS/JARTEL Development of Lufthansa‘s New Behaviour Marker System Crosslinking NOTECHS to LH Interpersonal Competences The Implementation Case: Lufthansa‘s Evaluation Method Usability Study Summary

Form FRA PC/L Human Factors 18.09.2018

Frequency Scale FRA PC/L Human Factors 18.09.2018

Frequencies FRA PC/L Human Factors 18.09.2018

Grading Scale FRA PC/L Human Factors 18.09.2018

Grading FRA PC/L Human Factors 18.09.2018

Repeat │ Debriefed │ Standard │ Excellent ↕ ↕ ↕ ↕ ↕ ↕ ↕ ↕ Unsatisfactory│ Marginal │ Satisfactory │ Excellent FRA PC/L Human Factors 18.09.2018

Basic Performance of Flight Crew Technical Manual Aeroplane Control (TM) Knowledge of Systems (TS) Use of Automation (TA) Procedural Knowledge of Procedures (PK) Adherence to Procedures (PA) Interpersonal Communication (IC) Leadership and Teamwork (IL) Workload Management (IW) Situation Awareness and Decision Making (ID) FRA PC/L Human Factors 18.09.2018

Reasoning FRA PC/L Human Factors 18.09.2018

Reasoning FRA PC/L Human Factors 18.09.2018

Identified Necessary Actions Define detailed observable rating criteria => Description of company standard By observing frequencies of behaviour the progress and the absolute performance to be described Instructors observe largely the same Make transparent for the trainee FRA PC/L Human Factors 18.09.2018

Performance and Progress Evaluation absolute actually observed (and recorded) frequencies compared to the „ready" pilot (Commander resp. First Officer) relative actually observed (and recorded) frequencies in relation to the training phase with pre-defined levels of performance {frequencies} Performance and Progress FRA PC/L Human Factors 18.09.2018

Definition The Landing Pilot is initially the Handling Pilot and will handle the take-off and landing, except in role reversal when he is the Non-Handling Pilot for taxi until the Handling Non-Landing Pilot hands the handling to the Landing Pilot at 80 knots. The Non-Landing (Non-Handling, since the Landing Pilot is handling) Pilot reads the checklist to the Handling Landing Pilot until after the Before Descent Checklist completion, when the Handling Landing Pilot hands the handling to the NonHandling Non-Landing Pilot who then becomes the Handling Non-Landing Pilot, eh langind nonsense nonhanging Polit non-checklist........... (from Birdspeed Update to Flight Operations Manual, courtesy of ABS Magazine in Wichita) from Flight International, 12.Mar.96 FRA PC/L Human Factors 18.09.2018

Observation FRA PC/L Human Factors 18.09.2018

Observation contd. FRA PC/L Human Factors 18.09.2018

X X X X X X X Reasoning (Unsatisfactory LS) X X X X X X Weitere Folie für Standard-Beurteilung (S bzw. E) erstellen (andere Phase) 7.2.5. Twice only in 800 Ft, once not at all established 7.3.1. Went below LDG-MIN 7.4.1 One forced landing iso g/a X X X X FRA PC/L Human Factors 18.09.2018

at least 17 out of 33 criteria not met Excerpt of possible observations X Communication X X X X X at least 17 out of 33 criteria not met Leadership & Teamwork X X X X X X Workload Management X X X X X X SA & Decision Making X X FRA PC/L Human Factors 18.09.2018

Observation contd. FRA PC/L Human Factors 18.09.2018

What Next? Principles of Behaviour Assessment in NOTECHS/JARTEL Development of Lufthansa‘s New Behaviour Marker System Crosslinking NOTECHS to LH Interpersonal Competences The Implementation Case: Lufthansa‘s Evaluation Method Usability Study Summary

JAR OPS Requirement Line Check Fulfilment CRM Assessment Methodology Observation of behaviour Criteria for observation (BM) FRA PC/L Human Factors 18.09.2018

Evaluation Summary Observe Note (record) Evaluate structured and standardized criteria which define required or desired quality Note (record) quantitative observations Evaluate qualitative comments based on achieved observation frequencies FRA PC/L Human Factors 18.09.2018

Vision: System to be used for Quality Chain Pilot School Line & Simulator Recurrent Initial Training Advanced Training Jet Orientation Simulator Transition Line Training Proficiency Checks Line Checks Refresher FRA PC/L Human Factors 18.09.2018