"A system that trains the evaluator" From NOTECHS to a New System of Observing Behavioural Markers as a Basis for Assessing Pilots' Performance "A system that trains the evaluator" Cpt. Harry Neb, Lufthansa German Airlines FRA PC/L Human Factors Operational Input

Influences and Sources LH Interpersonal Competence LH Safety Study (1997) External Information (scientific, other airlines, RAF, etc.) NOTECHS JAA LH CRM-Markers (1992) FRA PC/L Human Factors 24.11.2018

Influences and Sources JAA FRA PC/L Human Factors 24.11.2018

Frequency Distribution by Event-Configurations 9.1% 46.2% 30.8% Now we are going to take a look on the results - where the factors are left in their configurations and compared by the frequency of reported occurrence. As we can see, the number of incidents is dependent on the complexity of aspects, with increasing complexity the frequency increases as well. Although this doesn´t come as a complete surprise, it is the first time - that we are able to prove this connection. 14.5% FRA PC/L Human Factors 24.11.2018

Aggravating Social Factors contributed to 71% of events. Deficiencies in Involvement Information management 67.2% Quality of communication 52.3% Social climate 27.2% FRA PC/L Human Factors 24.11.2018

FRA PC/L Human Factors 24.11.2018

Interpersonal Competence Elements Communication Generally, communication includes information transfer and social aspects. Crew members share information, and assure reception and understanding. Suggestions of other crew members are considered, even if one does not agree. Ambiguities and uncertainties are announced. Workload Management Leadership and Teamwork Crew members clearly prioritise operational tasks and distribute them appropriately. Available external and internal resources are used to accom-plish task completion in good time. Stress and error are inherent factors of flight, and crew members aim to minimise their negative effects. Led by the commander, the crew achieves a safe and efficient performance in a climate that is rational and free of intimidation. Social interaction conflicts have to be addressed and managed. Every crew member takes initiative to be an active and constructive part of the team. Interpersonal Competence Situation Awareness and Decision Making Crew members recognise and anticipate factors affecting the flight. After these factors are evaluated, they choose the appropriate course of action. To achieve a favourable outcome, crew members actively monitor execution and development of the situation. FRA PC/L Human Factors 24.11.2018

Descriptors Communication Atmosphere Information Transfer encourage open and honest communication achieve a positive first impression listen actively consider suggestions Information Transfer share information assure reception assure understanding Information Management clearly state plans and intentions announce ambiguities announce uncertainties speak frankly about problems within the crew FRA PC/L Human Factors 24.11.2018

Descriptors Leadership & Teamwork Command- ability Team Ability take the lead of the crew as commander establish goals, control outcome, and correct consider condition of others Team Ability act as a constructive member of the team take initiative encourage others to co-operate support others seek ideas and views from others present own point of view provide appropriate feedback propose alternative ideas if appropriate Conflict Management address and manage conflicts achieve rational climate avoid intimidation adopt assertive behaviour if appropriate and persist until attention of others is gained or corrective action taken accept appropriate criticism avoid competition between crew members FRA PC/L Human Factors 24.11.2018

Descriptors Workload Management Task Time Stress and Error prioritise operational tasks distribute tasks appropriately complete tasks in good time use external and internal resources Time plan ahead allocate time to tasks appropriately Stress and Error aim to minimise negative effects of stress aim to minimise effects of error FRA PC/L Human Factors 24.11.2018

Descriptors Situation Awareness and Decision Making Preparation act with respect to time available avoid distractions anticipate factors affecting the flight recognise factors affecting the flight Processing evaluate factors affecting the flight choose appropriate course of action monitor execution monitor development of the situation apply FOR-DEC for complex decisions F-Facts, O-Options, R-Risks & Benefits, D-Decision, E-Execution, C-Check Interaction involve others in the process discuss discrepancies FRA PC/L Human Factors 24.11.2018

Evaluation A New Concept. Why?

Evaluation Prerequisites Appropriateness and Relevance Standardization of contents and execution Validity: Accuracy of correspondence with the relevant learning goals Reliability: Constant accuracy and correspondence with comparable results Objectivity: Same results with application by different evaluators Transparency and Reasonableness: No traps and tricks Fairness: Equal treatment of candidates from: K. Steininger, Handbuch der Flugpädagogik FRA PC/L Human Factors 24.11.2018

Present Status Description of performance in the words of the individual evaluator Subjective view due to different priorities Interpretation of assessment by the training management CRM criteria (behavioural markers) not integrated And, to top all of it, the JAR demand for assessment of CRM (primarily during line checking) had to be fulfilled. But nobody within LH so far had developed distinct and detailed criteria for this. FRA PC/L Human Factors 24.11.2018

Present Status contd. Quality description in two steps: proficient repeat Trend not recognizable Flaws in the system not identifiable Another point in the present system is the problem of recognizing trends. Only yes or no is simply not sufficient to define a trend curve. And what, if deficiencies in the performance of trainees was a result of deficiencies in the training system? FRA PC/L Human Factors 24.11.2018

For all assessments Goal (Highlights) Uniformity Objectivity As I mentioned before, we started with initial line training for FO or upgrading to Commander. But developing a sound concept meant also to consider all training and check events. Take calculated measures at the earliest possible stage for any performance below standard FRA PC/L Human Factors 24.11.2018

Uniformity Demand the same performance under the same circumstances Ask the same questions Observe the same defined behaviour Treat each trainee equally FRA PC/L Human Factors 24.11.2018

For all assessments Goal (Highlights) Uniformity Objectivity As I mentioned before, we started with initial line training for FO or upgrading to Commander. But developing a sound concept meant also to consider all training and check events. Take calculated measures at the earliest possible stage for any performance below standard FRA PC/L Human Factors 24.11.2018

Objectivity Application by different evaluators leads to the same result FRA PC/L Human Factors 24.11.2018

For all assessments Goal (Highlights) Uniformity Objectivity As I mentioned before, we started with initial line training for FO or upgrading to Commander. But developing a sound concept meant also to consider all training and check events. Take calculated measures at the earliest possible stage for any performance below standard FRA PC/L Human Factors 24.11.2018

Expected Outcome Assess quality to manage quality Identify trends to assure (maintain or improve) quality with Trainee Instructor System FRA PC/L Human Factors 24.11.2018

Identified Necessary Actions Define detailed observable rating criteria => Description of company standard By observing frequencies of behaviour the progress and the absolute performance to be described Lead instructors to observe largely the same Make transparent for the trainee FRA PC/L Human Factors 24.11.2018

Identified Necessary Actions Differentiate quality of performance Define behavioural markers to describe the interpersonal competence FRA PC/L Human Factors 24.11.2018

Quality Assurance for: TRAINEE by providing well aimed support for cases of performance below standard {below satisfactory} (because identifiable) FRA PC/L Human Factors 24.11.2018

Quality Assurance for: TRAINER (Evaluator) by enabling comparable assessments (for objectivity) FRA PC/L Human Factors 24.11.2018

Quality Assurance for: SYSTEM to improve the training process (by discovering weak spots) FRA PC/L Human Factors 24.11.2018

! Goal (Trainer): No substantial change of methodical and didactic means for training (The system does not intervene in the personal style of training) FRA PC/L Human Factors 24.11.2018

Definition Trainer: "A trainer is an employee trained in training who is instructed by the traineeship provider to train trainees." Hamburg Senate "Definitions" FRA PC/L Human Factors 24.11.2018

Goal (Trainer): Keep the sequence: Observe Note Evaluate FRA PC/L Human Factors 24.11.2018

Observation Prerequisite for evaluation is observation of Quantity and Quality of behaviour how often what FRA PC/L Human Factors 24.11.2018

What Defines Quality Specified OBSERVATION CRITERIA QUALITY (i.e. behavioural markers resp. observable behaviour) QUALITY FRA PC/L Human Factors 24.11.2018

Therefore observation criteria have to be: relevant extensive observable FRA PC/L Human Factors 24.11.2018

general descriptors of the areas Structure 9 defined phases of flight (from "preparation" to "leaving aeroplane") + general descriptors of the areas technical procedural interpersonal FRA PC/L Human Factors 24.11.2018

Break FRA PC/L Human Factors 24.11.2018

What Next? Principles of Behaviour Assessment in NOTECHS/JARTEL Development of Lufthansa‘s New Behaviour Marker System Crosslinking NOTECHS to LH Interpersonal Competences The Implementation Case: Lufthansa‘s Evaluation Method Usability Study Summary