1. "A system that trains the evaluator"
From NOTECHS to a New System of Observing Behavioural Markers as a Basis for Assessing Pilots' Performance
"A system that trains the evaluator"
Cpt. Harry Neb,
Lufthansa German Airlines
FRA PC/L
Human Factors Operational Input

2. Influences and SourcesLH
Interpersonal
Competence
LH Safety Study (1997)
External Information (scientific,
other airlines, RAF, etc.)
NOTECHS JAA
LH CRM-Markers (1992)
FRA PC/L Human Factors

3. Influences and Sources
JAA
FRA PC/L Human Factors

4. Frequency Distribution by Event-Configurations
9.1%
46.2%
30.8%
Now we are going to take a look on the results - where the factors are left in their configurations and compared by the frequency of reported occurrence. As we can see, the number of incidents is dependent on the complexity of aspects, with increasing complexity the frequency increases as well. Although this doesn´t come as a complete surprise, it is the first time - that we are able to prove this connection.
14.5%
FRA PC/L Human Factors

5. Aggravating Social Factors
contributed to 71% of events.
Deficiencies in Involvement
Information management %
Quality of communication %
Social climate %
FRA PC/L Human Factors

6. FRA PC/L Human Factors

7. Interpersonal Competence
Elements
Communication
Generally, communication includes information transfer and social aspects.
Crew members share information, and assure reception and understanding. Suggestions of other crew members are considered, even if one does not agree. Ambiguities and uncertainties are announced.
Workload Management
Leadership and Teamwork
Crew members clearly prioritise operational tasks and distribute them appropriately. Available external and internal resources are used to accom-plish task completion in good time.
Stress and error are inherent factors of flight, and crew members aim to minimise their negative effects.
Led by the commander, the crew achieves a safe and efficient performance in a climate that is rational and free of intimidation.
Social interaction conflicts have to be addressed and managed. Every crew member takes initiative to be an active and constructive part of the team.
Interpersonal Competence
Situation Awareness and Decision Making
Crew members recognise and anticipate factors affecting the flight. After these factors are evaluated, they choose the appropriate course of action.
To achieve a favourable outcome, crew members actively monitor execution and development of the situation.
FRA PC/L Human Factors

8. Descriptors Communication Atmosphere Information Transfer
encourage open and honest communication
achieve a positive first impression
listen actively
consider suggestions
Information Transfer
share information
assure reception
assure understanding
Information Management
clearly state plans and intentions
announce ambiguities
announce uncertainties
speak frankly about problems within the crew
FRA PC/L Human Factors

9. Descriptors Leadership & Teamwork Command- ability Team Ability
take the lead of the crew as commander
establish goals, control outcome, and correct
consider condition of others
Team Ability
act as a constructive member of the team
take initiative
encourage others to co-operate
support others
seek ideas and views from others
present own point of view
provide appropriate feedback
propose alternative ideas if appropriate
Conflict Management
address and manage conflicts
achieve rational climate
avoid intimidation
adopt assertive behaviour if appropriate and persist until attention of others is gained or corrective action taken
accept appropriate criticism
avoid competition between crew members
FRA PC/L Human Factors

10. Descriptors Workload Management Task Time Stress and Error
prioritise operational tasks
distribute tasks appropriately
complete tasks in good time
use external and internal resources
Time
plan ahead
allocate time to tasks appropriately
Stress and Error
aim to minimise negative effects of stress
aim to minimise effects of error
FRA PC/L Human Factors

11. Descriptors Situation Awareness and Decision Making Preparation
act with respect to time available
avoid distractions
anticipate factors affecting the flight
recognise factors affecting the flight
Processing
evaluate factors affecting the flight
choose appropriate course of action
monitor execution
monitor development of the situation
apply FOR-DEC for complex decisions
F-Facts, O-Options, R-Risks & Benefits, D-Decision, E-Execution, C-Check
Interaction
involve others in the process
discuss discrepancies
FRA PC/L Human Factors

12. Evaluation
A New Concept. Why?

13. Evaluation Prerequisites
Appropriateness and Relevance
Standardization of contents and execution
Validity: Accuracy of correspondence with the relevant learning goals
Reliability: Constant accuracy and correspondence with comparable results
Objectivity: Same results with application by different evaluators
Transparency and Reasonableness: No traps and tricks
Fairness: Equal treatment of candidates
from: K. Steininger, Handbuch der Flugpädagogik
FRA PC/L Human Factors

14. Present Status
Description of performance in the words of the individual evaluator
Subjective view due to different priorities
Interpretation of assessment by the training management
CRM criteria (behavioural markers) not integrated
And, to top all of it, the JAR demand for assessment of CRM (primarily during line checking) had to be fulfilled. But nobody within LH so far had developed distinct and detailed criteria for this.
FRA PC/L Human Factors

15. Present Status contd. Quality description in two steps:
proficient
repeat
Trend not recognizable
Flaws in the system not identifiable
Another point in the present system is the problem of recognizing trends. Only yes or no is simply not sufficient to define a trend curve. And what, if deficiencies in the performance of trainees was a result of deficiencies in the training system?
FRA PC/L Human Factors

16. For all assessments Goal (Highlights) Uniformity Objectivity
As I mentioned before, we started with initial line training for FO or upgrading to Commander. But developing a sound concept meant also to consider all training and check events.
Take calculated measures at the earliest possible stage for any performance below standard
FRA PC/L Human Factors

17. Uniformity Demand the same performance under the same circumstances
Ask the same questions
Observe the same defined behaviour
Treat each trainee equally
FRA PC/L Human Factors

18. For all assessments Goal (Highlights) Uniformity Objectivity
As I mentioned before, we started with initial line training for FO or upgrading to Commander. But developing a sound concept meant also to consider all training and check events.
Take calculated measures at the earliest possible stage for any performance below standard
FRA PC/L Human Factors

19. Objectivity
Application by different evaluators leads to the same result
FRA PC/L Human Factors

20. For all assessments Goal (Highlights) Uniformity Objectivity
As I mentioned before, we started with initial line training for FO or upgrading to Commander. But developing a sound concept meant also to consider all training and check events.
Take calculated measures at the earliest possible stage for any performance below standard
FRA PC/L Human Factors

21. Expected Outcome Assess quality to manage quality
Identify trends to assure (maintain or improve) quality with
Trainee
Instructor
System
FRA PC/L Human Factors

22. Identified Necessary Actions
Define detailed observable rating criteria => Description of company standard
By observing frequencies of behaviour the progress and the absolute performance to be described
Lead instructors to observe largely the same
Make transparent for the trainee
FRA PC/L Human Factors

23. Identified Necessary Actions
Differentiate quality of performance
Define behavioural markers to describe the interpersonal competence
FRA PC/L Human Factors

24. Quality Assurance for:
TRAINEE
by providing well aimed support for cases of performance below standard {below satisfactory} (because identifiable)
FRA PC/L Human Factors

25. Quality Assurance for:
TRAINER (Evaluator)
by enabling comparable assessments (for objectivity)
FRA PC/L Human Factors

26. Quality Assurance for:
SYSTEM
to improve the training process (by discovering weak spots)
FRA PC/L Human Factors

27. !
Goal (Trainer):
No substantial change of methodical and didactic means for training (The system does not intervene in the personal style of training)
FRA PC/L Human Factors

28. Definition
Trainer:
"A trainer is an employee trained in training who is instructed by the traineeship provider to train trainees."
Hamburg Senate "Definitions"
FRA PC/L Human Factors

29. Goal (Trainer): Keep the sequence: Observe Note Evaluate
FRA PC/L Human Factors

30. Observation
Prerequisite for evaluation is observation of Quantity and Quality of behaviour
how often
what
FRA PC/L Human Factors

31. What Defines Quality Specified OBSERVATION CRITERIA QUALITY
(i.e. behavioural markers resp. observable behaviour)
QUALITY
FRA PC/L Human Factors

32. Therefore observation criteria have to be:
relevant
extensive
observable
FRA PC/L Human Factors

33. general descriptors of the areas
Structure
9 defined phases of flight (from "preparation" to "leaving aeroplane") +
general descriptors of the areas
technical
procedural
interpersonal
FRA PC/L Human Factors

34. Break
FRA PC/L Human Factors

35. What Next? Principles of Behaviour Assessment in NOTECHS/JARTEL
Development of Lufthansa‘s New Behaviour Marker System
Crosslinking NOTECHS to LH Interpersonal Competences
The Implementation Case: Lufthansa‘s Evaluation Method
Usability Study
Summary