BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Research & Technology Europe Usability Study of Lufthansa’s Performance Evaluation System Hans-Juergen.

Slides:



Advertisements
Similar presentations
Good Evaluation. Good Evaluation Should … be simple be fair be purposeful be related to the curriculum assess skills & strategies set priorities use multiple.
Advertisements

Sue Sears Sally Spencer Nancy Burstein OSEP Directors’ Conference 2013
The Technology Integration Planning Model
BR&TC H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Research & Technology Europe Crosslinking NOTECHS to Lufthansa’s Interpersonal Competences Hans-Juergen.
Professor or Editor? Time-Saving Strategies for Effective Grading of Writing Assignments DR. DAVID S. HOGSETTE.
INITIAL ON BOARDING COACHING
Exercise Swaps Community Emergency Response Team.
CERT Program Manager: Training and Exercises
Software Quality Assurance Plan
Chapter 2 Analyzing the Business Case.
Workplace-based Assessment. Overview Types of assessment Assessment for learning Assessment of learning Purpose of WBA Benefits of WBA Miller’s Pyramid.
Preparing for Your Performance Review (A Staff Perspective) Preparing for Your Performance Review (A Staff Perspective)
1 Introduction to Workforce Planning and Development in State of Alaska Executive Branch Departments.
Final Report Document. Format Title Page Executive Summary Table of Contents Introduction Mission Statement Main PDS items Brief justification of the.
Unit 8: Tests, Training, and Exercises Unit Introduction and Overview Unit objectives:  Define and explain the terms tests, training, and exercises. 
Formative and Summative Evaluations
1 Southern Connecticut State University Graduate Council Academic Standards Committee Procedures for Southern Connecticut State University.
The reform of A level qualifications in the sciences Dennis Opposs SCORE seminar on grading of practical work in A level sciences, 17 October 2014, London.
APPRAISING AND MANAGING PERFORMANCE
How to write a Report On Assessment Source: AUN Secretariat.
2010 Performance Evaluation Process Information Session for Staff
Lesson planning? It can’t be that difficult! Svetla Tashevska, NBU.
Mentorship. More people are applying for and completing mentorship Mentorship recommended  15 Completing mentorship  2.
Conducting the IT Audit
/1203 © 2003 Business & Legal Reports, Inc. BLR’s Human Resources Training Presentations Training Strategies I: E-Learning.
Internal Auditing and Outsourcing
Solid Performance Solutions This presentation was created to give you the ideas and support to present “Rethinking New Product Training Development” in.
+ Measuring Teaching Quality in the Online Classroom Ann H. Taylor Director, Dutton e-Education Institute College of Earth and Mineral Sciences.
Project co-financed by European Union Project co- financed by Asean European Committee for Standardization Implementing Agency 1 Module 13 GMP Workshop.
National Incident Management System 5-Year Training Plan Al Fluman, Acting Director Incident Management Systems Division (IMSD), National Integration Center.
Unit 5:Elements of A Viable COOP Capability (cont.)  Define and explain the terms tests, training, and exercises (TT&E)  Explain the importance of a.
Topic 4 How organisations promote quality care Codes of Practice
Purpose of Network Evaluation Increase understanding of the relationship between the network design, objectives and functions and the outcomes achieved.
SiTEL LMS Focus Group Executive Summary Prepared: January 25, 2012.
Trends in Corporate Governance Dr. Sandra B. Richtermeyer, CMA, CPA President, Institute of Management Accountants (IMA) June 21, 2011.
Introduction to Development Centres Sandra Schlebusch The Consultants.
1 of 27 How to invest in Information for Development An Introduction Introduction This question is the focus of our examination of the information management.
COMP 208/214/215/216 – Lecture 8 Demonstrations and Portfolios.
D1.HRD.CL9.06 D1.HHR.CL8.07 D2.TRD.CL8.09 Slide 1.
Student Peer Review An introductory tutorial. The peer review process Conduct study Write manuscript Peer review Submit to journal Accept Revise Reject.
Designing a Training Program RATIONALE OF THE TRAINING Background or introduction of what the training is all about –Developments in the field/discipline/area.
Chapter 8 Evaluating and Controlling Performance
 Definition of a quality Audit  Types of audit  Qualifications of quality auditors  The audit process.
The Conceptual Framework: What It Is and How It Works Linda Bradley, James Madison University Monica Minor, NCATE April 2008.
THOMAS COOK CRM STANDARDS ASSESSMENT Captain Karen Varney CRM Manager Thomas Cook Airlines UK.
IAEA International Atomic Energy Agency Methodology and Responsibilities for Periodic Safety Review for Research Reactors William Kennedy Research Reactor.
Evaluating Student Presentations Speaking Intensive Workshop.
Development of the Egyptian Code of Practice for Student Assessment Lamis Ragab, MD, MHPE Hala Salah, MD.
Workshop For Reviewers Operating the Developmental Engagements Prof. Dr. Hala SalahProf. Dr. Hoda ELTalawy.
Dr. Salwa El-Magoli Chairperson of the National Quality Assurance and Accreditation Committee. Former Dean of the Faculty of Agricultural, Cairo university.
Final Presentation, European Cooperative House Brussels, 16 Dec.2009 Training +45 “Teachers Facilitating Learning among Elders” WP5 Test and Evaluation.
Road Safety Education Program (RSEP) A first Summary July 2011 Putting People First.
EXPECTATIONS AND OUTCOMES Prof. Yohana Mashalla, MD, PhD Co-Director, Afya Bora Consortium 1 Dar es Salaam, July 2014.
NOTECHS. Objectives By the end of this workshop you will:  Understand facilitation techniques, and be able to apply them to manage a constructive debrief.
IMS Implementation Project
agenda What is the new Performance Management System
Understanding Standards Event Fashion & Textile Technology
Monitoring and Evaluation Systems for NARS Organisations in Papua New Guinea Day 3. Session 8. Routine monitoring.
CRMSG meeting 18th March 2010.
Evaluating performance management
How to do an Internal Audit
"A system that trains the evaluator"
Tasks & Grades for MET2.
Tasks & Grades for MET3.
Tasks & Grades for MET5.
IAESB Meeting IFAC Headquarters, New York July , 2018
Tasks & Grades for MET4.
Form FRA PC/L Human Factors
THOMAS COOK CRM STANDARDS ASSESSMENT
Presentation transcript:

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Research & Technology Europe Usability Study of Lufthansa’s Performance Evaluation System Hans-Juergen Hoermann, PhD Boeing Research & Technology Europe RAeS, April 30, 2004

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Overview Introduction Design of the usability study Quantitative results concerning the practicability and training benefits of the new Proficiency Reporting Form (PPR) Summary of written user comments Recommendations for a train-the-trainer course Conclusions

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Coinciding Inputs From Different Initiatives LH Flight safety survey Update of performance criteria for LH flight-crew members New concepts for proficiency reporting forms JAA requirements (NOTECHS/JARTEL)

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Special Features of the New PPR Based on the LH Competence System with the three areas: technical, procedural, and interpersonal Frequency scale for quantitative observations Four-point grading scale for quality of performance Competence codes for feedback and further training recommendations Training goals explicitly included

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Usability Study User Input & Feedback Form: –15 yes/no items with plenty of room for free comments, concerning... Structure, layout and contents of the new Evaluation System Practicability and use of different scales Clearness and adequacy of results Value of Comp. Codes for debriefing and training

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Usability Study Design and procedure: –N = 45 TCs from 3 LH fleets –One-day introductory course –Telephone hotline –4 months trial period with three forms –N = 20 returned with full data –Based on training events with 27 captain candidates –Follow-up crosscheck of the revised form within another LH subsidiary (N = 13 TCs)

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Usability Study User feedback in form of: –76 (98) specific comments –13 (18) general comments

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Structure, Layout, and Contents Sufficiently clear? % Criteria complete? 80% Comments: no gaps, suggestions for reducing items, clear structure significantly reduces efforts to fill out PPR clear structure significantly reduces efforts to fill out PPR

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Practicability and Use of Scales Famil- course OK? 95% Freq count practicable? Req. time justifiable? 94% 85%

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Practicability and Use of Scales Comments: The clear structure makes it less time-consuming than before, Difficult to handle the papers especially when also active as CM Quantity is not always equivalent with quality

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Clearness and Adequacy of Results Unambiguous? Definite and adequate? Different to previous? 15% 90% 65% Comments: Scale anchor „Always“ is too strict. Minor errors would lead to a downgrading of otherwise good performance 75%

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Use of Comp. Codes for Training & Debrief Useful for training? Facilitate debrief? 90% 50-60% Comments: Training goals more transparent for trainer and trainee. Weaknesses can be identified earlier. CCs helpful for finding adequate training recommendations.

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Distribution of Comp. Codes in the Study

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr General Comments Observation is determined by more specific criteria instead of the overall impression Debriefing could become a little longer but also more objective and supportive All in all, a little more effort for the benefit of a more adequate evaluation and feedback of the candidate‘s performance More objective because of more observation items (CLH) Because of the versatility of the observation items the form is good for a fair and more standardised evaluation (CLH)

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Conclusions User feedback has provided valuable insights for further revisions of the PPR and contents of the familiarization course Acceptance of deviation from absolute perfection (100%  99+%) High degree of acceptance for the integration of interpersonal skills, training should precede checking Full transparency of training goals offers better opportunity for self-evaluation and preparation of trainees A long-term evaluation study is scheduled to detect trends and tailor future training methods

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Training Course for NOTECHS Examiners In addition to the required training of instructors and examiners it is recommended to have at minimum A two-day training session with a trainee-trainer ratio not exceeding 12:1 Timed in vicinity to the first training mission of the new instructor/examiner A thorough briefing on the respective non-technical skills concept including categories, elements, and desired/undesired behaviour anchors A thorough briefing on the rating scale, the evaluation criteria (cut-offs), and the forms A practical training session with specific video examples for each of the evaluation categories. This training includes feedback and discussion. Guidance material and practical training sessions of how to conduct a constructive debriefing on non-technical skills. Both in examiner and examinee roles. Regular refresher for re-calibration Written material, including definitions and criteria should be available and agreed throughout the company

BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr Lufthansa‘s Basic Competence System Finally, we arrived at the summary…

What is left? BR&TE Principles of Behaviour Assessment in NOTECHS/JARTEL Development of Lufthansa‘s New Behaviour Marker System Crosslinking NOTECHS to LH Interpersonal Competences The Implementation Case: Lufthansa‘s Evaluation Method Usability Study Summary