The Effects of F eedback on the Accuracy of Completing Flight Checklists William Rantz Western Michigan University November 14, 2007.

Slides:



Advertisements
Similar presentations
Experimental Design Some Terms.
Advertisements

Normal Operations Safety Survey (NOSS)
Robert L. Helmreich, Ph.D. FRAes The University of Texas
Methodology and Explanation XX50125 Lecture 2: Experiments Dr. Danaë Stanton Fraser.
Experimental Designs Dr. Farzin Madjidi Pepperdine University
Defining Characteristics
Effects of Self-Directed Summary of Performance on Students’ Participation in Person Centered Planning Meetings For additional information, please contact:
Digital Training to Analog Flying: Is There a Problem? Western Michigan University, College of Aviation Geoff Whitehurst, Tom Grossman, William Rantz Great.
Chapter 5: Improving and Assessing the Quality of Behavioral Measurement Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition.
Teaching Children With Autism To Follow Activity Schedules on an iPad 3 Using Manual Prompts and Edible Reinforcement Mark Mautone 1, Kenneth F. Reeve.
LECT 5 1 TREATMENT INTEGRITY (Peterson et al., 1982) Treatment integrity: Was the ___ implemented as the experimenter _____________ it to be? Double standard.
How do you know it worked
Chapter 6 – Making and Interpreting Graphs in Single-Subject Research
BHS Methods in Behavioral Sciences I
Chapter 12: Single-Subject Designs An alternative to experimental designs Purpose: To draw conclusions about the effects of treatment based on the responses.
Assessing and Programming Generalized Behavioral Reduction Across Multiple Stimulus Parameters: A Review Megan Duffy Caldwell College.
Single-Case Designs. AKA single-subject, within subject, intra-subject design Footnote on p. 163 Not because only one participant (although might sometimes)
Single-Subject Research
Chapter 7 Correlational Research Gay, Mills, and Airasian
Chapter 10: Planning and Evaluating Applied Behavior Analysis Research
Copyright © 2011 Pearson Education, Inc. All rights reserved. Doing Research in Behavior Modification Chapter 22.
Doing Research in Behavior Modification
Chapter 11 Research Methods in Behavior Modification.
Introduction Results Increasing Eye Contact in Children with Autism Alia F. Groth, Tina M. Franzke, & Kevin P. Klatt University of Wisconsin-Eau Claire.
Introduction to Experimental Design
Orientation to 2008 Regional Training: Building and Sustaining Quality in NRS Data National Webinar June 24, 2008 Larry Condelli Steve Coleman.
Covers SOL CH.1.  In Order to conduct a meaningful scientific investigation, experimental design, hypothesis formation, data gathering, data analysis.
READY OR NOT THE FLIGHT REVIEW. FLIGHT REVIEW A FLIGHT REVIEW IS REQUIRED WITHIN THE PREVIOUS 24 CALENDAR MONTHS TO ACT AS PIC.
Toward quantifying the effect of prior training on task performance MURI Annual Review September 26-27, 2006 Bill Raymond.
CHAPTER 1 LESSON 3 & 4 MATH IN SCIENCE + GRAPHS. WHAT ARE SOME MATH SKILLS USED IN SCIENCE? SOME MATH SKILLS USED IN SCIENCE WHEN WORKING WITH DATA INCLUDE.
Training Interventionists to Implement a Brief Experimental Analysis of Reading Protocol to Elementary Students: An Evaluation of Three Training Packages.
The Limited Name: Robin Annis Class: BSAP Semester: Spring 2007 TA Name: Allison Mueller.
Chapter 11 Overview of Changing Criterion Design.
For ABA Importance of Individual Subjects Enables applied behavior analysts to discover and refine effective interventions for socially significant behaviors.
Determinants of Olfactory Memory Span in Rats MacQueen, D. A., Bullard, L. A., & Galizio, M. University of North Carolina Wilmington Introduction Dudchenko,
Training Individuals to Implement a Brief Experimental Analysis of Oral Reading Fluency Amber Zank, M.S.E & Michael Axelrod, Ph.D. Human Development Center.
Introduction The authors of this research would like to thank the University of Wisconsin-Eau Claire’s Office of Research and Sponsored Programs for financial.
Generalization of the Behavior Sit in Canines to Novel Trainers Nicole C. Scharrer, Jeffrey R. Miller, & Daniel D. Holt Psychology Department, University.
Reversal Designs. Overview One of the most important designs you can use Can be used in a variety of settings Can be very powerful in detecting changes.
Skip Hudspeth and Gordon Hayhoe 112/20/2015. Pavement Roughness Subjective Pilot Rating Study Phase I - Develop a surface roughness model on the B
Treatment Integrity.
READY OR NOT THE FLIGHT REVIEW.
Single-Subject and Correlational Research Bring Schraw et al.
Single- Subject Research Designs
Investigating the Use of Video Modeling to Teach the Expressive Use of Personal Pronouns to Children with Autism Katie Lichtblau and Kevin P. Klatt Psychology.
Finding Answers. Steps of Sci Method 1.Purpose 2.Hypothesis 3.Experiment 4.Results 5.Conclusion.
Lecture №4 METHODS OF RESEARCH. Method (Greek. methodos) - way of knowledge, the study of natural phenomena and social life. It is also a set of methods.
Do Multiple Trainers Increase the Speed of Canine Ability to Generalize a Learned Behavior? Kelsey M. Johnson, Jessica L. Pernsteiner, & Daniel D. Holt.
Topic of the Month November
EKT 314/4 WEEK 2 : CHAPTER 1 INTRODUCTION TO EI ELECTRONIC INSTRUMENTATION.
CHELSEA VANDERWOUDE EXAMINING THE EFFECTS OF A MIRROR ON IMITATION SKILLS IN CHILDREN WITH AUTISM.
AH Biology: Unit 3 Investigative Biology Topic 2- Experimentation: Pilot Studies.
FAA Flight Standards AFS-220/430 FPAW 2017 Summer
Teresa Mulhern1, Siri Ming1, Laura Moran2, & Dr. Ian Stewart1
Single Subject Research
Designing & Implementing Instructional Plans
Experimental Research Designs
Math in Science + Graphs
Nall Report on Aviation Safety
FAA and EASA Sport Aviation Accident Data Richard T Garrison FAI CIMP USA Paris France 2013.
Just What Is Science Anyway???
Evaluating Intervention Effects
ABAB Design Ethical considerations
Designing an Experiment
Correlated-Groups and Single-Subject Designs
Designing an Experiment
Neural Mechanisms of Speed-Accuracy Tradeoff
Factorial Designs Factorial design: a research design that includes two or more factors (Independent Variables) A two-factor design has two IVs. Example:
Designing Experimental Investigations
LESSON 4: THE DATA CYCLE Unit 1
Presentation transcript:

The Effects of F eedback on the Accuracy of Completing Flight Checklists William Rantz Western Michigan University November 14, 2007

2 Overview  Rationale & Purpose  Location & Duration  PC-ATD  Participants  Checklists & Flight Pattern  Dependent Variables  Independent Variables & Integrity of IVs  Experimental Design  Results  Discussion & Future Research

3 Rationale  Improper use of checklist (Degani, 1992, 2002; Diez, Boehm- Davis, & Holt, 2003; Turner, 2001) (Degani, 1992, 2002; Diez, Boehm- Davis, & Holt, 2003; Turner, 2001)  Most common error cited in LOSA data Observational data Observational data 54% of errors 54% of errors (Helmreich et al., 2001)  Contributing factor to numerous accidents Improper configuration of aircraft Improper configuration of aircraft (NTSB, 1969, 1975,1982…)

4 Purpose  To examine whether pilots would complete airplane checklists more accurately when they received post- flight graphic and verbal feedback  No study in aviation has attempted to increase checklist accuracy using experimental manipulation of IVs

5 Location & Duration  The PC-ATD laboratory is located in a building adjacent to WMU’s Aviation Education Center in Battle Creek, MI  Data collection took approximately 66 sessions March 20, 2007 through April 14, 2007 March 20, 2007 through April 14, flight trials 200 flight trials

6 Personal Computer-Aviation Training Device

7 C-172 Instrumentation Panel

8 Observation Room

9 Participants  8 WMU flight students 1 female & 7 males (20 – 26 years old) 1 female & 7 males (20 – 26 years old) Private Pilot Certificate Private Pilot Certificate 125 minimum flight hours 125 minimum flight hours Instrument rated Instrument rated

10 Main Dependent Variable  The number of checklist items completed correctly per flight

11 Secondary Dependent Variable 1  The percentage of total errors for each of the six flight segments during each experimental phase (baseline, feedback, and reversal) per participant

12 Secondary Dependent Variable 2  The percentage of baseline trials participants performed each of the checklist items incorrectly

13 Experimental Phases  Baseline Only technical feedback of flight performance was given Only technical feedback of flight performance was given  Checklist Graphic Feedback & Vocal Praise Technical feedback of flight performance Technical feedback of flight performance Graphic feedback on the total number of checklist items completed correctly per flight Graphic feedback on the total number of checklist items completed correctly per flight Graphic feedback on the number of items completed correctly, completed incorrectly, and omitted for each of the six flight segments per flight Graphic feedback on the number of items completed correctly, completed incorrectly, and omitted for each of the six flight segments per flight Vocal praise for any improvement Vocal praise for any improvement  Reversal Only technical feedback of flight performance was given Only technical feedback of flight performance was given

14 Graphic Feedback: Total Items Correct

15 Graphic Feedback: Flight Segment

16 Integrity of the IV  Technical flight and checklist feedback were read from prepared scripts  Participants were asked to initial the technical flight diagrams and the checklist feedback graphs and returned to the experimenter.  Integrity of IV = 100%

17 Experimental Design  A multiple baseline with reversal design across paired individuals Initial phase changes occurred when performance was judged as stable upon visual inspection Initial phase changes occurred when performance was judged as stable upon visual inspection

18 Results  All participants increased performance accuracy over baseline when post-flight checklist feedback and praise was added  Improvements in performance remained during intervention withdrawal

19 Results-Figure 1  Average percentage of checklist items completed correctly increased from 53% during the baseline phase to 98% during the last three sessions of the intervention phase  The average percentage of checklist items completed correctly was 99% during the return to baseline condition

20 Results

21 Results -Figure 2  During all flights, 1,973 total errors were observed  The average percentage of segment errors was highest for the after take-off segment (88%, range = 71.43% - 100%)  The average percentage of segment errors was lowest for the pre take-off segment (32%, range = 5.88% %)  Please see Figure 2 hand out

22 Results-Figure 3  Generally, the percentage of errors by flight segment varied across participants and flight segments  Errors decreased considerably for all participants during intervention.  Errors were very low during reversal  Please see Figure 3 hand out

23 Results-Table 1  Percentages that are 50% or greater are shaded for each participant. Also, the checklist item name is shaded if the percentage of error was 50% or greater for four or more participants  The highest frequency of errors occurred for two items in the after take-off segment: Checking flaps and engine instruments, with 99 errors each.  The pre landing items were the second highest errors  The after landing segment had the third highest errors  Please see Table 1 hand out

24 Recap of Results  Checklist errors were reduced or eliminated during the intervention phase  Performance improvement maintained during reversal

25 Possible Confounding Variables  These variables could account for variability in pilot performance: PC-ATD experience level PC-ATD experience level Recency of flight experience Recency of flight experience Recency of flight in aircraft type Recency of flight in aircraft type Fatigue/stress Fatigue/stress

26 Limitations  Limited timeline of semester  Transferability to other simulator platforms  Transferability to actual flight training  Partial out intervention components (graph vs vocal)

27 Future Research  Replicating the current study during actual training flights when flight conditions such as weather and airport traffic differ  Replicating the current study and ascertaining whether checklist compliance transfers to actual flight  Determining how long gains in checklist accuracy would continue in the absence of post-flight feedback and praise  Investigating the nature of the rule changes and whether accurate checklist use would generalize to actual flight

28