1. The Effects of F eedback on the Accuracy of Completing Flight Checklists William Rantz Western Michigan University November 14, 2007

2. 2 Overview  Rationale & Purpose  Location & Duration  PC-ATD  Participants  Checklists & Flight Pattern  Dependent Variables  Independent Variables & Integrity of IVs  Experimental Design  Results  Discussion & Future Research

3. 3 Rationale  Improper use of checklist (Degani, 1992, 2002; Diez, Boehm- Davis, & Holt, 2003; Turner, 2001) (Degani, 1992, 2002; Diez, Boehm- Davis, & Holt, 2003; Turner, 2001)  Most common error cited in LOSA data Observational data Observational data 54% of errors 54% of errors (Helmreich et al., 2001)  Contributing factor to numerous accidents Improper configuration of aircraft Improper configuration of aircraft (NTSB, 1969, 1975,1982…)

4. 4 Purpose  To examine whether pilots would complete airplane checklists more accurately when they received post- flight graphic and verbal feedback  No study in aviation has attempted to increase checklist accuracy using experimental manipulation of IVs

5. 5 Location & Duration  The PC-ATD laboratory is located in a building adjacent to WMU’s Aviation Education Center in Battle Creek, MI  Data collection took approximately 66 sessions March 20, 2007 through April 14, 2007 March 20, 2007 through April 14, 2007 200 flight trials 200 flight trials

6. 6 Personal Computer-Aviation Training Device

7. 7 C-172 Instrumentation Panel

8. 8 Observation Room

9. 9 Participants  8 WMU flight students 1 female & 7 males (20 – 26 years old) 1 female & 7 males (20 – 26 years old) Private Pilot Certificate Private Pilot Certificate 125 minimum flight hours 125 minimum flight hours Instrument rated Instrument rated

10. 10 Main Dependent Variable  The number of checklist items completed correctly per flight

11. 11 Secondary Dependent Variable 1  The percentage of total errors for each of the six flight segments during each experimental phase (baseline, feedback, and reversal) per participant

12. 12 Secondary Dependent Variable 2  The percentage of baseline trials participants performed each of the checklist items incorrectly

13. 13 Experimental Phases  Baseline Only technical feedback of flight performance was given Only technical feedback of flight performance was given  Checklist Graphic Feedback & Vocal Praise Technical feedback of flight performance Technical feedback of flight performance Graphic feedback on the total number of checklist items completed correctly per flight Graphic feedback on the total number of checklist items completed correctly per flight Graphic feedback on the number of items completed correctly, completed incorrectly, and omitted for each of the six flight segments per flight Graphic feedback on the number of items completed correctly, completed incorrectly, and omitted for each of the six flight segments per flight Vocal praise for any improvement Vocal praise for any improvement  Reversal Only technical feedback of flight performance was given Only technical feedback of flight performance was given

14. 14 Graphic Feedback: Total Items Correct

15. 15 Graphic Feedback: Flight Segment

16. 16 Integrity of the IV  Technical flight and checklist feedback were read from prepared scripts  Participants were asked to initial the technical flight diagrams and the checklist feedback graphs and returned to the experimenter.  Integrity of IV = 100%

17. 17 Experimental Design  A multiple baseline with reversal design across paired individuals Initial phase changes occurred when performance was judged as stable upon visual inspection Initial phase changes occurred when performance was judged as stable upon visual inspection

18. 18 Results  All participants increased performance accuracy over baseline when post-flight checklist feedback and praise was added  Improvements in performance remained during intervention withdrawal

19. 19 Results-Figure 1  Average percentage of checklist items completed correctly increased from 53% during the baseline phase to 98% during the last three sessions of the intervention phase  The average percentage of checklist items completed correctly was 99% during the return to baseline condition

20. 20 Results

21. 21 Results -Figure 2  During all flights, 1,973 total errors were observed  The average percentage of segment errors was highest for the after take-off segment (88%, range = 71.43% - 100%)  The average percentage of segment errors was lowest for the pre take-off segment (32%, range = 5.88% - 68.91%)  Please see Figure 2 hand out

22. 22 Results-Figure 3  Generally, the percentage of errors by flight segment varied across participants and flight segments  Errors decreased considerably for all participants during intervention.  Errors were very low during reversal  Please see Figure 3 hand out

23. 23 Results-Table 1  Percentages that are 50% or greater are shaded for each participant. Also, the checklist item name is shaded if the percentage of error was 50% or greater for four or more participants  The highest frequency of errors occurred for two items in the after take-off segment: Checking flaps and engine instruments, with 99 errors each.  The pre landing items were the second highest errors  The after landing segment had the third highest errors  Please see Table 1 hand out

24. 24 Recap of Results  Checklist errors were reduced or eliminated during the intervention phase  Performance improvement maintained during reversal

25. 25 Possible Confounding Variables  These variables could account for variability in pilot performance: PC-ATD experience level PC-ATD experience level Recency of flight experience Recency of flight experience Recency of flight in aircraft type Recency of flight in aircraft type Fatigue/stress Fatigue/stress

26. 26 Limitations  Limited timeline of semester  Transferability to other simulator platforms  Transferability to actual flight training  Partial out intervention components (graph vs vocal)

27. 27 Future Research  Replicating the current study during actual training flights when flight conditions such as weather and airport traffic differ  Replicating the current study and ascertaining whether checklist compliance transfers to actual flight  Determining how long gains in checklist accuracy would continue in the absence of post-flight feedback and praise  Investigating the nature of the rule changes and whether accurate checklist use would generalize to actual flight

28. 28