Beth Blickensderfer, Ph.D. Robert Thomas, M.S., ATP, CFII Assessing Aviation Weather Knowledge in General Aviation Pilots: Overview and Initial Results General Aviation Pilots’ Aviation Weather Knowledge:  Research Results and Implications for Training and Assessment   A Proposed Aviation Weather Knowledge Taxonomy for GA Pilots John Lanicci, Ph.D. Meteorologist Assessing Aviation Weather Knowledge in General Aviation Pilots:  Overview and Initial Results Beth Blickensderfer, Ph.D. Human Factors Psychologist GA Pilot Aviation Weather Resources:  Challenges and Suggestions for Improvement Tom Guinn, Ph.D.   Meteorologist  Bob Thomas, M.S.A., CFII Aviation Professor  Weather Salience in GA Pilots:  An initial study  Jennifer Thropp, Ph.D.  Human Factors Psychologist  Using a Scenario-Based Approach to Assessing Aviation Weather Knowledge in GA Pilots Jessica Cruit, M.S., Ph.D Recent Human Factors Ph.D. graduate! Beth Blickensderfer, Ph.D. John Lanicci, Ph.D. Thomas Guinn Ph.D. Robert Thomas, M.S., ATP, CFII Jayde King, M.S. Yolanda Ortiz, M.S. Presentation given at the Friends and Partners of Aviation Weather (FPAW) Meeting; Nov 2-3, 2016; Orlando, FL

Background Research indicates: FAA Knowledge exams for private and commercial pilots are out of date and too easy. GA Pilots may lack adequate aviation weather knowledge. Are knowledge gaps a contributing factor to accident rate? Aviation weather knowledge assessment tools: Practical use (e.g. FAA Exams  prompt better instruction) Research use (e.g. to identify aviation weather training needs; validate aviation weather training strategies)

Purpose Develop and validate Aviation Weather knowledge questions for use with subsequent General Aviation Weather research.

Knowledge questions 95 Aviation Weather questions (“items”) Team: 2 meteorologists, 1 flight instructor, 1 I-O/HF Psychologist, 2 HF graduate students Item content: driven by task analysis, FAA documents, ACS codes, AFS 630 content guidelines Item format: driven by AFS-630 item writing guide Item level of learning: driven by research guidelines and AFS-630 item difficulty level guidelines (Rote, understanding, application, correlation). Content validation: FAA personnel Rote (Verbal information; memory) Understanding (Verbal information) Application (Cognitive skill) Correlation (Cognitive skill) Is the question content relevant to the private/commercial pilot certificate? Pertinent to safety and relevant to safe operations? Is it more effective to introduce a new question or revise existing? Is the subject matter relevant to the airman everyday or only occasionally?

Method As shown in AC

Participants N = 204 (June – September 2016) ERAU Affiliated = 133; Non-ERAU = 71 Part 61 = 60; Part 141/142 = 143 Flight hours Mean = 201.4 Median = 131 Pilot Certificate and/or Rating Student pilots = 41 Private pilots = 72 Instrument = 50 Commercial = 41 Years Flying; Mean = 3.6 Flight training Part 61 – 52 141 Collegiate – 118 141 Non-collegiate – 9 Mixture – 3 Region in which majority of flight hours were flown NW = 2 SW = 10 NE = 12 SC = 4 EC = 36 NE = 18 SE = 99

Procedure Informed consent Completed Demographic information and Attitudinal measures Self-efficacy (Confidence) Weather salience Completed knowledge questions Computer-based (at ERAU); Randomized Paper-based (OshKosh) Paid $20 + $0.31 per correctly answered question (ERAU students) Debriefed by Experimenter (Graduate Research Assistant)

RESULTS

Overall Aviation Weather Knowledge Score (% Correct) 95 Questions (Cronbach’s alpha = .92)   n M (SD) Private-in- Training 41 47.65 (13.61) Private 72 56.62 (15.67) Private with Instrument 50 61.77 (12.93) Commercial with Instrument 65.62 (14.50) * One-way ANOVA Significant between groups effect A Tukey post hoc test revealed that the overall percent correct of Student Pilots (M = 47.65, SD = 13.61) was significantly less than that of Private pilots (M = 56.62, SD = 15.67, p=.009), Private pilots with Instrument rating (M = 61.77, SD = 12.93, p <. 0005), and Commercial pilots with Instrument rating (M = 65.62, SD = 14.50, p < .0005). The post hoc test also revealed that Commercial pilots with Instrument rating had significantly higher composite test scores compared to Private pilots (p = .009). No other between groups differences appeared.

Scores on Aviation Weather Knowledge Categories (Lanicci et al Weather Phenomena 31 Questions; alpha = .76 Weather Hazard Products 80 Questions; alpha = .91 Weather Hazard Product Sources 10 Questions; alpha = .66 3x4 Mixed ANOVA 2 Significant Main effects No significant Interaction effect A main effect did occur for knowledge categories, Wilks’ Lambda = .62, F(2, 199) = 62.19, p = .000, partial η2= .39. Post hoc paired-samples t-tests with a bonferroni correction of the three knowledge categories revealed no significant difference between scores on knowledge of Weather phenomenology and Weather hazard products, t(203)=1.82, p = .07. Weather hazard product source scores (M = 68.73, SD = 20.64) were significantly higher than both Weather phenomenology (M = 58.98, SD = 16.26) with t(203) = 9.74, p = .000, and Weather hazard products (M = 58.05, SD = 16.05) with t(203)= 11.45, p = .000.   Higher scores on Weather hazard product sources questions may be indicative of the product source questions being easier than the questions about phenomenology and/or weather products themselves. Alternately, it may be pilots are better trained in weather product sources than the other two categories of knowledge. The main effect comparing the four pilot ratings was also significant, F(3, 200) = 11.07, p = .000, partial η2= .14, suggesting there was a difference between the ratings on knowledge scores. Post hoc analysis showed student pilots (M = 51.82, SD = 2.38) scored significantly lower than private (M = 60.41, SD = 1.80), private w/ instrument (M = 65.79, SD = 2.16), and commercial w/ instrument pilots (M = 69.95, SD = 2.38). However, private pilots did not differ significantly from privates pilots with instrument ratings (p = .23) and private pilots with instrument ratings did not differ significantly than commercial pilots with instrument (p = .57).

Weather Phenomena Subcategories 4 x7 Mixed ANOVA Impact of Pilot Certificate/Rating and Weather Phenomena Subcategories on Score Both main effects were significant; no interaction Main effect for Weather Phenomena Icing and Turbulence (≈ 70%) Definitions of LIFR, IFR, MVFR, and VFR (≈ 65%) Thunderstorms, Satellite, Radar, and Lightning concepts (≈ 60% and below) A 3 x 7 mixed analysis of variance was conducted to evaluate the impact of Pilot Rating (Student, Private, Instrument, Commercial) and Weather Phenomenology Subcategory (1201, 1202, 1204, 1206, 1003, 1011, 1013) on knowledge score. There was a significant main effect of Weather Phenomenology Subcategories on score, Wilks’ Lambda = 0.43, F (6, 195) = 43.14, p < 0.05, partial eta squared = 0.57. Post hoc pairwise comparisons were performed on Weather Phenomenology Category levels to investigate differences of scores. Participants scores on Icing and Turbulence (1206 and 1202) were significantly higher than their scores on definitions of LIFR, IFR, MVFR and VFR (1201e) which, in turn, were significantly higher than their scores on Thunderstorms, Satellite, Radar, and Lightening concepts (1204, 1003 1011, 1013). Regardless of the weather phenomenology subcategories, there was a significant main effect of Pilot rating on scores, F (3, 200) = 12.35, p < 0.05, partial eta squared = 0.156. Similar to the analysis on weather phenomenology analysis reported previously in this report, Bonferroni Post Hoc comparisons were performed to evaluate differences in scores within pilot rating levels. Student pilots performed significantly lower on weather phenomenology questions than Private, Instrument and Commercial rated pilots, p<0.05. Private rated pilots scores were significantly lower than commercial rated pilot scores, p<0.05 but not lower than instrument p>0.05. There was also not a significant difference between instrument and Commercial rated pilot scores, p>0.05.

Weather Hazard Products Subcategories 4 x7 Mixed ANOVA Impact of Pilot Certificate/Rating and Weather Hazard Product Subcategories on Knowledge Score Two significant main effects; no interaction Main effect for Weather Hazard Product Interpreting upper level charts (≈75%) Interpreting convective SIGMETs and surface charts (≈ 65 %) Interpreting surface weather and PIREPS, AIRMETS, satellite data, infrared visible, water vapor, and radar (≈55%) A 4 x 7 mixed ANOVA was conducted to evaluate the impact of Pilot Certificate/Rating (Student, Private, Instrument, Commercial) and Weather Hazard Product Subcategory (2001, 2002, 2003a, 2005, 2022, 2023, 2026) on knowledge score. Regardless of Pilot certificate/rating, there was a significant main effect of Weather Hazard Product Subcategories on score, Wilks’ Lambda = 0.274, F (6, 195) = 86.31, p < 0.05, partial eta squared = .726. Post hoc pairwise comparisons were performed on the Weather Hazard Products Subcategories to investigate differences in scores. Participants’ scores were significantly higher on interpreting upper level charts (2002) than the scores on interpreting convective SIGMETs and surface charts (2003 and 2026), which in turn, were significantly higher than the scores on interpreting surface weather and PIREPs, interpreting AIRMETs, interpreting satellite data, infrared visible, and water vapor, and interpreting weather radar (2001, 2005, 2022, 2023). Regardless of Weather Hazard Product Subcategory, there was a significant main effect of Pilot certificate/rating on score, F (3, 200) = 11.85, p < 0.05, partial eta squared = 0.151. Bonferroni Post Hoc comparisons were performed to evaluate differences in scores within pilot rating levels. Student rated pilots performed significantly lower than Private, Instrument and Commercial rated pilots, p < 0.05. Private rated pilots scores were significantly lower than commercial rated pilot scores, p < 0.05 but not significantly lower than instrument rated pilots scores p > 0.05. Scores of instrument rated pilots did not significantly differ from those of Commercial rated pilots p > 0.05. No significant interaction occurred between Weather Hazard Product Subcategory and Pilot rating, Wilks’ Lambda = .0.928, F(18, 552) = 0.826, p = .67, partial eta squared = .025. Indicating that there is not a combined effect of Pilot rating and Weather Hazard Product Subcategory on score.

Weather Product Hazard Source Subcategories 4 x 3 Mixed ANOVA Impact of Pilot Certificate/Rating and Weather Hazard Product Sources Subcategories on Knowledge Score Both main effects were significant; no interaction Main effect for Weather Hazard Product Source Subcategories When to use weather product sources (≈ 72%) Weather issues in Flight planning in general (≈ 70%) How flight plan weather products are constructed (≈ 70%) 4 x 3 mixed ANOVA was conducted to evaluate the impact of Pilot Certificate/Rating (Student, Private, Instrument, Commercial) and Weather Hazard Product Source Subcategory (3007, 3007a, 3201) on knowledge score. Regardless of Pilot certificate/rating, there was a significant main effect of Weather Product Source Subcategories on scores, Wilks’ Lambda = .732, F (1, 200) = 73.27, p < 0.05, partial eta squared = .268. Post hoc pairwise comparisons were performed on the Weather Products Source Subcategories to investigate differences. Participant’s scored significantly higher on knowledge of when to use flight planning product sources questions (3201) than they did on knowledge of how flight planning products are constructed and flight planning in general (3007a and 3007). A closer inspection of the means in Table 15, however, indicates that the reason the 3201 was higher was from the pilots with both commercial and instrument ratings scoring highly on that subcategory. Regardless of subcategory of weather hazard product sources, there was also a significant main effect on the levels within Pilot rating, F (3, 200) = 6.428, p < 0.05, partial eta squared = .088. Bonferroni Post Hoc comparisons were performed to evaluate differences in scores within pilot rating levels. Student rated pilots performed significantly lower on weather products source questions than Instrument and Commercial rated pilots, p < 0.05. However, there was not a significant difference between Student pilots’ scores and Private pilots’ scores, p > 0.05. Private rated pilots scores were significantly lower than commercial rated pilot scores, p < 0.05. However, there was not a significant difference between instrument and commercial with instrument scores, p > 0.05. No significant interaction occurred between Pilot Rating and Weather Hazard Products Source subcategories, Wilks’ Lambda = .975, F(3, 200) = 1.73, p = .162, partial eta squared = .025. Indicating that there is not a combined effect of Pilot rating and Weather Hazard Products Source subcategories on score.

Good news – Training helps! * One-way ANOVA Significant between groups effect

Training Experience Estimated Months since last Weather Training Last Weather Training Experience (in Months) Training Experience Estimated Months since last Weather Training   M (SD) Median Private-in-Training 4.53 (7.81) 2.00 Private 12.55 (29.46) 5.50 Instrument 12.53 (27.51) Note commercial only Mean (SD) = 19.28 (39.71); median = 7.5; private with instrument = 7.69(8.28); median = 4.0 (Note: Private with instrument reported 8 months, and Commercial pilots reported 19+ months)

DISCUSSION and CONCLUSION

Discussion Test questions/Instrument: GA Pilots knowledge Used a systematic approach that followed guidelines in assessment instrument development. Measure has content validity and initial evidence that scores discriminate between pilots of differing levels of training. Instrument generated a spread of scores reflecting both high and low aviation weather knowledge. GA Pilots knowledge Results indicate gaps in aviation weather knowledge! Limitations/Future Research Need to assess criterion validity of questions Need older GA pilots to take the questions Current study provides an instrument that can assess GA pilot weather knowledge, and in turn, assess future Wx Training Programs. study used a systematic approach that followed FAA (2015) guidelines in the assessment instrument development. The measure has content validity and initial evidence that test scores discriminate between pilots of differing levels of training as well as between different type of aviation weather information. The test also generated a spread of scores that reflect both high and low aviation weather knowledge.

Questions?