1. Improving Aviation Safety with Information Visualization: Airflow Hazard Display for Pilots Cecilia R. Aragon IEOR 170 UC Berkeley Spring 2006

2. IEOR 1702 Acknowledgments This work was funded by the NASA Ames Full- Time Graduate Study Program (Ph.D. in Computer Science at UC Berkeley) Thanks to my advisor at UC Berkeley, Professor Marti Hearst, and Navy flight test engineer Kurtis Long Thanks to Advanced Rotorcraft Technology, Inc. for the use of their high-fidelity flight simulator

3. Spring 2006IEOR 1703 Talk Outline Introduction Related Work Preliminary Usability Study Flight Simulation Usability Study Conclusions and Further Work

4. Spring 2006IEOR 1704 Introduction

5. Spring 2006IEOR 1705 Motivation Invisible airflow hazards cause aircraft accidents –Wind shear –Microbursts –Vortices (turbulence) –Downdrafts –Hot exhaust plumes Crash of Delta Flight 191 at DFW 1985 (microburst) NTSB database 1989-99 –21,380 aircraft accidents –2,098 turbulence/wind related

6. Spring 2006IEOR 1706

7. Spring 2006IEOR 1707

8. Spring 2006IEOR 1708 The Problem Invisible airflow hazards cause aircraft accidents –Air is invisible, so pilots can’t see hazards –If air flows past obstacles, flow will become more turbulent Helicopters are especially vulnerable –Rotorcraft aerodynamics –Must operate in confined spaces –Operationally stressful conditions (EMS, military operations, shipboard operations)

9. Spring 2006IEOR 1709 A Possible Solution If pilots could see hazards, could take appropriate action New lidar technology suggests a solution –Lidar (light detection and ranging) is essentially laser radar. A laser transmits light which is scattered by aerosols or air molecules and then collected by a sensor. Doppler lidar can detect the position and velocity of air particles. My research focuses on the human interface -- how to visualize the sensor data for pilots -- too much information could overload pilot during critical moments

10. Spring 2006IEOR 17010 Research Approach User-centered (iterative) design process Simulated interface for head-up display (HUD) based on lidar sensors that scan area ahead of helicopter and acquire airflow velocity data Focused on helicopter-shipboard landings Importance of realism: –Used actual flight test data from shipboard testing, high-fidelity helicopter simulator, experienced military and civilian helicopter pilots

11. Spring 2006IEOR 17011 Rationale for using Shipboard Landings Why focus on helicopter shipboard landings? –Problem is real: dangerous environment, want to improve safety –Ship superstructures always produce airwake Large quantities of flight test data due to demanding environment

12. Spring 2006IEOR 17012 Related Work

13. Spring 2006IEOR 17013 Related Work Flow visualization Aviation displays Navy “Dynamic Interface” flight tests

14. Spring 2006IEOR 17014 Flow visualization Detailed flow visualizations designed for scientists or engineers to analyze at length Much work has been done in this area [Laramee et al 04] –Streamlines, contour lines (instantaneous flow) [Buning 89], [Strid et al 89], [Helman, Hesselink 91] –Spot noise [van Wijk 93], line integral convolution [Cabral, Leedom 93], flow volumes [Max, Becker, Crawfis 93], streaklines, timelines [Lane 96], moving textures [Max, Becker 95] (unsteady flow) –Automated detection of swirling flow [Haimes, Kenwright 95] –Terrain and turbulence visualization [LeClerc et al 02] But usually no user tests [Laidlaw et al 01], and not real-time

15. Spring 2006IEOR 17015 Aviation displays Synthetic and enhanced vision and augmented-reality displays [Hughes et al 02], [Parrish 03], [Spitzer et al 01], [Kramer 99], [Wickens 97] Weather visualization, microburst detection [NASA AWIN, TPAWS], [Latorella 01], [Spirkovska 00], turbulence detection/prediction [Britt et al 02], [Kaplan 02] Wake vortex visualization [Holforty 03]

16. Spring 2006IEOR 17016 Navy Ship-Rotorcraft Compatibility Flight Testing (“Dynamic Interface”) Very hazardous environment [Wilkinson et al 98] Significant amounts of flight testing [Williams et al 99] Recognized need for pilot testing Goal: improve safety

17. Spring 2006IEOR 17017 Current state of the art Ship/helicopter flight tests, wind tunnel tests, CFD Develop operational envelopes –Limit allowable landing conditions significantly –Envelopes are conservative for safety reasons Pilots use intuition, but accidents still occur

18. Spring 2006IEOR 17018 Preliminary Usability Study

19. Spring 2006IEOR 17019 Preliminary usability study: goals Assess efficacy of presenting airflow data in flight Obtain expert feedback on presentation of sample hazard indicators to refine design choices

20. Spring 2006IEOR 17020 Usability study: low-fidelity prototype Rhino3D (3D CAD modeling program) –Easy access to ship models, ease of rapid prototyping –Chosen over 2D paper prototype, MS Flight Simulator, WildTangent, VRML-based tools, Java and Flash Series of animations simulating helicopter’s final approach to landing Different types of hazard indicators Get pilot feedback and suggestions (interactive prototyping)

21. Spring 2006IEOR 17021 Low-fi usability study screen shots

22. Spring 2006IEOR 17022 Low-fi usability study screen shots

23. Spring 2006IEOR 17023 Low-fi usability study participants Navy helicopter test pilot, 2000 hours of flight time, 17 years experience Navy helicopter flight test engineer, 2000+ hours of simulator time, 100 hours of flight time, 17 years experience Civilian helicopter flight instructor, 1740 hours of flight time, 3 years experience

24. Spring 2006IEOR 17024 Low-fi usability study results All participants said they would use system Feedback on hazard indicators: –Color: all preferred red/yellow only –Transparency: should be visible enough to get attention, but must be able to see visual cues behind it –Depth cueing: all preferred shadows below object, #1 said shadows alone sufficient. #2 wanted connecting line. No one wanted tick marks or numeric info. –Texture: #1, #2 didn’t want. #3 suggested striping –Shape: Rectilinear and cloud shapes favored. Keep it simple! Watch for conflicting HUD symbology.

25. Spring 2006IEOR 17025 Low-fi usability study results (cont’d) Motion is distracting! 1: absolutely no motion 2: didn’t like motion 3: slow rotation on surface of cloud OK, nothing fast

26. Spring 2006IEOR 17026 Low-fi usability study conclusions They want it! Keep it simple –Color: red & yellow only (red = danger, yellow = caution) –Less complex shapes preferred Use accepted symbology/metaphors –Watch for conflicting HUD symbology Decision support system, not scientific visualization system –Show effects rather than causes –Don’t want distraction during high-workload task –Preference for static rather than dynamic indicators

27. Flight Simulation Usability Study

28. Spring 2006IEOR 17028 Flight Simulation Usability Study Implement visual hazard display system in simulator based on results from low-fidelity prototype Advanced Rotorcraft Technology, Inc. in Mountain View, CA, USA –High-fidelity helicopter flight simulator –Accurate aerodynamic models Use existing ship and helicopter models, flight test data Simulated hazardous conditions, create scenarios, validated by Navy pilots and flight engineers

29. Spring 2006IEOR 17029 Flight Simulation Usability Study: Participants 16 helicopter pilots –from all 5 branches of the military (Army, Navy, Air Force, Coast Guard, Marines) –civilian test pilots (NASA) –wide range of experience 200 to 7,300 helicopter flight hours (median 2,250 hours) 2 to 46 years of experience (median 13 years) age 25 to 65 (median age 36) No previous experience with airflow hazard visualization

30. Spring 2006IEOR 17030 Simulation Experiment Design 4 x 4 x 2 within-subjects design (each pilot flew the same approaches) 4 shipboard approach scenarios 4 landing difficulty levels (US Navy Pilot Rating Scale - PRS 1-4) Each scenario was flown at all difficulty levels both with and without hazard indicators Orders of flight were varied to control for learning effects

31. Spring 2006IEOR 17031 Airflow Hazard Indicators in Simulator

32. Spring 2006IEOR 17032 Simulation Experiment Design Red/None Test benefit of hazard indicator combined with pilot SOP Controllability in question; safe landings not probable LD 4 Yellow/None Test benefit of hazard indicator Maximum pilot effort required; repeated safe landings may not be possible LD 3 Yellow/None Test negative effects of hazard indicator Moderate pilot effort required; most pilots able to land safely LD 2 None ControlNo problems; minimal pilot effort required LD 1 Hazard indicator PurposeDescriptionLanding difficulty

33. Spring 2006IEOR 17033

34. Spring 2006IEOR 17034 Dependent Variables Objective data: sampled at 10 Hz from simulator –aircraft velocity and position in x, y, z –lateral and longitudinal cyclic position and velocity –collective and pedal positions and velocities –landing gear forces and velocities –(A “crash” was defined as an impact with the ship deck with a vertical velocity of more than 12 fps) Subjective data: 21-probe Likert-scale questionnaire administered to pilots after flight

35. Spring 2006IEOR 17035 Hypotheses 1. Crash rate will be reduced by the presence of hazard indicator (LD 3). 2. Crashes will be eliminated by red hazard indicator if a standard operating procedure (SOP) is given to the pilots (LD 4). 3. Hazard indicator will not cause distraction or degradation in performance in situations where adequate performance is expected without indicator (LD 2). 4. Pilots will say they would use airflow hazard visualization system

36. Spring 2006IEOR 17036 Hypothesis 1 confirmed Presence of the hazard indicator reduces the frequency of crashes during simulated shipboard helicopter landings (t-test for paired samples, t=2.39, df=63, p=0.00985). 19% -- > 6.3%

37. Spring 2006IEOR 17037 Hypothesis 2 confirmed Presence of the red hazard indicator combined with appropriate instructions to the pilot prevents crashes (t=4.39, df=63, p 0%

38. Spring 2006IEOR 17038 Hypothesis 3 No negative effect of hazard indicator. 8%-- > 8%

39. Spring 2006IEOR 17039 Hypothesis 3 (cont’d) Pilots believe hazard indicators were not distracting (Probe 6 results).

40. Spring 2006IEOR 17040 Hypothesis 4 confirmed Pilots would use the system (Probe 21 results).

41. Spring 2006IEOR 17041 Pilot workload: Power spectrum analysis of control inputs

42. Spring 2006IEOR 17042 Go-Arounds (Aborted Landings) Does the presence of the hazard indicator increase the go-around rate? No significant differences found.

43. Spring 2006IEOR 17043 Analysis by Pilot Experience Level Does pilot experience level have any effect on the benefits produced by the hazard indicators? To find out, divide pilots into three groups:

44. Spring 2006IEOR 17044 Analysis by Pilot Experience Level (cont’d) Same general trends -- but small sample size No significant difference between the groups

45. Spring 2006IEOR 17045 Analysis of Subjective Data 94% found hazard indicators helpful

46. Spring 2006IEOR 17046 Analysis of Subjective Data (cont’d) Is motion (animation) helpful or distracting?

47. Spring 2006IEOR 17047 Conclusions and Further Work

48. Spring 2006IEOR 17048 Conclusions Flight-deck visualization of airflow hazards yields a significant improvement in pilot ability to land safely under turbulent conditions in simulator Type of visualization to improve operational safety much simpler than that required for analysis Success of user-centered design procedure

49. Spring 2006IEOR 17049 Further Work Additional data analysis Further studies Steps toward system deployment Extensions to other areas

50. Spring 2006IEOR 17050 Additional data analysis Power spectrum analysis of control input data Flight path deviations and landing dispersion Quantitative measures of landing quality

51. Spring 2006IEOR 17051 Further studies Quantitatively compare hazard indicators with other types –light/buzzer in cockpit –animated indicator –numeric information such as airflow velocity Adaptive displays –more detailed at beginning of approach, simpler at end –how adapt to pilot state? physiological sensors vs. pilot-selectable modes

52. Spring 2006IEOR 17052 Steps toward system deployment Collaboration with lidar developers, integration with real-time data Integration with synthetic vision displays Augmented reality image registration

53. Spring 2006IEOR 17053 Extensions to other areas Other aviation domains –aerial firefighting –search and rescue –offshore oil platforms –unmanned aerial vehicles (UAVs) –fixed-wing operations Space exploration Emergency response Automobiles or other motor vehicles

54. Spring 2006IEOR 17054 Extra Slides

55. Spring 2006IEOR 17055 Crash Statistics for All Landing Difficulties

56. Spring 2006IEOR 17056 Control group (LD 1) No significant difference between crash rate at LD 1 (control) and LD 2 with hazard indicator and LD 3 with hazard indicator. 9% - 8% - 6%

57. Spring 2006IEOR 17057 Learning Effects? First half: 25 crashes/224; second half: 22/224. Not a significant difference -- > no apparent bias.

58. Spring 2006IEOR 17058 Airflow Hazard Indicator (Aft Scenario)

59. Spring 2006IEOR 17059 Airflow Hazard Indicator (Bow Scenario)

60. Spring 2006IEOR 17060 Pilot Demographics

61. Spring 2006IEOR 17061

62. Spring 2006IEOR 17062

63. Spring 2006IEOR 17063

64. Spring 2006IEOR 17064

65. Spring 2006IEOR 17065

66. Spring 2006IEOR 17066

67. Spring 2006IEOR 17067

68. Spring 2006IEOR 17068

69. Spring 2006IEOR 17069

70. Spring 2006IEOR 17070

71. Spring 2006IEOR 17071

72. Spring 2006IEOR 17072

73. Spring 2006IEOR 17073

74. Spring 2006IEOR 17074 Low-fi usability study: methodology 1 ½-hour interview in front of projection screen, videotaped Two experimenters, one operates computer, one asks questions Display series of hazard indicators in Rhino3D Variables: –Shape –Color –Transparency –Texture –Depth cueing –Motion Ask specific and open-ended questions throughout the interview

75. Spring 2006IEOR 17075 “The Holy Grail” – Quote from Pilot #1 “The holy grail…” –increase safety and –increase operational capability Usually you either have: –increased safety but have operational restrictions…or –greater operational capability but have risks associated with employing that additional capability... “In this case you actually have a concept that could potentially give you both.”