1. Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department

2. Aircrew Information Needs Flight Information –Speed, altitude, attitude etc. Systems information –RPM, temperature, voltage, fuel etc. Navigation information –Where am I? Where is the airport? Where is that mountain? Mission Information –Targets, Traffic, Threats, etc

3. Evolution of the Navy Cockpit

4. Effective Cockpit Design Cockpits are designed to a happy medium

5. The Problem Cockpits are static Pilots’ information needs and capabilities are not –Performance changes with arousal –Modalities matter Central Executive Phonological loopVisuospatialsketchpad An adaptive cockpit would have to read the pilot’s mind

6. Reading the Mind Cognitive state is brain state Neuroimaging techniques monitor brain state

7. Neuroimaging Techniques Electromagnetic techniques –EEG/ERP, MEG/ERF Measure electromagnetic consequences of brain activity –Timing of brain activity –Intensity of brain activity Hemodynamic techniques –fMRI, PET, fNIRS Measure changes in blood flow and oxygenation associated with brain activity –Location of brain activity »Different locations are associated with different functions

8. AugCog and CogPit DARPA’s Improving Warfighter Information Intake Under Stress (formerly Augmented Cognition) Program –Apply neuroimaging to solve human factors problems Cognitive Cockpit (CogPit) –Joint QinetiQ/Alion/NAVAIR project to apply neuroimaging to aviation

9. Constraints on Techniques Equipment –Cost –Size –Power consumption –“Weird Stuff” –Comfort

10. Constraints on Techniques Signal processing

11. Techniques of Choice EEG –Spectral Power Density –Interelectrode coherence fNIRS –Blood Oxygenation

12. How it Works Hook the “pilot” up

13. Calibrate 3 levels of tracking task 3 levels of Bakan task 4 combinations of Bakan and Tracking 5

14. CogMon Records spectral power and interelectrode coherence Produces linear model relating physiological signal to different levels of tracking and Bakan task Rates subsequent recordings against model to produce judgment of pilot spatial and verbal workload in real time

15. ATAS Cockpit Low cost, maximally open source Allows great flexibility to prototype “mitigation strategies”

16. Knowledge Coordinator Assesses pilot workload and mission context. Triggers workload mitigations

17. CogPit Closed Loop Control Inputs Information Mitigations Mission Context Cognitive Workload Measurement EEG/fNIR

18. Attention –Declutter –Alerts Executive Function –Function Automation Working Memory –Sequencing Sensory Input –Multimodal Cueing Mitigations

19. Does it work? Tests at NAVAIR are just getting underway Specifics –Detection of cognitive state Recent results from CogPit are promising Actual flight tests of EEG workload monitor by other labs (AFRL, NRC Canada/U. of Iowa) –Mitigation Strategies Too early to tell for current CogPit Needs more focus

20. Will the user accept it? You make the call…

21. Road Ahead The full closed-loop technology is not ready –Imaging equipment is bulky, temperamental, and uncomfortable –Mitigations are still in development Bits of the technology could be useful in the near term.

22. First Step: Hypoxia? Poses danger to all high altitude operations Factor in at least 16 mishaps (3 fatalities) in Naval aviation since 2001 Naval Aviation has lost 0 TACAIR assets to enemy fire since Desert Storm

23. Hypoxia Currently mitigated with altitude chamber training Symptoms are subtle and variable Fundamentally strikes at cognitive ability Current countermeasure hinges on ability of broken system to diagnose itself

24. Hypoxia Detection with fNIRS fNIRS passes near infrared light through the skull and measures absorption of different wavelengths –Oxygenated and deoxygenated blood preferentially absorb different wavelengths –Yields measure of both blood volume and blood oxygenation

25. fNIRS potentially provides the basis for a hypoxia and G-LOC warning system Open questions: –How reliable would such a system be? –What do we do with the information?

26. Take Away CogPit is a platform to develop cockpits that “read the pilot’s mind” to provide the pilot with the right information at the right time The full technology is still a ways off, but subsets of it may be able to provide benefits to aviators much sooner