1. Adaptive Automation for Human Performance in Large-Scale Networked Systems Research Team: George Mason University Raja Parasuraman Tyler Shaw Ewart de Visser Amira Mohammed-Ameen Andre Garcia

2. Task allocation among humans/agents Probabilistic models of human decision- making in network situations Decentralized control search and planning Information fusion Network performance as a function of topology Communication, evolution, language CMU Psychology MIT Pitt Level 1,3 Level 2 Level 1,2 Level 1- 2.5 Level 1-3 Level 1,3 Level 1 Level 1- 2.5 Level 1,2 Level 3 ? Level 2 Level 1- 3 Level 1,2 Level 1,3, 4 Level 4 Level 2 Level 3 Level 2, 3 Adaptive automation Level 1,2 Level 1 Level 1,2Level 2 CMU Robotics CMU Robotics Level 1 Scaling of cognitive performance and workload Scaling of cognitive performance and workload Level 1,2 Cornell MIT GMU

3. 3 Research Goals Develop validated theories and techniques to predict behavior of large-scale, networked human-machine systems involving unmanned vehicles Model human decision making efficiency in such networked systems Investigate the efficacy of adaptive automation to enhance human-system performance

4. 1st Year Studies First Year Review October 2, 2009 Modeling Decision Making under High Cognitive Load  Complexity: Numbers of Enemy Assets and Messages Verification Behavior in Networked Systems (in progress)  Effects of Message Pedigree and Training Adaptive Automation to Support Human Supervision of multiple UAVs  Complexity: Number and types of UAVs  Playbook Interface vs. Scripts vs. Tools Neuroergonomics-Based Adaptive Automation (in progress)  Transcranial Doppler Sonography (TCD), Cerebral Blood Flow and Cognitive Load

5. Cognitive Load Varies with N (from Mike Lewis, U Pitt) O(1) O(n) O(>n) Cognitive limit = f(W) N of Robots Our studies focus on O(n) and O(>n) cases Cognitive limit varies with working memory capacity W

6. Study 1 Hypothesis: Cognitive load limit on operator in O(n) case is predictable from  Number of enemy asset targets  Message complexity  Individual working memory capacity, W?

7. Add picture Information Panel Approximate attack range Message Center Neutral Zone Teammate zone Neutral asset Enemy asset DDD STUDY 1

8. Study 1 DDD 4.0 Simulation  Destroy as many enemy assets (single or squadrons) as possible  Prevent enemy asset incursion into neutral zone  Communicate target information to teammate  (Neutral assets can also turn into enemy targets) Two levels of enemy cognitive load (low, high)  number of enemy asset incursions  Speed of enemy asset incursion Network messages from agent provide operator with critical information to achieve mission success  No messages  Noisy messages (only 20% of the messages directly relevant to mission objectives (e.g. destroy enemies)  Direct information (all messages directly relevant to mission objectives)

9. Study 1 Methods First Year Review October 2, 2009 Participants  30 adults (18 men, 12 women) aged 18-26 years, 15 were ROTC cadets Dependent measures  proportion of enemy asset incursions into neutral zone  number of enemy assets destroyed  time to destroy enemy targets  number of messages acknowledged  quality of messages passed to team-mate Overall performance score  red-zone incursions: - 2 points/second  destroy enemy asset: + 100 points  enemy attack on own asset: - 100 points  friendly fire: - 50 points

10. Study 1 Results: Red Zone Performance First Year Review October 2, 2009

11. Study 1 Results: Overall Performance First Year Review October 2, 2009

12. Study 1 Results Simple linear modeling of  Red zone performance  Overall mission score (combining all performance measures) Performance = w 1 + w 2 n e + w 3 p(m) +   n e = number of enemy assets  p(m) = proportion of relevant messages  w 1, w 2, w 3 = weights   = error Variance accounted for:  Red zone performance: 46%  Overall mission score: 42%

13. Study 1 Results Unexpected finding  Very high inter-individual variability in performance  Range of scores: red zone incursion proportion: 0 to 80% overall points: -2994 to 2600 Working memory capacity WM Individual differences in Operation span measure of working memory w 1 + w 2 n e + w 3 p(m) + w 4 WM +  Variance accounted for with:  Red zone performance: 76%  Overall mission score: 62%

14. 14 Follow-Up to Study 1 Collaboration with Cornell Probabilistic modeling of human decision-making performance Compare model to simple linear model Identify and quantify human “cognitive bottlenecks” Identify points for “adaptive tasking” or adaptive automation Scale up to larger networks (more UVs and agents)

15. Study 2 (in progress) Problem: Human operator under-trust (skepticism) and over-trust (complacency) can limit usefulness of information sources in large networked systems Hypothesis: Trust and complacency in networked systems can be indexed by verification behavior (information sampling)  Number and complexity of messages  Message pedigree (hub vs. isolated node in network)

16. Study 2 Forest firefighting simulation Operator services UAV requests to combat different types of forest fires with unique requirements Assets: 2 (simple) or 5 (complex) UAV types Operator receives messages from network requesting service Automated agent recommends decision choice Operator can verify UAV and fire information parameters Source of information: Base HQ (hub node) or team- mate (isolated node)

17. Training Manipulation Two 10 minute practice sessions (one for each UAV complexity level) in which 20 forest fires need to be serviced Information group  Told there will be unreliable messages, but experience none.  Told that messages from Base HQ more reliable than from isolated node Experience group  Base HQ: 90% reliable (9 of 10 messages give correct recommendation)  Isolated node: 60% reliable (6 of 10 messages give correct recommendation)

18. Complex scenario (5 UAV types) 1 1 1 1 1 1. Dumps Fire Retardant 1. Photographs with Infra-red 1. Photographs with TV 2. Carry smoke jumpers 2. Picks up victims 1. Photographs with TV

19. 26 June 2015 Operator receives automated dispatch requests. Click to Acknowledge Read Recommendation & Decide Select Assets to be sent and click OK Notice time FROM BASE HQ: SEND 1 BLUE FROM ISOLATED NODE: SEND 1 BLUE

20. Trust…. but Verify 26 June 2015 Operator can verify the automated agent recommendation by clicking “Display information.”

21. Study 2 Results First Year Review October 2, 2009 Full verification level Optimal verification“Skeptical” “Complacent”

22. Study 3 Hypothesis: Playbook adaptive automation enhances decision-making performance in a realistic, near-term [O(n), with n=3] simulation of heterogeneous human-UAV teams

23. Study 3 First Year Review October 2, 2009 Examine efficacy of adaptive automation — specifically the Playbook for supervisory control of multiple heterogeneous UAVs Compare three Levels of Automation  Tools  Scripts  Playbook

24. Title First Year Review October 2, 2009 Multi-UAV Simulation 3 Fixed-Wing UAVs Map Display with Named Areas of Interest (NAIs) Sensor Display Multi-Function Display UAV capabilities Alpha (blue) – paint civilian targets and – AutoTrack Targets Bravo (pink)  1) civilian targets,  2) AutoTrack Targets Ø laze weaponized* targets (cannot prosecute) Charlie (orange) – paint civilian targets – AutoTrack – Prosecute weaponized targets (cannot lase)

25. Tools (Low LOA) Manually move waypoints to position UAVs to monitor NAI. Manually paint civilian targets Autotrack humvees Manually lase weaponized targets Manually prosecute weaponized targets MUSIM STUDY 1

26. Scripts (Intermediate LOA) Automatically task individual UAV’s to monitor 1 or 2 NAI’s Manually paint civilian targets Autotrack humvees Automatically lase weaponized targets. Manually prosecute weaponized targets. MUSIM STUDY 1

27. Plays (Flexible LOA) Automatically assign ALL UAV’s to monitor ALL NAI’s Manually paint civilian targets Autotrack humvees Reconfigure team so that 2 UAVs monitor 3 NAIs while the other UAV tracks a target. Simultaneously lase and prosecute MUSIM STUDY 1

28. 3 LOAs MUSIM STUDY 1 Tools Scripts Plays

29. Study 3 Results Plays reduced target acquisition and prosecution times Plays also reduced overall mental workload

30. Study 4 (in progress) Hypothesis: Cognitive load limit on operator in O(n) case can be assessed dynamically by measuring cerebral blood flow with Transcranial Doppler Sonography (TCD) TCD measurement of operator cognitive load could be used to drive adaptive automation to support the operator

31. Transcranial Doppler Sonography (TCD) and Cerebral Blood Flow

32. Study 4 Methods First Year Review October 2, 2009 Participants  16 adults aged 18-30 (to date) Air defense task (DDD 4.0, as in Study 1)  Task load (number of enemy assets) increased unpredictably from low to high in the middle of mission  No messages or messages present conditions Cerebral blood flow velocity measured in left and right hemispheres of the brain

33. Point of transition

34. Transition Results to Date Carryover effect of cognitive load?

35. Publications Grier, R. A., Parasuraman, R., Entin, E. E., Bailey, N., & Stelzer, E. (2008). A test of intra- versus inter-modality interference as a function of time pressure in a warfighting simulation. In Proceedings of the Human Factors and Ergonomics Society. Santa Monica, CA, pp, 1229-1232. Parasuraman, R. (2009). Neuroergonomics applied to adaptive automation for supervision of multiple unmanned vehicles. In Proceedings of the Army Science Confererence, Orlando, FL. Parasuraman, R., Cosenzo, K., & de Visser, E. (2009). Adaptive automation for human supervision of multiple uninhabited vehicles: Effects on change detection, situation awareness, and mental workload. Military Psychology, 21.270-297. Parasuraman, R., de Visser, E., & Shaw, T. (2009). Individualized adaptive automation. In Proceedings of the Human Factors and Ergonomics Society Conference. San Antonio, TX.

36. Planned Studies First Year Review October 2, 2009 Extend DDD study to 2-person teams Extend modeling of decision-making performance performance to 2-person teams Extend network message verification study to examine effects of cost of verification, message reliability, and network size and topology Examine feasibility of implementing adaptive automation to mitigate cognitive load based on measurement of cerebral blood flow