1. 1 Ruben Strenzke 2011 Assuming the Human‘s Cognitive State as Basis for Assistant System Initiative © UniBwM / LRT-13 / 2011 Ruben Strenzke Universität der Bundeswehr München Faculty of Aerospace Sciences Institute of Flight Systems (IFS) Dagstuhl Seminar on Plan Recognition 03.-08. April 2011

2. 2 Ruben Strenzke 2011 Projects‘ Background –Manned-Unmanned Teaming Application –The Cooperative Automation Approach for Assistant Systems Assuming the Human‘s Cognitive State –Project 1: Mixed-Initiative Mission Planning –Project 2: Operator Workload Estimation Agenda

3. 3 Ruben Strenzke 2011 Unmanned Aerial Vehicle (UAV) Guidance Trend Operator-to-vehicle ratio n : 1 1 : 1 1 : n Helicopter commander onboard Ground Control Station 1 : n Today Future

4. 4 Ruben Strenzke 2011 Manned-Unmanned Teaming (MUM-T) Mission HB Homebase C Corrdidor PZ Pickupzone DZ1 Dropzone DZ2 Alt Dropzone L Leg N HB FLOT FLET C3 DZ1 C2 L1 L2 L3 L4 L5 PZ C1 DZ2 L4

5. 5 Ruben Strenzke 2011 MUM-T Human-Machine Concept [Strenzke&Schulte 2011 ] MUM-T Work System Supervisory Control (Task-based Guidance) proactive, rational, multi-aircraft- oriented agent rather reactive, greedy and single-aircraft- oriented agents

6. 6 Ruben Strenzke 2011 MUM-T Research Simulator [Uhrmann et al. 2009, 2010]

7. 7 Ruben Strenzke 2011 Multi-UAV Guidance: Human-Machine Interface Task-based interaction Recconaissance Tasks UAV selection

8. 8 Ruben Strenzke 2011 Requirement 1: The assistant system has to guide the attention of the human operator to the most urgent task. Requirement 2 : If the human operator cannot carry out the most urgent task because of overtaxing, then the assistant system has to take initiative to transfer this situation into one which can be handled normally by operator. Requirement 3: If there are tasks, the human operator is not capable to accomplish, or which are of too risky or costly, these tasks have to be allocated to the automation. Basic Requirements for Assistant Systems Cognitive Automation Approach [Onken&Schulte 2010]  Assistant system shall provide: Situation awareness improvement, work load reduction, error avoidance, error elimination… but be silent and invisible most of the time – cannot be accessed by human initiative knows and follows the same work object as the human does

9. 9 Ruben Strenzke 2011 Assistance System: Human-Machine Interface Speech synthesis monolog and display dialog 3 escalation steps: advice  proposal generation  task re-allocation

10. 10 Ruben Strenzke 2011 Example for Mixed-Initiative Dialog Assistant System takes initiative: “UAV1 needs follow-up task” Operator presses proposal-button Assistant System proposes: “add task transit A B for UAV1” Operator presses accept-button Assistant System affirms: “added task transit A B for UAV1” asf. Assistant Sytem needs to solve these questions: - What is the operator planning? - What is he/she doing? - Is his/her plan good? - Does he/she follow the plan or make errors? - Is he/she overtaxed?

11. 11 Ruben Strenzke 2011 Projects‘ Background –Manned-Unmanned Teaming Application –The Cooperative Automation Approach for Assistant Systems Assuming the Human‘s Cognitive State –Project 1: Mixed-Initiative Mission Planning –Project 2: Operator Workload Estimation Agenda

12. 12 Ruben Strenzke 2011 Regarding the Human Operator from Outside Application-specific operator model: (disregarding intermediate goals in hierarchical plans, and top-level goals are known) goals plan action + e p + e a has knows measures estimates chooses

13. 13 Ruben Strenzke 2011 Regarding Human Error and its Source workload goals plan action behavior + e b (e.g. + sweating) effects + e p + e a has knows measures assumes estimates Cognitive state shows measures generates suffers from chooses First project: Mixed-Initiative Planning

14. 14 Ruben Strenzke 2011 Online Plan Recognition/Reasoning by Planning I generate a plan and enter the UAV tasks into the system. I cannot know what the human is planning because the entering of the plan is never explicitly finished, but I treat each planned UAV task as a constraint and I try to solve the mission problem while regarding these additional constraints. If there is no solution, the human must have made a mistake (or problem is unsolvable. If there is a solution, I can check its costs against my own solution and also check if the human forgot to enter any tasks.  online possible plan set generation

15. 15 Ruben Strenzke 2011 Mixed-Initiative Mission Planner (MMP) Concept 4. Assistant system takes initiative on basis of assumed human plan (action scheduling, missing plan elements) or reference plan (suboptimal assumed human plan, infeasible system plan)  5. Assistant system urges human to make system plan converge to one of its plans if necessary 1. System plan is entered incrementally  partial or complete human plan 2. Assistant System has direct access to system plan 3. Assumed human plan is a possible completion of the system plan, i.e. it shall converge to human plan with every detail added to the system plan

16. 16 Ruben Strenzke 2011 Slave MMP Free MMP Assistant Core Diverse information sources Mission Control There is a new objective! UAV Operator I want UAV1 to... and helo to… Ground vehicle spotted! The MMP is based on the PDDL 2.2 compatible planner „LPG-td“ [Gerevini et al. 2004] user goals, user plan!  non-probabilistic, non-hierarchical, CWA

17. 17 Ruben Strenzke 2011 Given and User Goals Example  Free Slave 

18. 18 Ruben Strenzke 2011 Operator Workload Estimation workload goals plan action behavior + e b (e.g. + sweating) effects + e p + e a has knows measures assumes suffers from estimates Cognitive state shows generates chooses measures Second project: Workload Estimation

19. 19 Ruben Strenzke 2011 Workload Estimation by HMMs [Donath et al. 2010, 2011] search for objectsidentify objectfurther tasks... normal (excl. e a /e b )HMM1HMM3... too high (incl. e a /e b )HMM2HMM4... task workload  Video! -Workload can be estimated by regarding operator behavior -behavior differs between tasks and workload levels -Measuring of visual and manual interaction with the system -has been done offline -for one person:

20. 20 Ruben Strenzke 2011 Challenges: -Online Generative Plan Recognition -Having Data-Fragments from Different Process Steps workload goals plan action behavior + e b effects + e p + e a has Cognitive state shows If we have assumptions about his plan, it‘s easier to assume what he is doing! If we have assumptions about his goal(s), it‘s easier to assume what he is planning! generates suffers from chooses knows measures assumes estimates measures

21. 21 Ruben Strenzke 2011 The Obligatory Last Slide