1. Mutual Empowerment in Human-Agent-Robot Teams 16 December 2010 HART Workshop Jurriaan van Diggelen

2. Problem statement Achieve more with less people Automation can help to: –Make better use of available semi-structured information sources –Support decision makers in dealing with the complexity of problems (war amongst the people)

3. The big number cruncher Monolithic approach, BNC replaces existing infrastructure AI-complete Sensor data Twitter data UAV images Problem solution

4. Towards a human-machine team solution Solution must be provided by a human machine team Mutual empowerment seeks to improve team performance by: –Compensating weaknesses of humans and machines –Optimizing strengths of humans and machines

5. Types of Mutual Empowerment humanmachine HMI Intelligent Interfaces humanmachine CMI CCI HMI User empowerment Distributed Artificial Intelligence Collective Intelligence

6. ME handbook Goal

9. Methodology Functional design PrototypingValidation Use cases Claims Cognitive requirements Ontologies Performance measures Tests/benchmarks System requirements Functional modules RDF interface specifications Prototypes Mixed reality validation Data collection Tool support Domain Exploration Domain Human Factors Technology

10. Situated Cognitive Engineering Methodology supports –Incremental design –Reuse of earlier work (Prototypes, tests, requirements, use cases) –Collaborative development

11. Example

12. Phase 1: domain exploration Domain –USAR –UGV, UAV –Operators in field Human Factors –Maintaining situation awareness –Cognitive overload –Adaptive teams Technology –Collaborative tagging, crowd sourcing –Mixed initiative systems –Adaptive/ adaptable automation

13. Phase 2: Functional design (1) Use cases Cognitive requirements Claims UC 23 UAV classifies camera image as victim with certainty-level Unsure Operator of Robot1 is notified of the potential victim and views the camera images Operator of Robot1 classifies the image as victim with certainty level Certain Operator of Robot2 is notified about the victim … CR 5.1 Uncertainty management Operators and agents can publish and change the certainty value of information Use cases: UC 23 CR 5.1 + improves situation awareness of operators and agents - increases cognitive taskload

14. Phase 2: Functional design (2) Ontologies Performance measures –E.g. situation awareness measure Tests/benchmarks –Test for evaluating performance something actioneventitem victimrobot

15. Phase 3: Prototyping Develop system requirements that implement the cognitive requirements. Bundle system requirements in functional modules. Reuse existing base platform

16. Trex

17. Filter: which data do you want to see? selection of semantic tags in Sparql Projection: How do you want to see the data? graphical object with attachment- points for semantic tags

18. Functional modules supported by Trex User configurable information filters User configurable information visualization Realtime semi-structured data exploration Collective relevance assessment Uncertainty management Human-in-the-loop AI PQRST HumanMachineCrowdMachine Human-in-the-loop AI

19. DEMO

20. Future work Develop functional modules for: –Joint conflict resolution –Adaptive Interruptiveness –Network awareness –Policy awareness –Capability awareness –Activity awareness

21. Conclusion Mutual Empowerment library provides a flexible way to –Increase application possibilities of AI –Employ potential of collective intelligence –Reuse and structure our knowledge of human-machine collaboration tools

22. Technology Investigation Domain AnalysisHuman Factors Metrics Cognitive Requirements ClaimsUse cases Ontologies Tests Exploration Functional design Prototyping Core functions Functional modules System Requirements Prototype RDF interfaces Testing SimulationTest participantsEmpirical results