Human-Information Interaction TA6 – Project 4 David Mott IBM UK

Project Overview COALITION NEEDS Project 4 aims to advance our understanding of the factors that affect shared understanding and information exploitation in military coalition environments to be assisted in collaborative problem-solving by systems that share common understanding of the knowledge and reasoning with human users to engineer the coalition socio-technical environment in ways that support collective/team cognition to exploit large bodies of diverse, unstructured information content and other information assets in the execution of their tasks PROJECT FOCUS understanding the interaction between cognitive, social and technological factors in collaborative problem-solving contexts, by cognitively-rich agent simulation supporting fact extraction from plain text by using deep linguistic processing and facilitating human-machine cognitive capabilities by externalising reasoning in a Controlled Natural Language understanding how to support human decision-makers by matching information assets to requirements and by use of meaning-rich conversational interactions ITA Peer Review, Sept. 2014

Multidisciplinary Research Team IBM UK David Braines David Mott Paul Stone Carnegie Mellon University Katia Sycara Yuqing Tang PSU Tom La Porta Nan Hu Airbus Group Gavin Powell Matthew Roberts University of Cambridge Ann Copestake Diarmuid O Seaghdha Boeing Anne Kao Stephen Poteet Ping Xue University of Cardiff Alun Preece Will Webberley IBM US Geeth de Mel University of Southampton Paul Smart Darren Richardson Dstl Kris Challa UCLA Mani Srivastava Robin Wentau Ouyang Matthew Johnson ARL Cheryl Giammanco Jon Bakdash

Key Technical Accomplishments Task 1: Collective Cognition in Coalition Environments a cognitive social simulation capability to support experimental studies of team cognition in complex task environments. an initial cognitive computational model of collaborative problem solving in a specific task context (i.e., the ELICIT task). a human experiment using the ELICIT task, designed to gather data relating to the effect of different information sharing strategies on task performance. integration of Controlled English (CE) components into the simulation capability to support linguistic forms of agent communication. ITA Peer Review, Sept. 2014

Key Technical Accomplishments Task 2: Fact Extraction and Reasoning using Controlled Natural language extraction of CE facts from natural language (NL) sentences, applying knowledge expressed in CE to transform linguistic semantics into domain semantics. extension of CE for more complex problem solving strategies with meta-reasoning, assumptions, rationale, as applied to realistic analytic tasks handling of NL ambiguities and uncertainties, extending linguistic mechanisms to allow the CE domain model to guide NL processing exploration of collaborative man-machine problem-solving systems as applied to “logic puzzles” used in training intel analysts. ITA Peer Review, Sept. 2014 5 5

Key Technical Accomplishments Task 3: Coalition Context-Aware Assistance for Decision Makers the definition and validation, including initial trials with human subjects, of a conversational protocol for free-flowing human-machine, machine-machine, and machine-human exchanges a model of resource allocation in network systems as a "stochastic knapsack problem" to handle uncertain factors like unreliable wireless medium or variable quality of sensor outputs a technology integration experiment to show how key research elements can be combined to support rapid but informed decision-making capabilities at lower echelons in coalition operations ITA Peer Review, Sept. 2014 6 6

ACT-R Cognitive Social Simulation Capability (CSSC) Insight: Cognitive architectures can be used to model socially- distributed cognitive processes and improve our understanding of the dynamics of team cognition. State-of-the-art Multi-agent simulation techniques have been used to model collective behavior. ….however, such models do not take account of the cognitive capabilities and limitations of human agents. Cognitive architectures provide computational frameworks for the development of cognitively-plausible agents. …however, few applications to team contexts. ACT-R: a mature cognitive architecture that can be used to model a broad range of cognitive processes. extend existing single-agent ACT-R research by focusing on interactions of multiple agents, allowing study of socially-distributed cognition extend ACT-R architecture to support multiple-agent research Cognitive social simulation: integration of cognitive architectures into social simulation results in cognitive social simulation. individual machine agents share the same kind of cognitive capabilities and limitations as their human counterparts. Experimentation: based on collaborative simulation of the DoD CCRP ELICIT identification task simulation capability instrumented to collect and measure performance factors based upon existing ELICIT metrics Coalition needs/benefits We need to know how to exploit the collective cognitive potential of coalition organizations. We need to assess the effect of human factors and socio-technical interventions without disrupting the operational environment. Paul Smart (Southampton) CMU, IBM UK, Airbus Group

ACT-R Cognitive Social Simulation Capability (CSSC) Goal Module Declarative Module Imaginal Module Vocal Module Procedural Module Matching Selection Execution ACT-R/CSSC Self Module Messaging Module Language Module Web Module Agent Characteristics Messaging Website Language Processor The ACT-R/CSSC is an extension to the core ACT-R architecture to support cognitive social simulation experiments.

ACT-R Cognitive Social Simulation Capability (CSSC)

ACT-R Cognitive Social Simulation Capability Result: Extended ACT-R cognitive architecture to support studies into team cognition. Custom modules support linguistic communication (language module), interaction with shared repositories (web module), and configuration of agent behaviour (self module). The simulation capability also supports the real-time participation of human subjects. Simulation capability supports studies into team-based problem solving. Current experimental efforts focused on the DoD CCRP ELICIT task. Modelling the performance of human subjects in different organizational environments. Coordinator Long-Term Goals Team Leader Understand effect of task environment features on collective cognitive performance: Team Member perform human experiments, and gather data, with a variant of the ELICIT task. extend cognitive computational models with machine agents embodied in the environment evaluate cognitive models with experimental simulation studies. Shared Repository (limited access) The ACT-R Cognitive Social Simulation capability provides a generic platform for exploring the interaction between cognitive, social and technological factors in collective cognition. ITA Peer Review, Sept. 2012

CE to support fact extraction and problem solving State-of-the-art CE can integrate modelling and reasoning capabilities to assist performance of complex cognitive tasks, including transformation of linguistic information from a deep parsing system into high value domain facts Traditional NL mechanisms have difficulty extracting detailed information about complex situations Deep parsing systems, can extract more subtle and detailed information from text, but it is a complex task to convert the linguistic output into domain-specific facts Performance of complex cognitive tasks requires integration of different modeling and reasoning capabilities in a common language Use of DELPH-IN linguistic resources The ERG, a deep and detailed grammar of English Minimal Recursion Semantics (MRS) to represent the output of the parse in a logical form Transforming linguistic semantics to domain semantics Rules expressed in CE for application of CE domain models Exploration of linguistic theories, in a way that is more natural than typical formal languages CE reasoning and modelling system User-defined domain model for expressing concepts, facts, logical rules and problem-solving strategies Assumptions to capture uncertainty, ambiguity and sentence interpretation Rationale and proof tables to visualise reasoning and sources of uncertainty Meta-model for reasoning about the language itself, e.g. mapping words to concepts and to create new rules from sentences Coalition needs/benefits Military tasks require text from NL sources to analyse, fuse and infer high value information A deep parsing system, with linguistic to domain semantic transformations, could extract more domain specific facts to support these tasks Collaborating human agents need support in problem-solving tasks in a way that is understandable and externalises their domain models and problem-solving strategies. David Mott (IBM UK) 11 Boeing, Cambridge, ARL ITA Peer Review, Sept. 2014 11

Linguistic reasoning to extract CE facts 12

Reasoning to support problem solving 13

CE to support fact extraction and problem solving CE linguistic reasoning can apply domain knowledge to fact extraction, bridging the gap between deep linguistic analysis and domain semantics; CE domain reasoning can externalise human problem solving to solve an analytic task in an explainable way. CE domain models, reasoning and problem solving strategies Long-Term Goals Improve linguistic processing and fact extraction linguistic reasoning to handle a wider range of sentences, e.g. following the MRS test suite construction of domain models from distributional semantics more domain knowledge to resolve ambiguities of sentence interpretation A machine agent that can participate in collective cognition and externalises human reasoning Improve problem solving capability Extend CE expressivity More sophisticated handling of assumptions Improve visualisation of rationale Handle more complex logical problem solving CE problem solving techniques can be applied to both fact extraction from NL sentences and reasoning from basic extracted facts to intel relevant high value information ITA Peer Review, Sept. 2012 14 14

Human-Machine Conversations State-of-the-art Insight: Human-machine conversations using natural language (NL) and Controlled English (CE) can facilitate information exploitation and asset tasking at the tactical edge. Reducing cycle time in rapid establishment of data-to-decision pipelines (D2D) still requires significant technical skills and training Handling of uncertainties in exploiting soft and hard information is a key problem, even for technical users A formal conversational protocol – captured as a CE model – enables HMC in the scope of a range of tactical-level use cases: “spot” reports (including uncertain QoI) information fusion (including rationale) asset tasking (including uncertain availability) Combining the expressivity of NL and the precision and reasoning capabilities of CE Coalition needs/benefits Low training overhead and flexible human-machine collaboration (HMC) to create D2D pipelines Exchange of information precisely and unambiguously whilst retaining human readability and flexibility, pinpointing sources of uncertainty and providing explanation Alun Preece (Cardiff) ITA Peer Review, Sept. 2014 15 IBM UK, IBM US, PSU, UCLA, ARL 15

Human-Machine Conversations NL/CE Conversational Protocol Aim: to enable conversational interactions that flow freely between natural language and CE, specifically to support dynamic establishment of D2D pipelines Conversational protocol draws on research in agent communication languages and philosophical linguistics (speech acts) Types of interaction include: Confirmatory dialogues to reduce ambiguity (“did you mean…?”) Query-response, with optional rationale (“Why?”) Machine-generated (text/graphical) “gist” forms to summarise complex CE ITA Peer Review, Sept. 2014 16

Human-Machine Conversations Main aims: to determine the degree to which a software agent can extract CE from unrestricted NL conversations to test the agent’s robustness with untrained users Experimentation: scenario-based Experimentation: human subjects 20 subjects viewed a series of scenes and described them in natural language via a text-based interface The system provided feedback in CE and a score (terms recognised by system) D2D vignette from earlier ITA work Technology integration experiment including: conversational agent (steps 1+4), CE Store (2), SAM (3) Demo with CERDEC AI TECD Jan 2014 Takes account of uncertainty of asset availability (e.g. bandwidth) and QoI (e.g. due to weather) ITA Peer Review, Sept. 2014 17

Human-Machine Conversations Result: a conversational approach founded on use of natural and controlled natural language (CE) allows users at the tactical edge to exploit soft and hard information – including coping with uncertainties and QoI – using mobile computing approaches with low training overhead. Human subject experiment shows rapidly-developed conversational agent able to extract exploitable soft information “There is two policemen are riding on a horse. The horses color are white and brown! They are riding in the same direction.” Longer-Term Goals support a wider range of conversational actions (queries, commands, narratives, model and lexical updates, interjection, etc) integrate information from more sources, e.g. sensor input from the mobile device, social media improve handling of uncertainties, and show utility of this via Experimentation Framework there is a group named '#35' that   has 'policemen' as description and   has the entity concept 'person' as member …. there is a group named '#38' that   has 'The horses' as description and   has the entity concept 'horse' as member and   has the colour 'brown' as colour and   has the colour 'white' as colour. 18 The D2D pipeline can be viewed as a collection of conversational interactions between human & machine agents: data sources, analytic services and decision-makers ITA Peer Review, Sept. 2012 18 18

Academic & Military Impact Technical Leadership to CERDEC in the use of CE for ontology development to linguistic researchers, including DELPH-IN and ARL, in the use of CNL for domain modelling and reasoning, and in the consideration of rationale as a new topic in academic research to SMEs (Prof Don Shemanski and David Alberts) in the use of CE as a means of building a collaborative man machine reasoning system to solve analytic puzzles Alun Preece et al received 2013 ACM MiSeNet MobiCom Workshop Best Paper Award Dave Braines gave an invited plenary speech at NS-CTA on CE for hybrid reasoning Alun Preece, Tien Pham and Dave Braines are on the TPC for NATO IST/SET 126 Symposium Transitions Dstl projects: use of a CNL for intelligence analysis (MIPS) CE for intelligence, cyber assets, (+P5) policy based service composition, (+P5) missions and assets, staff rotation at IBM Hursley to explore a CE agent for analysing Chinese ARL projects: application of CE and experimental framework for NS-CTA, TerraHarvest and ARL anomaly teams demonstration of the conversational capability to CERDEC use of CE to orchestrate high profile NS-CTA experiment, briefed to the NS-CTA board staff rotation of Westpoint cadets at IBM Hursley to build interactive CE database of intelligence profiles Potential: CE as a business language in IBM Information Visualisation product training in CE for modelling, for future transition with ARL analysts in the Agri-Development Teams assistance for training and simulation of emergency situations by linking ACT-R and Unity3D modelling conversational interface in advisory capacity to a UK police force at 2014 NATO Summit

Closing Remarks KEY PUBLICATIONS KEY COLLABORATIONS ENABLED Southampton and CMU have extended the ACT-R architecture to support cognitive social simulation experiments. IBM UK and Southampton have integrated CE components into the ACT-R-based simulation capability. Cambridge and IBM UK are transforming ERG linguistic semantics into CE domain facts Boeing and IBM UK have developed representations of NL uncertainties as CE assumptions Cardiff and IBM UK have developed conversational interfaces, with sensor mission-matching algorithms Cardiff and PSU have integrated measures of uncertainty into allocation algorithms Smart, P. R., Sycara, K., & Tang, Y. (2014) Using Cognitive Architectures to Study Issues in Team Cognition in a Complex Task Environment. Smart, P. R., Richardson, D. P., Sycara, K., & Tang, Y. (2014) Towards a Cognitively Realistic Computational Model of Team Problem Solving Using ACT-R Agents and the ELICIT Experimentation Framework. Mott, D., Poteet, S., Xue, P, Kao, A, Copestake, A. (2014), Natural Language Fact Extraction and Domain Reasoning using Controlled English Xue, P., Poteet, S., Kao, A., Mott, D., & Giammanco, C. (2014) Representing Uncertainty in CE Preece, A., Braines, D., Pizzocaro, D., & Parizas, C. (2014) Human-Machine Conversations to Support Multi-Agency Missions Preece, A., Gwilliams, C., Parizas, C., Pizzocaro, D., Bakdash, J., & Braines, D. (2014) Conversational Sensing 3 long papers, 9 short papers, 6 demonstrations, 1 workshop for the 2014 Fall Meeting Reaching In UCLA, IBM UK and Cardiff have extended conversational interfaces to crowdsourcing (+P6) Cardiff, IBM UK and NS-CTA have integrated CE models and reasoning into demonstrations (+NS-CTA) Aberdeen and IBM UK have linked argumentation and CE assumptions for hypothetical reasoning (+P6) IBM UK, Cardiff and RPI have applied CE to model policy based service composition (+P5) IBM UK and ARL have explored with Fraunhofer (FKIE) on CE and the Battle Management Language Reaching Out Metrics Publications Journal: 3 Conference: 34 5/12 to 5/14 PhD Students On board:4 + 1 post doc + 4 RAs Graduated: 0 8/12 to 8/14 ITA Peer Review, Sept. 2014