1. Human-Information Interaction
TA6 – Project 4
David Mott IBM UK

2. Project Overview
COALITION NEEDS
Project 4 aims to advance our understanding of the factors that affect shared understanding and information exploitation in military coalition environments
to be assisted in collaborative problem-solving by systems that share common understanding of the knowledge and reasoning with human users
to engineer the coalition socio-technical environment in ways that support collective/team cognition
to exploit large bodies of diverse, unstructured information content and other information assets in the execution of their tasks
PROJECT FOCUS
understanding the interaction between cognitive, social and technological factors in collaborative problem-solving contexts, by cognitively-rich agent simulation
supporting fact extraction from plain text by using deep linguistic processing and facilitating human-machine cognitive capabilities by externalising reasoning in a Controlled Natural Language
understanding how to support human decision-makers by matching information assets to requirements and by use of meaning-rich conversational interactions
ITA Peer Review, Sept. 2014

3. Multidisciplinary Research Team
IBM UK
David Braines
David Mott
Paul Stone
Carnegie Mellon University
Katia Sycara
Yuqing Tang
PSU
Tom La Porta
Nan Hu
Airbus Group
Gavin Powell
Matthew Roberts
University of Cambridge
Ann Copestake
Diarmuid O Seaghdha
Boeing
Anne Kao
Stephen Poteet
Ping Xue
University of Cardiff
Alun Preece
Will Webberley
IBM US
Geeth de Mel
University of Southampton
Paul Smart
Darren Richardson
Dstl
Kris Challa
UCLA
Mani Srivastava
Robin Wentau Ouyang
Matthew Johnson
ARL
Cheryl Giammanco
Jon Bakdash

4. Key Technical Accomplishments
Task 1: Collective Cognition in Coalition Environments
a cognitive social simulation capability to support experimental studies of team cognition in complex task environments.
an initial cognitive computational model of collaborative problem solving in a specific task context (i.e., the ELICIT task).
a human experiment using the ELICIT task, designed to gather data relating to the effect of different information sharing strategies on task performance.
integration of Controlled English (CE) components into the simulation capability to support linguistic forms of agent communication.
ITA Peer Review, Sept. 2014

5. Key Technical Accomplishments
Task 2: Fact Extraction and Reasoning using Controlled Natural language
extraction of CE facts from natural language (NL) sentences, applying knowledge expressed in CE to transform linguistic semantics into domain semantics.
extension of CE for more complex problem solving strategies with meta-reasoning, assumptions, rationale, as applied to realistic analytic tasks
handling of NL ambiguities and uncertainties, extending linguistic mechanisms to allow the CE domain model to guide NL processing
exploration of collaborative man-machine problem-solving systems as applied to “logic puzzles” used in training intel analysts.
ITA Peer Review, Sept. 2014 5 5

6. Key Technical Accomplishments
Task 3: Coalition Context-Aware Assistance for Decision Makers
the definition and validation, including initial trials with human subjects, of a conversational protocol for free-flowing human-machine, machine-machine, and machine-human exchanges
a model of resource allocation in network systems as a "stochastic knapsack problem" to handle uncertain factors like unreliable wireless medium or variable quality of sensor outputs
a technology integration experiment to show how key research elements can be combined to support rapid but informed decision-making capabilities at lower echelons in coalition operations
ITA Peer Review, Sept. 2014 6 6

7. ACT-R Cognitive Social Simulation Capability (CSSC)
Insight: Cognitive architectures can be used to model socially- distributed cognitive processes and improve our understanding of the dynamics of team cognition.
State-of-the-art
Multi-agent simulation techniques have been used to model collective behavior.
….however, such models do not take account of the cognitive capabilities and limitations of human agents.
Cognitive architectures provide computational frameworks for the development of cognitively-plausible agents.
…however, few applications to team contexts.
ACT-R:
a mature cognitive architecture that can be used to model a broad range of cognitive processes.
extend existing single-agent ACT-R research by focusing on interactions of multiple agents, allowing study of socially-distributed cognition
extend ACT-R architecture to support multiple-agent research
Cognitive social simulation:
integration of cognitive architectures into social simulation results in cognitive social simulation.
individual machine agents share the same kind of cognitive capabilities and limitations as their human counterparts.
Experimentation:
based on collaborative simulation of the DoD CCRP ELICIT identification task
simulation capability instrumented to collect and measure performance factors based upon existing ELICIT metrics
Coalition needs/benefits
We need to know how to exploit the collective cognitive potential of coalition organizations.
We need to assess the effect of human factors and socio-technical interventions without disrupting the operational environment.
Paul Smart (Southampton)
CMU, IBM UK, Airbus Group

8. ACT-R Cognitive Social Simulation Capability (CSSC)
Goal Module
Declarative Module
Imaginal Module
Vocal Module
Procedural Module
Matching
Selection
Execution
ACT-R/CSSC
Self Module
Messaging Module
Language Module
Web Module
Agent
Characteristics
Messaging
Website
Language Processor
The ACT-R/CSSC is an extension to the core ACT-R architecture to support cognitive social simulation experiments.

9. ACT-R Cognitive Social Simulation Capability (CSSC)

10. ACT-R Cognitive Social Simulation Capability
Result: Extended ACT-R cognitive architecture to support studies into team cognition. Custom modules support linguistic communication (language module), interaction with shared repositories (web module), and configuration of agent behaviour (self module). The simulation capability also supports the real-time participation of human subjects.
Simulation capability supports studies into team-based problem solving.
Current experimental efforts focused on the DoD CCRP ELICIT task.
Modelling the performance of human subjects in different organizational environments.
Coordinator
Long-Term Goals
Team Leader
Understand effect of task environment features on collective cognitive performance:
Team Member
perform human experiments, and gather data, with a variant of the ELICIT task.
extend cognitive computational models with machine agents embodied in the environment
evaluate cognitive models with experimental simulation studies.
Shared Repository (limited access)
The ACT-R Cognitive Social Simulation capability provides a generic platform for exploring the interaction between cognitive, social and technological factors in collective cognition.
ITA Peer Review, Sept. 2012

11. CE to support fact extraction and problem solving
State-of-the-art
CE can integrate modelling and reasoning capabilities to assist performance of complex cognitive tasks, including transformation of linguistic information from a deep parsing system into high value domain facts
Traditional NL mechanisms have difficulty extracting detailed information about complex situations
Deep parsing systems, can extract more subtle and detailed information from text, but it is a complex task to convert the linguistic output into domain-specific facts
Performance of complex cognitive tasks requires integration of different modeling and reasoning capabilities in a common language
Use of DELPH-IN linguistic resources
The ERG, a deep and detailed grammar of English
Minimal Recursion Semantics (MRS) to represent the output of the parse in a logical form
Transforming linguistic semantics to domain semantics
Rules expressed in CE for application of CE domain models
Exploration of linguistic theories, in a way that is more natural than typical formal languages
CE reasoning and modelling system
User-defined domain model for expressing concepts, facts, logical rules and problem-solving strategies
Assumptions to capture uncertainty, ambiguity and sentence interpretation
Rationale and proof tables to visualise reasoning and sources of uncertainty
Meta-model for reasoning about the language itself, e.g. mapping words to concepts and to create new rules from sentences
Coalition needs/benefits
Military tasks require text from NL sources to analyse, fuse and infer high value information
A deep parsing system, with linguistic to domain semantic transformations, could extract more domain specific facts to support these tasks
Collaborating human agents need support in problem-solving tasks in a way that is understandable and externalises their domain models and problem-solving strategies.
David Mott (IBM UK) 11
Boeing, Cambridge, ARL
ITA Peer Review, Sept. 2014 11

12. Linguistic reasoning to extract CE facts12

13. Reasoning to support problem solving13

14. CE to support fact extraction and problem solving
CE linguistic reasoning can apply domain knowledge to fact extraction, bridging the gap between deep linguistic analysis and domain semantics; CE domain reasoning can externalise human problem solving to solve an analytic task in an explainable way.
CE domain models, reasoning and problem solving strategies
Long-Term Goals
Improve linguistic processing and fact extraction
linguistic reasoning to handle a wider range of sentences, e.g. following the MRS test suite
construction of domain models from distributional semantics
more domain knowledge to resolve ambiguities of sentence interpretation
A machine agent that can participate in collective cognition and externalises human reasoning
Improve problem solving capability
Extend CE expressivity
More sophisticated handling of assumptions
Improve visualisation of rationale
Handle more complex logical problem solving
CE problem solving techniques can be applied to both fact extraction from NL sentences and reasoning from basic extracted facts to intel relevant high value information
ITA Peer Review, Sept. 2012 14 14

15. Human-Machine Conversations
State-of-the-art
Insight: Human-machine conversations using natural language (NL) and Controlled English (CE) can facilitate information exploitation and asset tasking at the tactical edge.
Reducing cycle time in rapid establishment of data-to-decision pipelines (D2D) still requires significant technical skills and training
Handling of uncertainties in exploiting soft and hard information is a key problem, even for technical users
A formal conversational protocol – captured as a CE model – enables HMC in the scope of a range of tactical-level use cases:
“spot” reports (including uncertain QoI)
information fusion (including rationale)
asset tasking (including uncertain availability)
Combining the expressivity of NL and the precision and reasoning capabilities of CE
Coalition needs/benefits
Low training overhead and flexible human-machine collaboration (HMC) to create D2D pipelines
Exchange of information precisely and unambiguously whilst retaining human readability and flexibility, pinpointing sources of uncertainty and providing explanation
Alun Preece (Cardiff)
ITA Peer Review, Sept. 2014 15
IBM UK, IBM US, PSU, UCLA, ARL 15

16. Human-Machine Conversations
NL/CE Conversational Protocol
Aim: to enable conversational interactions that flow freely between natural language and CE, specifically to support dynamic establishment of D2D pipelines
Conversational protocol draws on research in agent communication languages and philosophical linguistics (speech acts)
Types of interaction include:
Confirmatory dialogues to reduce ambiguity (“did you mean…?”)
Query-response, with optional rationale (“Why?”)
Machine-generated (text/graphical) “gist” forms to summarise complex CE
ITA Peer Review, Sept. 2014 16

17. Human-Machine Conversations
Main aims:
to determine the degree to which a software agent can extract CE from unrestricted NL conversations
to test the agent’s robustness with untrained users
Experimentation: scenario-based
Experimentation: human subjects
20 subjects viewed a series of scenes and described them in natural language via a text-based interface
The system provided feedback in CE and a score (terms recognised by system)
D2D vignette from earlier ITA work
Technology integration experiment including: conversational agent (steps 1+4), CE Store (2), SAM (3)
Demo with CERDEC AI TECD Jan 2014
Takes account of uncertainty of asset availability (e.g. bandwidth) and QoI (e.g. due to weather)
ITA Peer Review, Sept. 2014 17

18. Human-Machine Conversations
Result: a conversational approach founded on use of natural and controlled natural language (CE) allows users at the tactical edge to exploit soft and hard information – including coping with uncertainties and QoI – using mobile computing approaches with low training overhead.
Human subject experiment shows rapidly-developed conversational agent able to extract exploitable soft information
“There is two policemen are riding on a horse. The horses color are white and brown! They are riding in the same direction.”
Longer-Term Goals
support a wider range of conversational actions (queries, commands, narratives, model and lexical updates, interjection, etc)
integrate information from more sources, e.g. sensor input from the mobile device, social media
improve handling of uncertainties, and show utility of this via Experimentation Framework
there is a group named '#35' that
  has 'policemen' as description and
  has the entity concept 'person' as member ….
there is a group named '#38' that
  has 'The horses' as description and
  has the entity concept 'horse' as member and
  has the colour 'brown' as colour and
  has the colour 'white' as colour. 18
The D2D pipeline can be viewed as a collection of conversational interactions between human & machine agents: data sources, analytic services and decision-makers
ITA Peer Review, Sept. 2012 18 18

19. Academic & Military Impact
Technical Leadership
to CERDEC in the use of CE for ontology development
to linguistic researchers, including DELPH-IN and ARL, in the use of CNL for domain modelling and reasoning, and in the consideration of rationale as a new topic in academic research
to SMEs (Prof Don Shemanski and David Alberts) in the use of CE as a means of building a collaborative man machine reasoning system to solve analytic puzzles
Alun Preece et al received 2013 ACM MiSeNet MobiCom Workshop Best Paper Award
Dave Braines gave an invited plenary speech at NS-CTA on CE for hybrid reasoning
Alun Preece, Tien Pham and Dave Braines are on the TPC for NATO IST/SET 126 Symposium
Transitions
Dstl projects:
use of a CNL for intelligence analysis (MIPS)
CE for intelligence, cyber assets, (+P5) policy based service composition, (+P5) missions and assets,
staff rotation at IBM Hursley to explore a CE agent for analysing Chinese
ARL projects:
application of CE and experimental framework for NS-CTA, TerraHarvest and ARL anomaly teams
demonstration of the conversational capability to CERDEC
use of CE to orchestrate high profile NS-CTA experiment, briefed to the NS-CTA board
staff rotation of Westpoint cadets at IBM Hursley to build interactive CE database of intelligence profiles
Potential:
CE as a business language in IBM Information Visualisation product
training in CE for modelling, for future transition with ARL analysts in the Agri-Development Teams
assistance for training and simulation of emergency situations by linking ACT-R and Unity3D modelling
conversational interface in advisory capacity to a UK police force at 2014 NATO Summit

20. Closing Remarks KEY PUBLICATIONS KEY COLLABORATIONS ENABLED
Southampton and CMU have extended the ACT-R architecture to support cognitive social simulation experiments.
IBM UK and Southampton have integrated CE components into the ACT-R-based simulation capability.
Cambridge and IBM UK are transforming ERG linguistic semantics into CE domain facts
Boeing and IBM UK have developed representations of NL uncertainties as CE assumptions
Cardiff and IBM UK have developed conversational interfaces, with sensor mission-matching algorithms
Cardiff and PSU have integrated measures of uncertainty into allocation algorithms
Smart, P. R., Sycara, K., & Tang, Y. (2014) Using Cognitive Architectures to Study Issues in Team Cognition in a Complex Task Environment.
Smart, P. R., Richardson, D. P., Sycara, K., & Tang, Y. (2014) Towards a Cognitively Realistic Computational Model of Team Problem Solving Using ACT-R Agents and the ELICIT Experimentation Framework.
Mott, D., Poteet, S., Xue, P, Kao, A, Copestake, A. (2014), Natural Language Fact Extraction and Domain Reasoning using Controlled English
Xue, P., Poteet, S., Kao, A., Mott, D., & Giammanco, C. (2014) Representing Uncertainty in CE
Preece, A., Braines, D., Pizzocaro, D., & Parizas, C. (2014) Human-Machine Conversations to Support Multi-Agency Missions
Preece, A., Gwilliams, C., Parizas, C., Pizzocaro, D., Bakdash, J., & Braines, D. (2014) Conversational Sensing
3 long papers, 9 short papers, 6 demonstrations, 1 workshop for the 2014 Fall Meeting
Reaching In
UCLA, IBM UK and Cardiff have extended conversational interfaces to crowdsourcing (+P6)
Cardiff, IBM UK and NS-CTA have integrated CE models and reasoning into demonstrations (+NS-CTA)
Aberdeen and IBM UK have linked argumentation and CE assumptions for hypothetical reasoning (+P6)
IBM UK, Cardiff and RPI have applied CE to model policy based service composition (+P5)
IBM UK and ARL have explored with Fraunhofer (FKIE) on CE and the Battle Management Language
Reaching Out
Metrics
Publications
Journal: 3
Conference: 34
5/12 to 5/14
PhD Students
On board:4
+ 1 post doc + 4 RAs
Graduated: 0
8/12 to 8/14
ITA Peer Review, Sept. 2014