Diagrammatic Reasoning in Army Situation Understanding and Planning: Architecture for Decision Support and Cognitive Modeling B. Chandrasekaran, Bonny Banerjee, Unmesh Kurup, John Josephson The Ohio State University Robert Winkler US ARL

Outline of the Talk What is Diagrammatic Reasoning? Why is it important in & for Army Decision-Making? Basic research issues & brief outline of progress: Representation, Architecture Technology built on Science & Applications built on technology Some Remarks on the Future

Ubiquity of Diagrams in Army Operations The Army is about space: taking it, defending it, controlling it, avoiding it, going through it Army planning, situation assessment, situation monitoring, fusion, all use diagrammatic representations Standards for symbols to be used are defined in FMs

How DR Research Can Help Provide More Effective Decision Support Automating repetitive & routine reasoning tasks that involve diagrams E.g. Critiquing COA’s for vulnerability to ambush Better interface design. Understanding what makes a diagram good, i.e., makes relevant information readily available, without error, can help in design of decision interfaces Requires how human cognitive architecture & perception work in performing diagrammatic reasoning Cognitive modeling to evaluate diagrammatic interfaces

What Diagrammatic Reasoning is... DR is reasoning, i.e. making inferences and problem solving with visual representations, and involves a collaboration between two systems: A symbolic reasoning system that combines information from the diagram with other information to make inferences, to set up diagrammatic perception and action subgoals A diagrammatic representation from which perception obtains information about spatial relations and properties re. diagrammatic objects . The phenomena of interest can also take place in our imagination

Diagrammatic Reasoning is Not.. What it is not It is not image processing (such as processing satellite images for objects of interest, though DR can be used as part of it) Image processing may be required to extract the diagram from an image It is not computes graphics, though diagrams can often be usefully superimposed on such pictures It is not parallel array processing algorithms that solve problems such as shortest paths, though there is a role for such algorithms in the overall process of diagrammatic reasoning,

Some Scientific Issues we Have Made Progress In What is a diagram as a representation Specificity of a diagram. How are we able to solve a general problem from a specific diagram? Representation in the mind & in the computer. The nature of the architecture that can perform diagrammatic reasoning Opportunistic integration of diagrammatic & inferential operations How do diagrams get into long-term memory? How are diagrams composed to make new diagrams? Abstraction of diagrams

Computational Model of Diagrammatic Reasoning Recall two reasons we mentioned, to develop computational frameworks for diagrammatic reasoning: Automation or semi-automation. Building cognitive models Good News! A computational architecture that can be used for automation can also be used for modeling. Our bimodal cognitive architecture: BiSoar

Symbolic Inference & Perception from Diagrams Any system that can support symbolic representation & inference can be integrates with our DRS. Reasoning/ Problem Solving System Perception/ Action Routines Diagram Representation in DRS Soar & Act-R happen to be symbolic reasoning systems with especially useful properties for general intelligence.

BiSoar: a Bimodal Cognitive Architecture “Thinking” has been usually modeled in AI & Cognitive Science as syntactic operations on abstract symbols. Soar, Act-R, etc. BiSoar keeps the general architecture, but all states can be bi-modal; The agent can have both linguistic & pictorial representations’

Diagrammatic Representation System Diagrams consist of three types of objects – Points, Curves & Regions. Diagrams are not just images, they are a spatial configuration of spatial objects.

Emergent Objects & Relations The Role of Perception Perception and Action Routines: A set of algorithms that create or modify diagrams and “perceive” objects and spatial relations between elements in the diagram. Reasoning/ Problem Solving System Emergent Objects & Relations Perception/ Action Routines Perception/ Action Routines Diagram Representation in DRS

Perceptual Routines Recognize Emergent Objects and Relations ii. E C D iii. I G J H Base set domain-independent, open-ended New object recognition and extraction routines: Intersection-points between line objects, region when a line closes on itself, new regions when regions intersect, new regions when a line intersects with a region, extracting distinguished points on a line (such as end points) or in a region, extracting distinguished segments of a line (such as those created when two lines intersect), extracting periphery of a region as a closed line. Reverse operations are included – such as when a line is removed, certain region objects will no longer exist and need to be removed. Relational perception routines: Inside (I1,I2), Outside, Left-of, Right-of, Top-of, Below, Segment-of (Line1, Line2), Subregion-of (Region1, Region2), On (Point, Line), and Closed (Line1). Translation, rotation and scanning routines may be combined with routines in 1 and 2. Example, Intersect (Line, Rotate (90deg, Line 2)).

Action Routines Create diagrammatic objects, such as a path that goes from point1 to point2 while avoiding region2. Path finding and path modification routines are especially useful in Army applications.

Automatic Synthesis of PR’s & AR’s Banerjee’s Ph. D Thesis gives many techniques for automatic synthesis of PR’s & AR’s. Example: In the situation below, where c is a wall, A is a member of Red force, where can BT a member of Blue force hide? Once the problem is converted to the language of geometry, the set of all points p such that line Ap intersects c, his techniques can automatically construct algorithms to solve the problem.

Attention, Learning, & Memory BiSoar can be parametrized to mimic the limitations of human attention & short term memory. BiSoar can learn by a mechanism called “chunking.” As a result of attention & short term memory limitations, BiSoar’s LTM contains smoothed approximations of complex shapes.

Example of Automating DR Entity ReIdentification in ASAS “All Source Analysis System” Currently very human-analyst labor intensive, and many sightings are simply left unattended

Diagrammatic Reasoning in Information Fusion in an ASAS Problem The task is to decide for a newly sighted entity, T3, which of the previous sighted & identified entities it is. T3 Regions impassable for vehicle types of interest are marked and represented diagrammatically in the computer

Entities from Past Sightings Retrieved Two tanks, T1 and T2 were retrieved along with their locations and times of sighting The Fusion Engine asks for ways in which T1 & T2 could have gotten to the location of T3 within the available time

Architecture Combines Symbolic & Diagrammatic Reasoning For T1 &T2, DR finds eight possible routes, but rules out all but one. The figures shows the routes for T1 & T2.

Example of Action Routine The Database reveals that there are sensor fields but they didn’t report any vehicle crossings. A similar question about T2 reveals that T2 also crossed a sensor field, which also didn’t report any vehicles. However, DR says T2 could not have avoided the sensor field.

Numerous Other Applications Rerouting Ambush vulnerability analysis Plan critiquing in general Other uses in information fusion, where the hypothesis has significant spatial components

Examples of Cognitive Modeling Kurup’s thesis models & explores: How errors in geographical recall come about. Recalled spatial relationships between geographical entities show distortions Ex: What is the relationship between San-Diego and Reno?

Three Models Model 1: Agent has complete map Model 2: Agent has symbolic knowledge that SD is South of SF and Reno is East of SF. Model 3: Agent has knowledge that SD in California, Reno in Nevada and that California is West of Nevada. San-Diego Reno C N :(a) Map of SW U.S. in LTM (& WM) of Model 1. (b) & (c) are diagrams in WM constructed by Models 2 & 3 (a) (b) (c)

Models or Route Recall & Graph Comprehension Loss of detail in recall of routes Kurup’s Model posits attention limits as explanation Graph Comprehension Lele’s BiSoar model unifies a variety of observed phenomena Using external graphs requires mental imagistic operations!

DR Automation & Modeling Central to Decision Support The research reported here has laid come scientific & technological foundations of this area. Has also built some demonstration applications & models. But it’s still a baby, there’s potential, but needs to be nurtured to produce full benefit. Many important research issues: Extraction of DRS from physical diagrams How are appropriate diagrams to help solve problems generated?