1. Diagrammatic Reasoning in Army Situation Understanding and Planning: Architecture for Decision Support and Cognitive Modeling
B. Chandrasekaran, Bonny Banerjee, Unmesh Kurup, John Josephson
The Ohio State University
Robert Winkler
US ARL

2. Outline of the Talk
What is Diagrammatic Reasoning? Why is it important in & for Army Decision-Making?
Basic research issues & brief outline of progress:
Representation, Architecture
Technology built on Science & Applications built on technology
Some Remarks on the Future

3. Ubiquity of Diagrams in Army Operations
The Army is about space:
taking it, defending it, controlling it, avoiding it, going through it
Army planning, situation assessment, situation monitoring, fusion, all use diagrammatic representations
Standards for symbols to be used are defined in FMs

4. How DR Research Can Help Provide More Effective Decision Support
Automating repetitive & routine reasoning tasks that involve diagrams
E.g. Critiquing COA’s for vulnerability to ambush
Better interface design. Understanding what makes a diagram good, i.e., makes relevant information readily available, without error, can help in design of decision interfaces
Requires how human cognitive architecture & perception work in performing diagrammatic reasoning
Cognitive modeling to evaluate diagrammatic interfaces

5. What Diagrammatic Reasoning is...
DR is reasoning, i.e. making inferences and problem solving with visual representations, and involves a collaboration between two systems:
A symbolic reasoning system that combines information from the diagram with other information to make inferences, to set up diagrammatic perception and action subgoals
A diagrammatic representation from which perception obtains information about spatial relations and properties re. diagrammatic objects .
The phenomena of interest can also take place in our imagination

6. Diagrammatic Reasoning is Not..
What it is not
It is not image processing (such as processing satellite images for objects of interest, though DR can be used as part of it)
Image processing may be required to extract the diagram from an image
It is not computes graphics, though diagrams can often be usefully superimposed on such pictures
It is not parallel array processing algorithms that solve problems such as shortest paths, though there is a role for such algorithms in the overall process of diagrammatic reasoning,

7. Some Scientific Issues we Have Made Progress In
What is a diagram as a representation
Specificity of a diagram. How are we able to solve a general problem from a specific diagram?
Representation in the mind & in the computer.
The nature of the architecture that can perform diagrammatic reasoning
Opportunistic integration of diagrammatic & inferential operations
How do diagrams get into long-term memory?
How are diagrams composed to make new diagrams?
Abstraction of diagrams

8. Computational Model of Diagrammatic Reasoning
Recall two reasons we mentioned, to develop computational frameworks for diagrammatic reasoning:
Automation or semi-automation.
Building cognitive models
Good News!
A computational architecture that can be used for automation can also be used for modeling.
Our bimodal cognitive architecture: BiSoar

9. Symbolic Inference & Perception from Diagrams
Any system that can support symbolic representation & inference can be integrates with our DRS.
Reasoning/ Problem Solving System
Perception/ Action Routines
Diagram Representation in DRS
Soar & Act-R happen to be symbolic reasoning systems with especially useful properties for general intelligence.

10. BiSoar: a Bimodal Cognitive Architecture
“Thinking” has been usually modeled in AI & Cognitive Science as syntactic operations on abstract symbols. Soar, Act-R, etc.
BiSoar keeps the general architecture, but all states can be bi-modal; The agent can have both linguistic & pictorial representations’

11. Diagrammatic Representation System
Diagrams consist of three types of objects – Points, Curves & Regions.
Diagrams are not just images, they are a spatial configuration of spatial objects.

12. Emergent Objects & Relations
The Role of Perception
Perception and Action Routines: A set of algorithms that create or modify diagrams and “perceive” objects and spatial relations between elements in the diagram.
Reasoning/ Problem Solving System
Emergent Objects & Relations
Perception/ Action Routines
Perception/ Action Routines
Diagram Representation in DRS

13. Perceptual Routines Recognize Emergent Objects and Relations
ii. E C D
iii. I G J H
Base set domain-independent, open-ended
New object recognition and extraction routines:
Intersection-points between line objects, region when a line closes on itself, new regions when regions intersect, new regions when a line intersects with a region, extracting distinguished points on a line (such as end points) or in a region, extracting distinguished segments of a line (such as those created when two lines intersect), extracting periphery of a region as a closed line. Reverse operations are included – such as when a line is removed, certain region objects will no longer exist and need to be removed.
Relational perception routines: Inside (I1,I2), Outside, Left-of, Right-of, Top-of, Below, Segment-of (Line1, Line2), Subregion-of (Region1, Region2), On (Point, Line), and Closed (Line1).
Translation, rotation and scanning routines may be combined with routines in 1 and 2. Example, Intersect (Line, Rotate (90deg, Line 2)).

14. Action Routines
Create diagrammatic objects, such as a path that goes from point1 to point2 while avoiding region2. Path finding and path modification routines are especially useful in Army applications.

15. Automatic Synthesis of PR’s & AR’s
Banerjee’s Ph. D Thesis gives many techniques for automatic synthesis of PR’s & AR’s. Example: In the situation below, where c is a wall, A is a member of Red force, where can BT a member of Blue force hide?
Once the problem is converted to the language of geometry, the set of all points p such that line Ap intersects c, his techniques can automatically construct algorithms to solve the problem.

16. Attention, Learning, & Memory
BiSoar can be parametrized to mimic the limitations of human attention & short term memory.
BiSoar can learn by a mechanism called “chunking.” As a result of attention & short term memory limitations, BiSoar’s LTM contains smoothed approximations of complex shapes.

17. Example of Automating DR
Entity ReIdentification in ASAS “All Source Analysis System”
Currently very human-analyst labor intensive, and many sightings are simply left unattended

18. Diagrammatic Reasoning in Information Fusion in an ASAS Problem
The task is to decide for a newly sighted entity, T3, which of the previous sighted & identified entities it is. T3
Regions impassable for vehicle types of interest are marked and represented diagrammatically in the computer

19. Entities from Past Sightings Retrieved
Two tanks, T1 and T2 were retrieved along with their locations and times of sighting
The Fusion Engine asks for ways in which T1 & T2 could have gotten to the location of T3 within the available time

20. Architecture Combines Symbolic & Diagrammatic Reasoning
For T1 &T2, DR finds eight possible routes, but rules out all but one. The figures shows the routes for T1 & T2.

21. Example of Action Routine
The Database reveals that there are sensor fields but they didn’t report any vehicle crossings.
A similar question about T2 reveals that T2 also crossed a sensor field, which also didn’t report any vehicles. However, DR says T2 could not have avoided the sensor field.

22. Numerous Other Applications
Rerouting
Ambush vulnerability analysis
Plan critiquing in general
Other uses in information fusion, where the hypothesis has significant spatial components

23. Examples of Cognitive Modeling
Kurup’s thesis models & explores:
How errors in geographical recall come about.
Recalled spatial relationships between geographical entities show distortions
Ex: What is the relationship between San-Diego and Reno?

24. Three Models Model 1: Agent has complete map
Model 2: Agent has symbolic knowledge that SD is South of SF and Reno is East of SF.
Model 3: Agent has knowledge that SD in California, Reno in Nevada and that California is West of Nevada.
San-Diego
Reno C N
:(a) Map of SW U.S. in LTM (& WM) of Model 1. (b) & (c) are diagrams in WM constructed by Models 2 & 3
(a)
(b)
(c)

25. Models or Route Recall & Graph Comprehension
Loss of detail in recall of routes
Kurup’s Model posits attention limits as explanation
Graph Comprehension
Lele’s BiSoar model unifies a variety of observed phenomena
Using external graphs requires mental imagistic operations!

26. DR Automation & Modeling Central to Decision Support
The research reported here has laid come scientific & technological foundations of this area.
Has also built some demonstration applications & models.
But it’s still a baby, there’s potential, but needs to be nurtured to produce full benefit.
Many important research issues:
Extraction of DRS from physical diagrams
How are appropriate diagrams to help solve problems generated?