1. Navigation and Cognitive Map Formation using Aperiodic Neurodynamics
Derek Harter, Robert Kozma
University of Memphis
July 15, 2004
Simulation of Adaptive Behavior 2004

2. K-Sets Neural Population Model
Model aperiodic dynamics observed in olfactory system.
K-Sets
KA-Sets
discretization of K-Set ODE
speed, analyzability,
Skarda & Freeman (1987). How brains make chaos in order to make sense of the world. Behavioral and Brain Sciences, 10:
Freeman (1991). The physiology of perception. Scientific American, 264(2):78-85.
Harter & Kozma (under revision). Chaotic neurodynamics for autonomous agents. IEEE Transactions on Neural Networks.

3. Modeling Olfactory Dynamics: Perceptual Categorization
(Freeman 1986)

4. Table 1: Characterization of the hierarchy of K-sets
K – Set Hierarchy
Table 1: Characterization of the hierarchy of K-sets
Type
Structure
Inhrt Dynamics
Exs. in brain*
K-0
Single Unit
Nonlinear I/O function
All higher level K sets are composed of K0 units
K-I
Populations of excitatory or inhibitory units
Fixed point convergence to zero or nonzero value
PG, DG, BG, BS
K-II
Interacting populations of excitatory and inhibitory units
Periodic, limit cycle oscillations; frequency in the gamma band
OB, AON, PC, CA1, CA3, CA2, HT, BG, BS, Amygdala
K-III
Several interacting KII and KI sets
Aperiodic, chaotic oscillations
Cortex, Hippocamp, Midline Forebrain
K-IV
Interacting KIII sets
Spatio-temporal dynamics with global phase transitions (itinerancy)
Hemisphere cooperation of cortical, HF and MF by the Amygdala
* Notations: PG – periglomerular; OB - olfactory bulb; AON - anterior olfactory nucleus; PC- prepyriform cortex; HF - hippocampal formation; DG - dentate gyrus; CA1, CA2, CA3 - curnu ammonis sections of the hippocampus; MF - midline forebrain; BG - basal ganglia; HT - hypothalamus; DB - diagonal band; SP – septum
KA-Ie/i
KA-II E1 E2 I1 I2 + - E1 E2 I1 I2
KA-III E1 E2 I1 I2
Layer 1 d
receptors
Layer 2
Layer 3
Named in honor of Katchalsky, a famous and influential neuroscientist)
Maximal slope of transfer function is displaced to the excitatory side of the resting level. An excitatory input to the population raises the activity level of its neurons, and also increases their sensitivity to input from each other, so that they interact more strongly.
The asymmetry of the sigmoid function means that the OB is destabilized by input.

5. Characterization of the Hierarchy of K-sets
Type
Structure
Inherent dynamics
Examples in brain* K0
Single Unit
Nonlinear I/O function
All higher level K sets are composed of K0 units
KIII
Several interacting KII and KI sets
Aperiodic, chaotic oscillations
Cortex, Hippocampal Formation, Midline Forebrain
KIV
Interacting KIII sets
Spatio-temporal dynamics with global phase transitions (itinerancy)
Hemisphere-wide cooperation of cortical, HF and MF areas coordinated by the Amygdala
* Notations: PG – periglomerular; OB - olfactory bulb; AON - anterior olfactory nucleus; PC- prepyriform cortex; HF - hippocampal formation; DG - dentate gyrus; CA1, CA2, CA3 - curnu ammonis sections of the hippocampus; MF - midline forebrain; BG - basal ganglia; HT - hypothalamus; DB - diagonal band; SP - septum
K-Ie
KI Populations of Fixed point convergence connected (excitatory)
excitatory or to zero or nonzero value, periglomerular cells
inhibitory units positive feedback
KII Interacting Periodic, limit cycle connections of excitatory
populations of oscillations, frequency in populations with inhibitory
excitatory and the gamma band, populations, such as in the
inhibitory units negative feedback olfactory bulb. E1 K0
Excitatory + + + + E2 K0
Excitatory

6. Characterization of the Hierarchy of K-sets
Type
Structure
Inherent dynamics
Examples in brain* K0
Single Unit
Nonlinear I/O function
All higher level K sets are composed of K0 units
KIII
Several interacting KII and KI sets
Aperiodic, chaotic oscillations
Cortex, Hippocampal Formation, Midline Forebrain
KIV
Interacting KIII sets
Spatio-temporal dynamics with global phase transitions (itinerancy)
Hemisphere-wide cooperation of cortical, HF and MF areas coordinated by the Amygdala
* Notations: PG – periglomerular; OB - olfactory bulb; AON - anterior olfactory nucleus; PC- prepyriform cortex; HF - hippocampal formation; DG - dentate gyrus; CA1, CA2, CA3 - curnu ammonis sections of the hippocampus; MF - midline forebrain; BG - basal ganglia; HT - hypothalamus; DB - diagonal band; SP - septum
K-II E1 I1
KI Populations of Fixed point convergence connected (excitatory)
excitatory or to zero or nonzero value, periglomerular cells
inhibitory units positive feedback + - - + E2 +
KII Interacting Periodic, limit cycle connections of excitatory
populations of oscillations, frequency in populations with inhibitory
excitatory and the gamma band, populations, such as in the
inhibitory units negative feedback olfactory buld. I2 - - + + -

7. Characterization of the Hierarchy of K-sets
Type
Structure
Inherent dynamics
Examples in brain* K0
Single Unit
Nonlinear I/O function
All higher level K sets are composed of K0 units
KIV
Interacting KIII sets
Spatio-temporal dynamics with global phase transitions (itinerancy)
Hemisphere-wide cooperation of cortical, HF and MF areas coordinated by the Amygdala
* Notations: PG – periglomerular; OB - olfactory bulb; AON - anterior olfactory nucleus; PC- prepyriform cortex; HF - hippocampal formation; DG - dentate gyrus; CA1, CA2, CA3 - curnu ammonis sections of the hippocampus; MF - midline forebrain; BG - basal ganglia; HT - hypothalamus; DB - diagonal band; SP - septum
K-III E1 E2 I1 I2 + -
KI Populations of Fixed point convergence connected (excitatory)
excitatory or to zero or nonzero value, periglomerular cells
inhibitory units positive feedback
KII Interacting Periodic, limit cycle connections of excitatory
populations of oscillations, frequency in populations with inhibitory
excitatory and the gamma band, populations, such as in the
inhibitory units negative feedback olfactory bulb.
KIII Several interacting Aperiodic, chaotic Cortex, Hippocampal
KII and KI sets oscillations Formation, Midline
Forebrain

8. Characterization of the Hierarchy of K-sets
Type
Structure
Inherent dynamics
Examples in brain* K0
Single Unit
Nonlinear I/O function
All higher level K sets are composed of K0 units
KIV
Interacting KIII sets
Spatio-temporal dynamics with global phase transitions (itinerancy)
Hemisphere-wide cooperation of cortical, HF and MF areas coordinated by the Amygdala
* Notations: PG – periglomerular; OB - olfactory bulb; AON - anterior olfactory nucleus; PC- prepyriform cortex; HF - hippocampal formation; DG - dentate gyrus; CA1, CA2, CA3 - curnu ammonis sections of the hippocampus; MF - midline forebrain; BG - basal ganglia; HT - hypothalamus; DB - diagonal band; SP - septum
K-III E1 E2 I1 I2 E1 E2 I1 I2
KI Populations of Fixed point convergence connected (excitatory)
excitatory or to zero or nonzero value, periglomerular cells
inhibitory units positive feedback
KII Interacting Periodic, limit cycle connections of excitatory
populations of oscillations, frequency in populations with inhibitory
excitatory and the gamma band, populations, such as in the
inhibitory units negative feedback olfactory bulb.
KIII Several interacting Aperiodic, chaotic Cortex, Hippocampal
KII and KI sets oscillations Formation, Midline
Forebrain E1 E2 I1 I2

9. Characterization of the Hierarchy of K-sets
Type
Structure
Inherent dynamics
Examples in brain* K0
Single Unit
Nonlinear I/O function
All higher level K sets are composed of K0 units
KIV
Interacting KIII sets
Spatio-temporal dynamics with global phase transitions (itinerancy)
Hemisphere-wide cooperation of cortical, HF and MF areas coordinated by the Amygdala
* Notations: PG – periglomerular; OB - olfactory bulb; AON - anterior olfactory nucleus; PC- prepyriform cortex; HF - hippocampal formation; DG - dentate gyrus; CA1, CA2, CA3 - curnu ammonis sections of the hippocampus; MF - midline forebrain; BG - basal ganglia; HT - hypothalamus; DB - diagonal band; SP - septum
K-III
receptors E1 E2 I1 I2 OB
KI Populations of Fixed point convergence connected (excitatory)
excitatory or to zero or nonzero value, periglomerular cells
inhibitory units positive feedback d
KII Interacting Periodic, limit cycle connections of excitatory
populations of oscillations, frequency in populations with inhibitory
excitatory and the gamma band, populations, such as in the
inhibitory units negative feedback olfactory bulb.
KIII Several interacting Aperiodic, chaotic Cortex, Hippocampal
KII and KI sets oscillations Formation, Midline
Forebrain
AON E1 E2 I1 I2 d E1 E2 I1 I2 PC

10. KA, K-Set and Rat Power Spectra
KA-III model – Layer 1

11. Summary K and KA sets are a neural population model
originally developed to model and understand chaotic dynamics observed in olfactory perceptual system
Aperiodic dynamics are normal background state of olfactory sense
Upon recognition of an order (categorization) falls into a new (lower dimensional) chaotic attractor
Aperiodic dynamics offer advantages in forming perceptual categorization
speed of convergance
flexibility (randomness)
Use aperiodic dynamics to form perceptual categorization and action selection in agents.

12. Hippocampal Simulation and Formation of Place Cells using a KA-III

14. Hebbian Modification CA2 (KA-II) (1) L1 DG (KA-0) L2 CA3 (KA-II) (8x8)ne e se s sw w nw vc c mc m mf f vf d
CA2
(KA-II)
(1) L1 n ne e se s sw w nw vc c mc m mf f vf d DG
(KA-0)
(8x8) L2
CA3
(KA-II)
(8x8)
CA1
(KA-II)
(8x8)
Orientation Beacons n ne e se s sw w nw vc c mc m mf f vf d L3 n ne e se s sw w nw vc c mc m mf f vf d L4

15. Formation of AM Pats in KA-IIIc d b
0.5s of CA1
(Loc 1) (Loc 2) (Loc 3) (Loc 4) (Loc 5) (Loc 6) (Loc 7) (Loc 8)
(Test d) (Test c) (Test b) (Test a)
Narrator: This slide shows some of the results of the simulation. We show contour maps of the activity in the 8x8 CA1 layer. I will explain how the contour maps are generated more fully in a moment. But first, the test was performed by placing the agent at each of the 8 locations after learning in the environment had taken place for some time. We choose 4 test points at random within the detection area of a location (see upper right), thise gives a total of 4x8 or 32 tests in all. We put the agent at the test point for 500ms of (feeding in the input from the landmarks from where the agent is placed). We then captures the activity in the CA1 layer of the KA-III. For example, this (upper 8x8 time series) might represent the ½ second of activity captured in the 8x8 CA1 layer in response to being placed at test point a in location 8. Some of the units are highly active, while others are not so active in response to the input. We reduce each of these 64 time series to a single number that represents the amplitude of the activity for the ½ second of activity. This can be done simply, by for example finding the standard deviation of the signal during the ½ second. This gives us 64 numbers that represent the amplitude of the 8x8 units in the CA1. We can then visualize the activity by plotting contour plots of the amplitudes of activity.
Narrator: Though it is not immediately clear, you will be able to see that the AM patterns of dynamic activity are more similar at a location (e.g. location 1) to each other than they are to other locations. For example, Location 8 shows a good example a strong peak of amplitude at the bottom center, and a valley on the left and right of the spatial CA1. We will next prove more empirically that the KA-III has formed 8 attractors which is somewhat reflected in the contour maps. It should be noted that the contour maps are only pale snapshots of the rich dynamics being performed and captured here in the KA-III. The time series (top right) shows that the underlying activity of the units in the CA1 are aperiodic as we have shown before.

16. Comparison of Closest AM Pattern
Target
Loc-Test
Closest
1-a 1-b
1-b 1-a
1-c 1-b
1-d 1-a
2-a 2-d
2-b 2-c
2-c 2-b
2-d 2-b
3-a 3-b
3-b 3-a
3-c 3-b
3-d 3-a
4-a 4-c
4-b 4-d
4-c 4-a
4-d 4-b
5-a 5-b
5-b 5-a
5-c 5-d
5-d 5-c
6-a 6-b
6-b 6-d
6-c 6-b
6-d 6-b
7-a 7-d
7-b 7-d
7-c 7-d
7-d 7-a
8-a 8-c
8-b 8-d
8-c 8-a
8-d 8-c a c d b b d c a

17. Appetitive/Aversive Behavior using a KA-III
Harter, D. and Kozma, R. (2004).   Aperiodic Dynamics for Appetitive/Aversive Behavior in Autonomous Agents. Accepted in  Proceedings of the 2004 IEEE International Conference on Robotics and Automation (ICRA).

18. 3 Environment Key 1 1 1 Agent Morphology 1 3 2 2 Edible food source
Poisonous
food source
Agent Morphology 1 3
Distance sensor
Light sensor
Front
Wheel &
Motor 2 2

19. 3 Environment Key 1 1 1 Agent Morphology 1 3 2 2 Edible food source
Poisonous
food source
Agent Morphology 1 3
Distance sensor
Light sensor
Front
Wheel &
Motor 2 2

20. Architecture of Appetitive/Aversive Experiment
Smell (Light)
(KA-0 10)
Touch
(KA-I 5)
Distance (IR)
(KA-I 8)
KA-III OB
(8x8 KA-II)
AON
(8x8 KA-II) V PC
(8x8 KA-II)
Mapp
Mave
Msearch
Valence
Hebbian Modification
Tasteapp
Tasteave

21. Agent Path No Learn (left) and Learn (Right)
Environment Key 1
Edible food
source 1
Poisonous
food source 1 3 1 3 1 3 1 3 2 2 2 2

22. Experiments Summary
Aperiodic dynamics can be shaped to form location categorization in a “Hippocampus-like” architecture.
Consistent with so called “place cell” phenomena of hippocampus.
Aperiodic dynamics can be used to build plastic control systems that learn categories and associate them with behavioral responses.
How aperiodic dynamics in hippocampal formation of remembered locations might be used to do goal-directed navigation still an open question.

23. Evolution of Intentional & Deliberative Behavior
3.5 billion years: single-cell entities
550 million years: fish & vertebrates
430 million years: insects
370 million years: reptiles
330 million years: dinosaurs
250 million years: mammals
120 million years: primates
18 million years: great apes
2.5 million years: man
5000 years: writing
Basic Limbic System
Primitive Hippocampus
Long-term memory
Beyond stimulus/response
Episodic Memory
Cognitive Maps
Narrator: If we look at the evolution of capabilities in biological organism, we see that it took a very long time, billions of years, to progress from the tropic behaviors of single-celled organisms up to reptiles. Progress after that is relatively much faster, taking only about 400 million years to progress from the reptiles to modern man.
Narrator: The really hard part for nature was to get the level where creatures could move around, have sensory abilities, orient in environment e.g. intentionality
Narrator: One of the basic premises of the situated and embodied approach is that if we do not understand this sensory-motor basis of behavior, we have no chance of ever understanding intelligence.
Narrator: The main innovation that appears to have occurred 400 million years ago, was the development of basic long-term and episodic memory structures to the basic vertebrate limbic system (Click) . The evolution of a primitive hippocampus takes place about this time, which is believe today to be the main structure for laying down these types of long term memorys. It is known to play functions in long-term and episodic memory. For example, cognitive maps appear to be formed primarly by hippocampal functions. Cognitive maps are a type of idealized or abstracted episodic memory, where we learn the gross features of our environment and the spatial relations among important locations.

24. Limbic System: Simplest Complete Intentional System
Far from equilibrium, therodynamic systems
Mechanisms of self-organization
competition
cooperation
autocatalytic loops
hierarchy & mesh
Aperiodic dynamics
Expectation or reafference
Embodiment
environment/organism coupling
Small worlds type divergent-convergent systems connections
Narrator: This is a model of the basic limbic system that we have proposed and are developing for NASA. Although we don’t have time to go into it in detail I wanted to point out 3 things. 1) our limbic system model consists of 4 areas Perceptual, Hippocampal (for long-term memory) Midline-forebrain (involved in goals by monitoring internal factors) and motor systems. 2) Unlike traditional sense-think-act cycle models, the biological limbic system is highly recurrant. Our model captures the major connectivity between the limbic system areas. 3) aperiodic dynamics (e.g. those that are not simple point or limit cycle attractors) dominate the dynamics of observed brain activity. Our model seeks to understand and replicate some of the imporant features of aperiodic dynamics in the production of cognition.
Narrator: (Click) click through the points, say a little about them.
(Harter & Kozma 2004)

25. Talk Conclusions Aperiodic dynamics are the norm in biological brains.
Result of intrinsic population effects (as well as external noisy stimulation).
May be useful properties for perception.
Such dynamics are being explored to determine usefulness in biological and artificial systems.