Aperiodic Dynamics for Appetitive/Aversive Behavior in Autonomous Agents Derek Harter and Robert Kozma University of Memphis Computational Neurodynamics Lab ICRA’04 April 26, 2004

Discrete difference eq. ANN Discrete KA K Discrete difference eq. Continuous ODE Spiking Population Feed-Forward Highly Recurrent Back-Propogation Hebbian

Table 1: Characterization of the hierarchy of K-sets KA – Set Hierarchy KA-0 Unit Difference Equation: Table 1: Characterization of the hierarchy of K-sets Type Structure Inhrt Dynamics Exs. in brain* KA-0 Single Unit Nonlinear I/O function All higher level K sets are composed of K0 units KA-I Populations of excitatory or inhibitory units Fixed point convergence to zero or nonzero value PG, DG, BG, BS KA-II Interacting populations of excitatory and inhibitory units Periodic, limit cycle oscillations; frequency in the gamma band OB, AON, PC, CA1, CA3, CA2, HT, BG, BS, Amygdala KA-III Several interacting KII and KI sets Aperiodic, chaotic oscillations Cortex, Hippocamp, Midline Forebrain KA-IV Interacting KIII sets Spatio-temporal dynamics with global phase transitions (itinerancy) Hemisphere cooperation of cortical, HF and MF by the Amygdala * Notations: PG – periglomerular; OB - olfactory bulb; AON - anterior olfactory nucleus; PC- prepyriform cortex; HF - hippocampal formation; DG - dentate gyrus; CA1, CA2, CA3 - curnu ammonis sections of the hippocampus; MF - midline forebrain; BG - basal ganglia; HT - hypothalamus; DB - diagonal band; SP - septum Transfer function: KA-III E1 E2 I1 I2 Layer 1 d receptors Layer 2 Layer 3 KA-Ie KA-Ii Named in honor of Katchalsky, a famous and influential neuroscientist) Maximal slope of transfer function is displaced to the excitatory side of the resting level. An excitatory input to the population raises the activity level of its neurons, and also increases their sensitivity to input from each other, so that they interact more strongly. The asymmetry of the sigmoid function means that the OB is destabilized by input. E1 E2 I1 I2 KA-II E1 E2 I1 I2 + -

KA-III =0.058 =0.-044 =0.0109 =0.152 E1 E2 I1 I2 Layer 1 d receptors Layer 2 Layer 3

KA, K-Set and Rat Power Spectra KA-III model – Layer 1

Appetitive/Aversive Behavior using a KA-III

3 Environment Key 1 1 1 Agent Morphology 1 3 2 2 Edible food source Poisonous food source Agent Morphology 1 3 Distance sensor Light sensor Front Wheel & Motor 2 2

3 Environment Key 1 1 1 Agent Morphology 1 3 2 2 Edible food source Poisonous food source Agent Morphology 1 3 Distance sensor Light sensor Front Wheel & Motor 2 2

Architecture of Appetitive/Aversive Experiment Mapp Mave DS0 DS1 DS2 DS3 DS4 DS5 LS7 LS0 LS1 LS2 LS3 LS4 LS5 LS6 Left Obs No Obs Right Obs Left Grad Right Grad + + + + + + + - - - Turn Left Move Fwd Turn Right Left App Right App Left Ave Right Ave - - - + - + - - + + - - + + - + + Front 2 3 1 4 5 LM RM Wheel & Motor 7 6 Distance sensor Light sensor

Architecture of Appetitive/Aversive Experiment Smell (Light) (KA-0 10) KA-III OB (8x8 KA-II) AON (2x2 KA-II) V PC (8x8 KA-II) Mapp Mave Valence Hebbian Modification Tasteapp Tasteave

(Loc 1) (Loc 2) (Loc 3) (Loc 4) (Loc 5) (Loc 6) (Loc 7) (Loc 8) a c d b (Loc 1) (Loc 2) (Loc 3) (Loc 4) (Loc 5) (Loc 6) (Loc 7) (Loc 8) (Test d) (Test c) (Test b) (Test a)

Agent Path No Learn (left) and Learn (Right) Environment Key 1 Edible food source 1 Poisonous food source 1 3 1 3 1 1 3 3 2 2 2 2

No Learning Learning Edible Poisonous 59 82 98 40

Conclusion The KA simplification represents a new and very useful tool for exploring mesoscopic level neurodynamic models in autonomous agent simulations. Efficiency Comparability Analyzability Level of Abstraction The KA-III forms aperiodic dynamics with the same temporal and spectral characteristics of biological populations. It has been demonstrated that KA can be used as control mechanisms in autonomous agents. We have also demonstrated that KA can form aperiodic attractors in autonomous agents that represent environmental meanings and are useful in guiding behavior.