1. 2/11/20071 ACQ and the Basal Ganglia Jimmy Bonaiuto USC Brain Project 2/12/2007

2. 2/11/20072 Outline Alstermark’s Cat ACQ ACQ → Basal Ganglia Basal Ganglia Model Implementations (NSL) The Search for Executability

3. 2/11/20073 Alstermark’s Cat

4. 2/11/20074 ACQ

5. 2/11/20075 ACQ

6. 2/11/20076 ACQ - Executability -2D Gaussian kernel populations -Food location relative to mouth -Food location relative to paw -Food location relative to tube opening

7. 2/11/20077 Learning Executability - Success or failure is signaled by the match or mismatch between efferent signals and mirror system output

8. 2/11/20078 Learning Desirability - Eligibility signal computed from - Internal state - Mirror system output - Efferent signal

9. 2/11/20079 Priority Simplified form: priority = executability × desirability Leaky integrator form:

10. 2/11/200710 Action Selection - Winner declared when max CC layer element firing rate is greater or equal to ε 1 (0.9) and all other element firing rates are less than or equal to ε 2 (0.1).

11. 2/11/200711 ACQ

12. 2/11/200712 ACQ Selection Properties Contrast- Dependent Latency

13. 2/11/200713 ACQ Selection Properties -Approximation to Boltzmann equation: T=temperature

14. 2/11/200714 ACQ – TD Learning No initialized weights Eat initialized Reach-grasp initialized Effects of Desirability Weight Initialization on Mean Trial Length During TD Learning

15. 2/11/200715 ACQ – Simulation Results Final Desirability Weights Mean Trial Length Mean Unsuccessful Action Attempts

16. 2/11/200716 ACQ – Simulation Results MF - Eat MF – Grasp Jaw PF – Reach Food PF – Reach Tube

17. 2/11/200717 Where in the Brain is ACQ? Affordances –Posterior parietal cortex Object-directed motor schemas –Premotor cortex Winner-Take-All –Basal ganglia (Winner-Lose-All) Desirability Learning –Striatum with TD error signal from midbrain dopaminergic system (SNc, VTA) What about Executability?

18. 2/11/200718 Basal Ganglia Model Implementations (NSL) The following models are implemented in NSL and available for extension or experimentation: –GPR –Brown, Bullock, & Grossberg –RDDR

19. 2/11/200719 Gurney, Prescott, Redgrave (GPR) -Interlayer winner-lose-all -Control signal calculated from the sum of the cortical signal provides a gain signal to the competition

20. 2/11/200720 GPR GPi/SNr Str-D1CortexStr-D2STN GPe

21. 2/11/200721 GPR What does a consideration of the GPR model bring to ACQ? –Intralayer WTA → Interlayer WTA –WTA → WLA Do we need a control (gain) signal? –We may want to explore the possibility of chunking when two actions are activated to similar levels

22. 2/11/200722 Brown, Bullock, & Grossberg Ventral striatum → ventral pallidum → PPTN Learns to activate SNc given secondary reinforcer Cortex → Striosomes Learns to inhibit SNc response to primary reinforcer Learns timing between primary and secondary reinforcers

23. 2/11/200723 Brown, Bullock, & Grossberg

24. 2/11/200724 Brown, Bullock, & Grossberg

25. 2/11/200725 Brown, Bullock, & Grossberg

26. 2/11/200726 Brown, Bullock, & Grossberg What does a consideration of the Brown, Bullock, & Grossberg model bring to ACQ? –A neural method of computing the TD error signal –Can we extend it to have multiple primary reinforcers (dimensions of reinforcement)?

27. 2/11/200727 Reinforcement Driven Dimensionality Reduction (RDDR) Extension of PCA neural network methods to include reinforcement Feedforward connections: normalized multi-Hebbian with reinforcement Lateral connections: normalized anti-Hebbian

28. 2/11/200728 RDDR - Pretraining

29. 2/11/200729 RDDR – Mid-training

30. 2/11/200730 RDDR - Trained

31. 2/11/200731 RDDR - Retraining

32. 2/11/200732 RDDR - Retrained

33. 2/11/200733 RDDR What does a consideration of the Brown, Bullock, & Grossberg model bring to ACQ? –Maybe nothing, but it may be useful in chunking actions in hACQ

34. 2/11/200734 Where is Executability? We can map ACQ onto the basic BG architecture by modeling an interlayer WLA network with cortico-striatal connection weights encoding desirability and modified via TD learning How does executability fit in?

35. 2/11/200735 Parietal / Basal Ganglia Projections Petras (1971) – Projections from the inferior and superior parietal lobules to the striatum and thalamus Cavada & Goldman (1991) – Subregions of parietal area 7 project to portions of the striatum bilaterally Flaherty & Graybiel (1991) – Somatotopic projections from S1 to the striatum –Only innervates matrix – not striosomes Graziano & Gross (1993) – Bimodal somatotopic map in putamen Lawrence et al. (2000) –Dorsal stream projects to the anterodorsal striatum

36. 2/11/200736 ACQ Basal Ganglia Could executability and desirability be represented in segregated regions of the striatum and be combined in the globus pallidus? Or perhaps they are combined in the striatum?

37. 2/11/200737 References Bar-Gad, I., Morris, G., Bergman, H. (2003) Information processing, dimensionality reduction and reinforcement learning in the basal ganglia. Progress in Neurobiology, 71: 439–473. Brown, J., Bullock, D., Grossberg, S. (1999) How the Basal Ganglia Use Parallel Excitatory and Inhibitory Learning Pathways to Selectively Respond to Unexpected Rewarding Cues. J. Neurosci., 19(23): 10502-10511. Cavada, C., Goldman-Rakic, P.S. (1991) Topographic Segregation of Corticostriatal Projections from Posterior Parietal Subdivisions in the Macaque Monkey. Neuroscience, 42(3): 683-696. Flaherty, A.W., Graybiel, A.M. (1991) Corticostriatal Transformations in the Primate Somatosensory System. Projections from Physiologically Mapped Body-Part Representations. J. Neurophys. 66(4): 1249-1263. Graziano, M.S.A., Gross, C.G. (1993) A bimodal map of space: Somatosensory receptive fields in the macaque putamen with corresponding visual receptive fields. Exp Brain Res, 97: 96-109. Gurney, K., Prescott, T.J., Redgrave, P. (2001) A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol. Cybern. 84: 401-410. Lawrence, A.D., Watkins, L.H.A., Sahakian, B.J., Hodges, J.R., Robbins, T.W. (2000) Visual object and visuospatial cognition in Huntington’s disease: implications for information processing in corticostriatal circuits. Brain, 123: 1349-1364. Petras, J.M. (1971) Connections of the Parietal Lobe. J. Psychiat. Res., 8: 189-201.