Download presentation
Presentation is loading. Please wait.
7
FIGURE 4 Responses of dopamine neurons to unpredicted primary reward (top) and the transfer of this response to progressively earlier reward-predicting conditioned stimuli with training (middle). The bottom record shows a control baseline task when the reward is predicted by an earlier stimulus and not the light. From Schultz et al. (1995) with permission.
10
Odor Selective Cells in the Amygdala fire preferentially with regard to outcome or reward value of an odor prior to demonstration that the animal has learned this outcome or value. Odor Selective Cells in the Amygdala fire preferentially with regard to outcome or reward value of an odor simultaneous to demonstration that the animal has learned this outcome or value.
11
Cells in Orbitofrontal Cortex (OFC) show less selectivity to outcome, in rats without an amygdala. This demonstrates a role for the amygdala in conveying motivational/reward information to the OFC.
14
Dopamine, reward processing and optimal prediction ONLY AS A REFERENCE FOR THOSE WHO ARE INTERESTED IN BEGINNING TO CROSS THE NEUROBEHAVIORALCOMPUTATIONAL DIVIDE – Maybe after the Exam??
15
Human dopaminergic system
16
Cortical and striatal projections Schultz, 1998
17
Koob & Le Moal, 2001
18
Schultz, Dayan & Montague 1997
19
Expected Reward v = wu v : expected reward w : weight (association) u : stimulus (binary)
20
Rescorla-Wagner Rule Association update rule: w w + αδu w : weight (association) α : learning rate u : stimulus Prediction error: δ = r - v r : actual reward v : expected reward
21
Rescorla - Wagner provides account for: Some Pavlovian conditioning Extinction Partial reinforcement and, with more than one stimulus: Blocking Inhibitory conditioning Overshadowing … but not Latent inhibition (CS preexposure effect) Secondary conditioning
22
A recent update: uncertainty ( i ² ) Kakade, Montague & Dayan, 2001
23
Kalman weight update rule: w i w i + α i δ With associability : α i = i ² u i j j ² u j + E
24
An example:
25
U 1 U 2 U 3 U 4 U 5 U(t)U(t) input
26
U(t)U(t) r(t)r(t)
27
U(t)U(t) r(t)r(t) w(t)w(t)
28
U(t)U(t) ŵ(t)ŵ(t) v(t)v(t)
29
U(t)U(t) r(t)r(t) ŵ(t)ŵ(t) v(t)v(t)
30
U(t)U(t) r(t)r(t) ŵ(t)ŵ(t) v(t)v(t) δ (t)
31
(t) = r(t) - v(t) Error Rule
32
U(t) ŵ(t)ŵ(t) v(t)v(t) inset U i -input ii w i -uncertainty -weight
33
Uncertainty
34
Kalman learning & associability weight update rule: ŵ i (t+1) = ŵ i (t) + α i (t) δ (t) associability: α i (t) = i (t)² x i (t) j j (t)² x j (t) + E
36
Stimulus uncertainties
37
Reward prediction
38
Predicting future reward single time steps: v = wu v : expected reward w : weight (association) u : stimulus total predicted reward: v(t) = w(τ) u(t - τ) t : time steps in a trial τ : current time step tτ=0 tτ=0
39
Sum of discounted future rewards: With 0 ≤ γ ≤ 1 In recursive form: Schultz, Dayan & Montague, 1997
40
Exponential discounting, γ =.95
41
Temporal difference rule Total estimated future reward: v(t) = r(t)+ γv(t+1) r(t) = v(t)-γv(t+1) Temporal difference rule : δ = r(t)+γv(t+1)- v(t) ( With single time steps : δ = r - v r : actual reward v : expected reward )
42
Temporal difference rule Total estimated future reward: v(t) = r(t)+v(t+1) r(t) = v(t)-v(t+1) Temporal difference rule : δ = r(t) + v(t+1)- v(t) ( With single time steps : δ = r - v r : actual reward v : expected reward )
43
Schultz, Dayan & Montague, 1997
44
Schultz, 1996
45
Anatomical interpretation Schultz, Dayan & Montague, 1997
46
Temporal Difference Rule for Navigation between successive steps u and u’ δ = r a (u) + γ v(u’)-v(u)
47
Behavior evaluationHippocampal place field Foster, Morris & Dayan 2000
48
Spatial learning Foster, Morris & Dayan 2000
49
Conclusions Behavioral study of (nonhuman) neural systems is interesting Neural processes amenable to contemporary learning theory.. they may play distinct roles a normative framework of learning e.g. vta, hippocampus, subiculum, also- Ach in NBM/SI, NE in LC, 5-HT, ventral striatum, lateral connections, core/shell distinctions of the NAAC, patch-matrix anatomy in basal ganglia, the superior colliculus, psychoalphabetadiscobioaquadodoo
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.