FIGURE 4 Responses of dopamine neurons to unpredicted primary reward (top) and the transfer of this response to progressively earlier reward-predicting conditioned stimuli with training (middle). The bottom record shows a control baseline task when the reward is predicted by an earlier stimulus and not the light. From Schultz et al. (1995) with permission.

Odor Selective Cells in the Amygdala fire preferentially with regard to outcome or reward value of an odor prior to demonstration that the animal has learned this outcome or value. Odor Selective Cells in the Amygdala fire preferentially with regard to outcome or reward value of an odor simultaneous to demonstration that the animal has learned this outcome or value.

Cells in Orbitofrontal Cortex (OFC) show less selectivity to outcome, in rats without an amygdala. This demonstrates a role for the amygdala in conveying motivational/reward information to the OFC.

Dopamine, reward processing and optimal prediction ONLY AS A REFERENCE FOR THOSE WHO ARE INTERESTED IN BEGINNING TO CROSS THE NEUROBEHAVIORALCOMPUTATIONAL DIVIDE – Maybe after the Exam??

Human dopaminergic system

Cortical and striatal projections Schultz, 1998

Koob & Le Moal, 2001

Schultz, Dayan & Montague 1997

Expected Reward v = wu v : expected reward w : weight (association) u : stimulus (binary)

Rescorla-Wagner Rule Association update rule: w  w + αδu w : weight (association) α : learning rate u : stimulus Prediction error: δ = r - v r : actual reward v : expected reward

Rescorla - Wagner provides account for: Some Pavlovian conditioning Extinction Partial reinforcement and, with more than one stimulus: Blocking Inhibitory conditioning Overshadowing … but not Latent inhibition (CS preexposure effect) Secondary conditioning

A recent update: uncertainty (  i ² ) Kakade, Montague & Dayan, 2001

Kalman weight update rule: w i  w i + α i δ With associability : α i =  i ² u i  j  j ² u j + E

An example:

U 1 U 2 U 3 U 4 U 5 U(t)U(t) input

U(t)U(t) r(t)r(t)

U(t)U(t) r(t)r(t) w(t)w(t)

U(t)U(t) ŵ(t)ŵ(t) v(t)v(t)

U(t)U(t) r(t)r(t) ŵ(t)ŵ(t) v(t)v(t)

U(t)U(t) r(t)r(t) ŵ(t)ŵ(t) v(t)v(t) δ (t)

 (t) = r(t) - v(t) Error Rule

U(t) ŵ(t)ŵ(t) v(t)v(t) inset U i -input ii w i -uncertainty -weight

Uncertainty

Kalman learning & associability weight update rule: ŵ i (t+1) = ŵ i (t) + α i (t) δ (t) associability: α i (t) =  i (t)² x i (t)  j  j (t)² x j (t) + E

Stimulus uncertainties

Reward prediction

Predicting future reward single time steps: v = wu v : expected reward w : weight (association) u : stimulus total predicted reward: v(t) = w(τ) u(t - τ) t : time steps in a trial τ : current time step tτ=0 tτ=0

Sum of discounted future rewards: With 0 ≤ γ ≤ 1 In recursive form: Schultz, Dayan & Montague, 1997

Exponential discounting, γ =.95

Temporal difference rule Total estimated future reward: v(t) = r(t)+ γv(t+1) r(t) = v(t)-γv(t+1) Temporal difference rule : δ = r(t)+γv(t+1)- v(t) ( With single time steps : δ = r - v r : actual reward v : expected reward )

Temporal difference rule Total estimated future reward: v(t) = r(t)+v(t+1) r(t) = v(t)-v(t+1) Temporal difference rule : δ = r(t) + v(t+1)- v(t) ( With single time steps : δ = r - v r : actual reward v : expected reward )

Schultz, Dayan & Montague, 1997

Schultz, 1996

Anatomical interpretation Schultz, Dayan & Montague, 1997

Temporal Difference Rule for Navigation between successive steps u and u’ δ = r a (u) + γ v(u’)-v(u)

Behavior evaluationHippocampal place field Foster, Morris & Dayan 2000

Spatial learning Foster, Morris & Dayan 2000

Conclusions Behavioral study of (nonhuman) neural systems is interesting Neural processes amenable to contemporary learning theory.. they may play distinct roles a normative framework of learning e.g. vta, hippocampus, subiculum, also- Ach in NBM/SI, NE in LC, 5-HT, ventral striatum, lateral connections, core/shell distinctions of the NAAC, patch-matrix anatomy in basal ganglia, the superior colliculus, psychoalphabetadiscobioaquadodoo