1. Manipulating the teaching signal: effects of dopamine-related drugs on human learning systems Wellcome Trust Centre for NeuroImaging University College London Mathias Pessiglione

2. How can dopamine improve our decisions ? What we may learn from monkeys δ > 0 δ = 0 δ < 0 Schultz 1997 What we may learn from engineers reward prediction GPi / SNr cortex Barto 1995 δ

3. δ(t) = R(t)-Va(t) Va(t+1) = Va(t)+α*δ(t) [Va(t) Vb(t)] Pa(t) = 1/(1+exp((Va(t)-Vb(t))/β)) Behavioural choice Hidden values Learning rule fixation + cues + + choice outcome gain Testing and modelling behavioural choices rewarding cues: neutral cues: punishing cues : a = 0.8/0.2 * ‘gain’ £1/£0 a = 0.8/0.2 * ‘look’ £1/£0 a = 0.8/0.2 * ‘loss’ £1/£0 b = 0.2/0.8 * ‘gain’ £1/£0 b = 0.2/0.8 * ‘look’ £1/£0 b = 0.2/0.8 * ‘loss’ £1/£0

4. Using behavioural choices to optimise parameters Values (£) 0 1 130trials α = learning rate β = temperature δ(t) = R(t)-Va(t) Va(t+1) = Va(t)+α*δ(t) [Va(t) Vb(t)] Pa(t) = 1/(1+exp((Va(t)-Vb(t))/β)) = 0.8 * -£1 = 0.8 * £1 = 0.2 * £1 = 0.2 * -£1 0 50 100 130trials Choices (%) data model

5. Looking for values hidden in the brain ventral striatumanterior putamen posterior putamen δ = reward prediction error δ(t) = R(t)-Va(t) Va(t+1) = Va(t)+α*δ(t) [Va(t) Vb(t)] Pa(t) = 1/(1+exp((Va(t)-Vb(t))/β))

6. Striatal responses reflecting prediction errors LEVODOPAHALOPERIDOL Gain / £1 Gain / £0 Signal change (%) 1.30.7 GAIN Signal change (%) Time (sec) Loss / £0 Loss / -£1 1.0 Time (sec) 1.0 LOSS

7. Explaining drug effects on behavioural choices LEVODOPA PLACEBO HALOPERIDOL 0.8 * £1 Choice (%) 0 50 100 0.8 * -£1 1 30 trials data model

8. How can striatal regions influence behavioural choice ? Go - NoGo posterior putamen (response facilitation) Loss - NeutralGain - Neutral ventral striatum (reward prediction)

9. A brainy model Haber 2003 posterior putamen ventral striatum SNc / VTA REWARD δ dopamine ACTOR execution CRITIC prediction ACTION GPi / SNr

10. Thanks Ben Seymour Guillaume Flandin Ray Dolan Chris Frith Wellcome Trust Centre for NeuroImaging University College London