1. Neural Coding of Basic Reward Terms of Animal Learning Theory, Game Theory, Microeconomics and Behavioral Ecology Wolfram Schultz Current Opinion in Neurobiology 2004, 14: 139-147 Qing Yu Christina Weng October 10, 2008

2. Purpose To summarize current knowledge of the basic components of reward information extracted from environmental stimuli and processed by reward mechanisms in the brain. To explore neural correlates for the role of rewards in learning, behavior, and decision-making.

3. Background Animal Learning Theory: Functions of rewards 1)Positive reinforcement 2)Induce consummatory behavior 3)Induce positive emotions Learning requires: 1)a cause-and-affect association between a conditioned stimulus or movement and the reinforcer. 2)Contingency Types of reward-directed learning: 1)Pavlovian conditioning 2)Operant conditioning

4. Background Motivational Value Uncertainty: maximum at probability p = 0.5 assessed as entropy, variance,…. Negative Contrast Effect

5. Behavioral Theories and Terms Learning Theory —attempts to describe how organisms react to rewards and acquire new reactions to stimuli. Microeconomics —assess value of rewards for decision-making Game Theory —used in behavioral ecology; attempts to model behavior in strategic situations. Individuals will try to maximize utility of outcome and maintain stability based on behavior of other players  Nash Equilibrium— “efficient conditioned behavior” where players reach optimal returns. Reward Value = magnitude x probability

6. The Role of Expectation Reward-predicting stimuli: -Through learning, subjects are able to collect the award before it is administered. Reward expectation period: -Some neurons maintain elevated activity

7. Expectation and Response Increased neuronal activity Increased preference Increased expectation Increased expected reward value  elevated task-related activation of parietal neurons

8. Schematic forms of Reward Coding (a) Receipt of award  change in neuronal activity (simplest form) (b) Stimulus  neuronal response (reward prediction) (c) Behavioral reaction toward stimulus  elevated neuronal activity  reward (expectation)

9. Behavior-related Activation in Dopamine Neurons

10. The Effect of Probability Dopamine neurons: Increased probability of reward  increased response to reward-predicting stimulus Parietal neurons: Increased expected reward value  increased task-related activation Track recently experienced rewards

11. Neuronal Response with Respect to Food and Liquid Rewards Dopamine Neurons Orbitofrontal cortex Striatum/nucleus accumbens Amygdala Detect reward; expectation response Discriminate among stimuli on basis of predicted rewards Parietal cortex Dorsolateral prefrontal cortex Mnemonic and movement preparatory periods of delayed response tasks Decision-making; behavioral organization Stronger task-related activation in rewarded rather than unrewarded trials.

12. Preference Coding for Food Rewards in the Orbitofrontal Neuron

13. Neuronal Response with Respect to Food and Liquid Rewards Posterior orbitofrontal cortex: physical characteristics of rewards (ie. Glucose concentration, fat texture) Parietal neurons: activity reflects subjective utility for the individual animal rather than the expected value.

14. Difference in activation increases with difference in value between the unrewarded and rewarded trials. A neural correlate exists for reward-based behavior Utility of rewarding outcomes is coded following microeconomics Experiment Results and Conclusions

15. Reward Prediction Error Overestimated reward Expected reward Underestimated reward Dopamine neurons DepressedUnchangedActivated Unexpected Reward Orbitofrontal neuronsStriatal neurons Absent reward due to organism error Dorsolateral prefrontal cortex Anterior cingulate Posterior cingulate Frontal eye fields Neuronal Activation as a result of Unexpected and Failed Rewards

16. Neural-coding for goal-direction in the caudate nucleus neuron

17. Experiment Results and Conclusions Reward information influences neural activations related to arm and eye movements. These influences reflect the reward and action-reward relationship represented in goal-directed mechanisms.

18. The Underlying Mechanisms of Decision-Making Game Theory: Magnitude of reward Probability Utility Race Model: Threshold of neural activity  behavioral choice Constantly incoming information  approach decision

19. Conclusions and Future Directions Reward coding can be explained in terms of animal learning theory, microeconomics, and game theory Multiple mechanisms exist in different brain structures to extract reward information from stimuli Lingering Questions: What are the brain processes that evaluate reward values to make priority decisions? What is the effect of delay of reward on reward value? How is time coded?