A View from the Bottom Peter Dayan Gatsby Computational Neuroscience Unit.

Slides:

Advertisements

Similar presentations

Reinforcement Learning I: prediction and classical conditioning

Advertisements

Design of Experiments Lecture I

Detecting Conflict-Related Changes in the ACC Judy Savitskaya 1, Jack Grinband 1,3, Tor Wager 2, Vincent P. Ferrera 3, Joy Hirsch 1,3 1.Program for Imaging.

dopamine and prediction error

Quasi-Continuous Decision States in the Leaky Competing Accumulator Model Jay McClelland Stanford University With Joel Lachter, Greg Corrado, and Jim Johnston.

Computational Neuromodulation Peter Dayan Gatsby Computational Neuroscience Unit University College London Nathaniel Daw Sham Kakade Read Montague John.

Global plan Reinforcement learning I: –prediction –classical conditioning –dopamine Reinforcement learning II: –dynamic programming; action selection –Pavlovian.

Decision Dynamics and Decision States: the Leaky Competing Accumulator Model Psychology 209 March 4, 2013.

Journal club Marian Tsanov Reinforcement Learning.

Dopamine, Uncertainty and TD Learning CNS 2004 Yael Niv Michael Duff Peter Dayan Gatsby Computational Neuroscience Unit, UCL.

CS 182/CogSci110/Ling109 Spring 2008 Reinforcement Learning: Details and Biology 4/3/2008 Srini Narayanan – ICSI and UC Berkeley.

Introduction: What does phasic Dopamine encode ? With asymmetric coding of errors, the mean TD error at the time of reward is proportional to p(1-p) ->

Dopamine, Uncertainty and TD Learning CoSyNe’04 Yael Niv Michael Duff Peter Dayan.

Is it an error to be ‘too different’!? Neurobiology of social conformity Ale Smidts a Co-authors: Vasily Klucharev ab Kaisa Hytönen ab, Mark Rijpkema b.

Reward processing (1) There exists plenty of evidence that midbrain dopamine systems encode errors in reward predictions (Schultz, Neuron, 2002) Changes.

Decision-making I choosing between gambles neural basis of decision-making.

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 7: Coding and Representation 1 Computational Architectures in.

From T. McMillen & P. Holmes, J. Math. Psych. 50: 30-57, MURI Center for Human and Robot Decision Dynamics, Sept 13, Phil Holmes, Jonathan.

FIGURE 4 Responses of dopamine neurons to unpredicted primary reward (top) and the transfer of this response to progressively earlier reward-predicting.

Serotonin and Impulsivity Yufeng Zhang. Serotonin Originate from the median and dorsal raphe nuclei. Serotonin has been implicated in a variety of motor,

Episodic Control: Singular Recall and Optimal Actions Peter Dayan Nathaniel Daw Máté Lengyel Yael Niv.

Theory of Decision Time Dynamics, with Applications to Memory.

From Localization to Connectivity and... Lei Sheu 1/11/2011.

Prediction in Human Presented by: Rezvan Kianifar January 2009.

Reinforcement learning and human behavior Hanan Shteingart and Yonatan Loewenstein MTAT Seminar in Computational Neuroscience Zurab Bzhalava.

Decisions & the Brain. Neuroeconomics CCN Lecture Matteo Colombo 11 February 2010.

Dopamine enhances model-based over model-free choice behavior Peter Smittenaar *, Klaus Wunderlich *, Ray Dolan.

CS344 : Introduction to Artificial Intelligence Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 26- Reinforcement Learning for Robots; Brain Evidence.

What is a decision, and what’s going on in the brain? Elisabeth Rounis and Louise Whiteley The mathematical brain:

Basic Behavioral Neurobiology of Drug Addiction J. David Jentsch, Ph.D. Assistant Professor of Psychology (Behavioral Neuroscience)

Chapter 16. Basal Ganglia Models for Autonomous Behavior Learning in Creating Brain-Like Intelligence, Sendhoff et al. Course: Robots Learning from Humans.

Bayesian Modelling of Functional Imaging Data Will Penny The Wellcome Department of Imaging Neuroscience, UCL http//:

CHAPTER 10 Reinforcement Learning Utility Theory.

Dynamic Causal Modelling Will Penny Wellcome Department of Imaging Neuroscience, University College London, UK FMRIB, Oxford, May

Dynamic Decision Making in Complex Task Environments: Principles and Neural Mechanisms Annual Workshop Introduction August, 2008.

D ECIDING WHEN TO CUT YOUR LOSSES Matt Cieslak, Tobias Kluth, Maren Stiels & Daniel Wood.

Abstract We offer a formal treatment of choice behaviour based on the premise that agents minimise the expected free energy of future outcomes. Crucially,

Summary of part I: prediction and RL Prediction is important for action selection The problem: prediction of future reward The algorithm: temporal difference.

Neural Reinforcement Learning Peter Dayan Gatsby Computational Neuroscience Unit thanks to Yael Niv for some slides.

Ch15: Decision Theory & Bayesian Inference 15.1: INTRO: We are back to some theoretical statistics: 1.Decision Theory –Make decisions in the presence of.

The Computing Brain: Focus on Decision-Making

Neural Correlates of Learning Models in a Dynamic Decision-Making Task Darrell A. Worthy 1, Marissa A. Gorlick 2, Jeanette Mumford 2, Akram Bakkour 2,

Bayesian inference accounts for the filling-in and suppression of visual perception of bars by context Li Zhaoping 1 & Li Jingling 2 1 University College.

Global plan Reinforcement learning I: –prediction –classical conditioning –dopamine Reinforcement learning II: –dynamic programming; action selection –Pavlovian.

Accuracy, Reliability, and Validity of Freesurfer Measurements David H. Salat

Neural correlates of risk sensitivity An fMRI study of instrumental choice behavior Yael Niv, Jeffrey A. Edlund, Peter Dayan, and John O’Doherty Cohen.

Does the brain compute confidence estimates about decisions?

Neural Coding of Basic Reward Terms of Animal Learning Theory, Game Theory, Microeconomics and Behavioral Ecology Wolfram Schultz Current Opinion in Neurobiology.

Group Analyses Guillaume Flandin SPM Course London, October 2016

Dynamic Causal Modeling of Endogenous Fluctuations

Ale Smidtsa Co-authors: Vasily Klucharevab Kaisa Hytönenab,

Neuroimaging of associative learning

neuromodulators; midbrain; sub-cortical;

The Neurobiology of Decision: Consensus and Controversy

Emotion, Decision Making, and the Amygdala

Neuroimaging of associative learning

Braden A. Purcell, Roozbeh Kiani Neuron

Presented by: Rezvan Kianifar January 2009

Volume 62, Issue 5, Pages (June 2009)

Hedging Your Bets by Learning Reward Correlations in the Human Brain

The Neurobiology of Decision: Consensus and Controversy

Interaction of Sensory and Value Information in Decision-Making

Ralf M. Haefner, Pietro Berkes, József Fiser Neuron

Moral Judgments Recruit Domain-General Valuation Mechanisms to Integrate Representations of Probability and Magnitude Amitai Shenhav, Joshua D. Greene

Joseph T. McGuire, Matthew R. Nassar, Joshua I. Gold, Joseph W. Kable

Wallis, JD Helen Wills Neuroscience Institute UC, Berkeley

Erie D. Boorman, John P. O’Doherty, Ralph Adolphs, Antonio Rangel

Neuroimaging of associative learning

Neural Mechanisms Underlying Human Consensus Decision-Making

Mitsuo Kawato ATR Computational Neuroscience Labs

Presentation transcript:

A View from the Bottom Peter Dayan Gatsby Computational Neuroscience Unit

Neural Decision Making bewilderingly vast topic models playing a central role – so beware of self-confirmation + battles

3 Ethology/Economics(?) – optimality – logic of the approach Psychology – economic choices – instrumental/Pavlovian conditioning Computation Algorithm Implementation/Neurobiology neuromodulators; amygdala; prefrontal cortex nucleus accumbens; dorsal striatum prediction: of important events control: in the light of those predictions Neural Decision Making

Imprecision & Noise computation – Bayesian sensory inference – Kalman filtering and optimal learning – metacognition – exploration/exploitation – game theory

Imprecision & Noise algorithm – multiple methods of choice instrumental: model-based; model-free – (note influence on RTs) Pavlovian: evolutionary programming – uncertainty-modulated inference and learning – DFT/drift diffusion decision-making – MCMC methods for inference

Imprecision & Noise implementation – (where does the noise come from?) – evidence accumulation – Q-learning and dopamine – metacognition and the PFC – acetylcholine/norepinephrine and uncertainty- sensitive inference and learning

Diffusion to Bound Britten et al, 1992

Diffusion to Bound expected reward, priors affect starting point some evidence for urgency signal works for discrete evidence (WPT) less data on >2 options micro-stimulation works as expected decision via striatum/superior colliculus/etc? choice probability for single neurons Gold & Shadlen, 2007

9 dopamine and prediction error no predictionprediction, rewardprediction, no reward TD error VtVt R RL

Probability and Magnitude Tobler et al, 2005 Fiorillo et al, 2003

Risk Processing < 1 sec 0.5 sec You won 40 cents 5 sec ISI 19 subjects (dropped 3 non learners, N=16) 3T scanner, TR=2sec, interleaved 234 trials: 130 choice, 104 single stimulus randomly ordered and counterbalanced 2-5sec ITI 5 stimuli: 40¢ 20¢ 0 / 40¢ 0¢ 5 stimuli: 40¢ 20¢ 0 / 40¢ 0¢

Neural results: Prediction errors what would a prediction error look like (in BOLD)?

Neural results I: Prediction errors in NAC unbiased anatomical ROI in nucleus accumbens (marked per subject*) * thanks to Laura deSouza raw BOLD (avg over all subjects)

Value Independent of Choice Roesch et al, 2007

Metacognition Fleming et al, 2010 contrast staircase for performance; type II ROC for confidence

Structural Correlate also associated white matter (connections)

Discussion what can economics do for us? – theoretical, experimental ideas – experimental methods – like behaviorism… what can we do for economics? – large range of constraints – objects of experimental inquiry precisely aligned with economic notions – grounding/excuse for complexity…