BrainDisC PhD Conference: Active inference and epistemic value Karl Friston, University College London Abstract: I will talk about a formal treatment of choice behaviour based on the premise that agents minimise the expected free energy of future outcomes. Crucially, the negative free energy or quality of a policy can be decomposed into extrinsic and epistemic (intrinsic) value. Minimising expected free energy is therefore equivalent to maximising extrinsic value or expected utility (defined in terms of prior preferences or goals), while maximising information gain or intrinsic value; i.e., reducing uncertainty about the causes of valuable outcomes. The resulting scheme resolves the exploration-exploitation dilemma: epistemic value is maximised until there is no further information gain, after which exploitation is assured through maximisation of extrinsic value. This is formally consistent with the Infomax principle, generalising formulations of active vision based upon salience (Bayesian surprise) and optimal decisions based on expected utility and risk sensitive (KL) control. Furthermore, as with previous active inference formulations of discrete (Markovian) problems; ad hoc softmax parameters become the expected (Bayes-optimal) precision of beliefs about – or confidence in – policies. We focus on the basic theory – illustrating the minimisation of expected free energy using simulations. A key aspect of this minimisation is the similarity of precision updates and dopaminergic discharges observed in conditioning paradigms. Key words: active inference ∙ cognitive ∙ dynamics ∙ free energy ∙ epistemic value ∙ self-organization

Does the brain use continuous or discrete state-space models?

Sufficient statistics Does the brain use continuous or discrete state-space models? States and their Sufficient statistics

Approximate Bayesian inference for continuous states: Bayesian filtering States and their Sufficient statistics Free energy Model evidence

Impressions on the Markov blanket… “Objects are always imagined as being present in the field of vision as would have to be there in order to produce the same impression on the nervous mechanism” - von Helmholtz Hermann von Helmholtz Richard Gregory Impressions on the Markov blanket…

Bayesian filtering and predictive coding prediction update prediction error

sensations – predictions Making our own sensations sensations – predictions Prediction error Action Perception Changing sensations Changing predictions

Hierarchical generative models the Hierarchical generative models what where A simple hierarchy Descending predictions Ascending prediction errors Sensory fluctuations

Predictive coding with reflexes David Mumford Predictive coding with reflexes Action oculomotor signals reflex arc proprioceptive input pons Perception retinal input Prediction error (superficial pyramidal cells) Expectations (deep pyramidal cells) frontal eye fields geniculate Top-down or backward predictions Bottom-up or forward prediction error visual cortex

What does Bayesian filtering explain?

What does Bayesian filtering explain?

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics Specify a deep (hierarchical) generative model Simulate approximate (active) Bayesian inference by solving: prediction update

An example: Visual searches and saccades

“I am [ergodic] therefore I think [I will minimise free energy]” Likelihood Empirical priors Prior beliefs Free energy Entropy Energy “I am [ergodic] therefore I think [I will minimise free energy]” Extrinsic value Epistemic value or information gain Expected energy Expected entropy KL or risk-sensitive control In the absence of ambiguity: Expected utility theory In the absence of uncertainty or risk: Bayesian surprise and Infomax In the absence of prior beliefs about outcomes: Predicted divergence Extrinsic value Bayesian surprise Predicted mutual information

“I am [ergodic] therefore I think [I will minimise free energy]” Likelihood Empirical priors Prior beliefs Free energy Entropy Energy “I am [ergodic] therefore I think [I will minimise free energy]” Epistemic value or information gain 𝜎(𝑢 Extrinsic value stimulus visual input salience sampling Sampling the world to resolve uncertainty

Deep (hierarchical) generative model Parietal (where) Frontal eye fields Visual cortex Pulvinar salience map Fusiform (what) oculomotor reflex arc Superior colliculus

Saccadic eye movements Saccadic fixation and salience maps Hidden (oculomotor) states Visual samples vs. Conditional expectations about hidden (visual) states And corresponding percept

What about state space models? Is Bayesian filtering the only process theory for approximate Bayesian inference in the brain? What about state space models?

A (Markovian) generative model Likelihood Control states Empirical priors – hidden states – control states Hidden states Full priors

Prior beliefs about policies Quality of a policy = (negative) expected free energy Extrinsic value Epistemic value or information gain Bayesian surprise and Infomax In the absence of prior beliefs about outcomes: KL or risk-sensitive control In the absence of ambiguity: Expected utility theory In the absence of uncertainty or risk: Bayesian surprise Extrinsic value Predicted divergence Predicted mutual information

Generative models Continuous states Discrete states Bayesian filtering Hidden states Control states Bayesian filtering (predictive coding) Variational Bayes (belief updating)

Variational updates Functional anatomy Perception Action selection Incentive salience VTA/SN motor Cortex occipital Cortex striatum prefrontal Cortex hippocampus 0.2 0.4 0.6 0.8 1 20 40 60 80 Performance Prior preference success rate (%) FE KL RL DA Simulated behaviour

Variational updating Functional anatomy Perception Action selection Incentive salience VTA/SN motor Cortex occipital Cortex striatum prefrontal Cortex hippocampus 0.2 0.4 0.6 0.8 1 1.2 1.4 5 10 15 20 25 30 0.25 0.5 0.75 1.25 1.5 -0.02 0.02 0.04 0.06 0.08 40 50 60 70 80 90 -0.05 0.05 0.1 0.15 Simulated neuronal behaviour

What does believe updating explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics Specify a deep (hierarchical) generative model Simulate approximate (active) Bayesian inference by solving:

An example: Exploration and exploitation

Generative model (A,B,C) Control states Generative model (A,B,C) 1 2 3 4 location US US Hidden states CS location context Outcomes

Functional anatomy Variational updates Motor Cortex Dorsal prefrontal Action (and Bayesian model averaging) State estimation (planning) State estimation (habitual) Policy selection Precision Learning Functional anatomy Motor Cortex Dorsal prefrontal Striatum Hippocampus VTA/SN Occipital Cortex Ventral prefrontal Variational updates Cerebellum

Bayesian model average of next state Functional anatomy Predicted action Dorsal prefrontal Motor Cortex Bayesian model average of next state Policy selection Hippocampus Precision State estimation under plausible policies Ventral prefrontal Striatum & VTA/SN Occipital Cortex Evaluation of policies Sensory input Habit learning Cerebellum

What does Bayesian filtering explain? Cross frequency coupling (phase precession) Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics Unit responses time (seconds) 0.25 0.5 0.75 8 16 24 -0.02 0.02 0.04 0.06 0.08 0.1 0.12 0.14 Local field potentials Response

What does Bayesian filtering explain? Cross frequency coupling (phase synchronisation) Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics Time-frequency (and phase) response time (seconds) frequency 1 2 3 4 5 6 10 15 20 25 30 0.25 0.5 0.75 1.25 1.5 1.75 2.25 2.5 2.75 3.25 3.5 3.75 4.25 4.5 4.75 5.25 5.5 5.75 -0.2 0.2 0.4 Local field potentials Response 50 100 150 200 250 300 350 0.05 0.1 0.15 Phasic dopamine responses time (updates) change in precision

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics Time-frequency (and phase) response time (seconds) frequency 0.2 0.4 0.6 0.8 1 1.2 1.4 5 10 15 20 25 30 0.25 0.5 0.75 1.25 1.5 -0.02 0.02 0.04 0.06 0.08 Local field potentials Response 40 50 60 70 80 90 -0.05 0.05 0.1 0.15 Phasic dopamine responses time (updates) change in precision

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics 50 100 150 200 250 300 350 -0.06 -0.04 -0.02 0.02 0.04 0.06 0.08 0.1 0.12 0.14 Time (ms) LFP Difference waveform (MMN) oddball standard MMN 0.25 0.5 0.75 1 1.25 1.5 -0.1 0.1 0.2 0.3 0.4 Local field potentials time (seconds) Response 10 20 30 40 50 60 70 80 90 0.05 0.15 Phasic dopamine responses time (updates) change in precision

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance (transfer of responses) Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (devaluation) Interoceptive inference Communication and hermeneutics

Does the brain use continuous or discrete state space models or both? Hidden states Control states

Thank you And thanks to collaborators: And colleagues: Rick Adams Ryszard Auksztulewicz Andre Bastos Sven Bestmann Harriet Brown Jean Daunizeau Mark Edwards Chris Frith Thomas FitzGerald Xiaosi Gu Stefan Kiebel James Kilner Christoph Mathys Jérémie Mattout Rosalyn Moran Dimitri Ognibene Sasha Ondobaka Will Penny Giovanni Pezzulo Lisa Quattrocki Knight Francesco Rigoli Klaas Stephan Philipp Schwartenbeck And colleagues: Micah Allen Felix Blankenburg Andy Clark Peter Dayan Ray Dolan Allan Hobson Paul Fletcher Pascal Fries Geoffrey Hinton James Hopkins Jakob Hohwy Mateus Joffily Henry Kennedy Simon McGregor Read Montague Tobias Nolte Anil Seth Mark Solms Paul Verschure And many others