CDB Exploring Science and Society Seminar Thursday 19 November 2009 at 5.30pm Host: Prof Giorgio Gabella The Bayesian brain, surprise and free-energy Abstract Value-learning and perceptual learning have been an important focus over the past decade, attracting the concerted attention of experimental psychologists, neurobiologists and the machine learning community. Despite some formal connections; e.g., the role of prediction error in optimizing some function of sensory states, both fields have developed their own rhetoric and postulates. In work, we show that perceptual learning is, literally, an integral part of value learning; in the sense that perception is necessary to integrate out dependencies on the inferred causes of sensory information. This enables the value of sensory trajectories to be optimized through action. Furthermore, we show that acting to optimize value and perception are two aspects of exactly the same principle; namely the minimization of a quantity (free energy) that bounds the probability of sensory input, given a particular agent or phenotype. This principle can be derived, in a straightforward way, from the very existence of agents, by considering the probabilistic behavior of an ensemble of agents belonging to the same class.

“Objects are always imagined as being present in the field of vision as would have to be there in order to produce the same impression on the nervous mechanism” - Hermann Ludwig Ferdinand von Helmholtz Thomas Bayes Geoffrey Hinton Richard Feynman From the Helmholtz machine and the Bayesian Brain to Action and self-organization Hermann Haken

Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong and categorization Simulated lesions Action Active inference Reaching Policies Control and attractors The mountain-car problem

Particle density contours showing Kelvin-Helmholtz instability, forming beautiful breaking waves. In the self- sustained state of Kelvin-Helmholtz turbulence the particles are transported away from the mid-plane at the same rate as they fall, but the particle density is nevertheless very clumpy because of a clumping instability that is caused by the dependence of the particle velocity on the local solids-to-gas ratio (Johansen, Henning, & Klahr 2006) temperature pH falling transport Self-organization that minimises an ensemble density to ensure a limited repertoire of states are occupied (i.e., ensuring states have a random attracting set).

How can an active agent minimise its equilibrium entropy? This entropy is bounded by the entropy of sensory signals (under simplifying assumptions) Crucially, because the density on sensory signals is at equilibrium, it can be interpreted as the proportion of time each agent entertains them (the sojourn time). This ergodic argument means that entropy is the path integral of surprise experienced by a particular agent: This means agents minimise surprise at all times. But there is one small problem… Agents cannot access surprise; however, they can evaluate a free-energy bound on surprise, which is induced with a recognition density q :

Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

Action External states in the world Internal states of the agent (m) Sensations The free-energy principle Action to minimise a bound on surprisePerception to optimise the bound

The free-energy rests on expected Gibb’s energy and can be evaluated, given a generative model comprising a likelihood and prior: So what models might the brain use? The generative model

Processing hierarchy Backward (nonlinear) Forward (linear) lateral Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

Hierarchical (deep) dynamic models

Structural priors Dynamical priors Likelihood and empirical priors Hierarchal form Gibb’s energy: a simple function of prediction error Prediction errors

Synaptic gain Synaptic activity Synaptic efficacy Activity-dependent plasticity Functional specialization Attentional gain Enabling of plasticity Attention and salience Perception and inferenceLearning and memory The recognition density and its sufficient statistics Mean-field approximation: Laplace approximation:

Backward predictions Forward prediction error Perception and message-passing Synaptic plasticitySynaptic gain

Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong and categorization Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

Synthetic song-birds SyrinxVocal centre Time (sec) Frequency Sonogram

prediction and error time hidden states time Backward predictions Forward prediction error Causal states time (bins) Recognition and message passing stimulus time (seconds)

Perceptual categorization Frequency (Hz) Song A time (seconds) Song B Song C ABCABC time (seconds)

Generative models of birdsong: sequences of sequences Syrinx Neuronal hierarchy Time (sec) Frequency (KHz) sonogram Kiebel et al

Frequency (Hz) percept Frequency (Hz) no structural priors time (seconds) Frequency (Hz) no dynamical priors LFP (micro-volts) LFP LFP (micro-volts) LFP peristimulus time (ms) LFP (micro-volts) LFP Simulated lesion studies: a model for false inference in psychopathology?

Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

prediction From reflexes to action action dorsal root ventral horn True dynamics Generative model

From reflexes to action Jointed arm Movement trajectory Descending sensory prediction error visual input proprioceptive input

Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

Energies sensory prediction error sensory surprise surprise free-energy complexity How do policies minimise entropy? perceptual divergence Path integrals sensory entropy entropy Under ergodic assumptions free-action perceptionpolicy (model)action

Richard Bellman Cost-functions, value and optimal control (polices that lead to sparse distal goals) Using the Helmholtz decomposition flow ( i.e., policy) can be expressed in terms of scalar and vector potentials Where value V is proportional to negative surprise and can be defined as expected (negative) cost This means the cost-function is defined by the equilibrium density but not vice versa; this is the problem addressed by dynamic programming and reinforcement learning.

Cost-functions and attracting sets (polices with attractors) At equilibrium we have: This means maxima of the equilibrium density must have negative divergence. We can exploit this to ensure maxima lie in A, using a Langevin-based policy; where cost plays the role of dissipation Adriaan Fokker Max Planck

equations of motion Exploitation exploration Exploration and exploitation under Langevin dynamics

True equations of motion position height The mountain car problem positionsatiety The cost-function Policy (expected equations of motion) The environment

prediction and error time hidden states time position velocity Trajectory of one trial position height leaned (after 16 trials) and true potential Learning the environment With no cost (i.e., Hamiltonian dynamics)

conditional expectations time action velocity trajectories position force cost-function (priors) With cost (i.e., exploratory dynamics) Exploring & exploiting the environment

Using just the free-energy principle and a simple gradient ascent scheme, we have solved a benchmark problem in optimal control theory using a handful of learning trials. Note that we use reinforcement learning or dynamic programming. Adaptive policies and trajectories

time action position trajectories velocity satiety prediction and error time expected hidden states time Self-organisation with (happiness) dynamics on cost policy (expected flow) true flow

Thank you And thanks to collaborators: Jean Daunizeau Stefan Kiebel James Kilner Klaas Stephan And colleagues: Peter Dayan Jörn Diedrichsen Paul Verschure Florentin Wörgötter