2. Internal models, adaptation, and uncertainty
Reza Shadmehr
Johns Hopkins School of Medicine
Joern Diedrichsen
Siavash Vaziri
Maurice Smith
Part of the pleasure of doing research on the brain is that you can get pleasure, indeed inspiration, from observing seemingly everything events. I want to show you one such event that has been fascinating for me. Do the book on left hand task.
Ali Ghazizadeh
Konrad Koerding

3. Internal models predict the sensory consequences of motor commands
What are internal models? The best example of one comes from the oculomotor literature, where there is clear evidence that the brain predicts the sensory consequences of oculomotor commands.
Duhamel, Colby, & Goldberg Science 255, (1992)

4. Measured sensory consequences
force
State change
Motor commands
muscles
Body part
Integration
Bayesian mixture
Sensory system
Proprioception
Vision
Audition
Measured sensory consequences
Why should the brain predict the sensory consequences of movement? A number of reasons have been given through out the years, but for me the most compelling reason comes from the work of Daniel Wolpert … Despite the fact that the best evidence for IM comes in the oculomotor literature, there has been no direct test of it. To test it, we need to measure estimate of state in three conditions: via the forward model, sensory feedback alone, and when both sources of information are available.
Predicted sensory consequences
Forward model

5. Reach endpoints with respect to target
Time (msec)
Vaziri, Diedrichsen, Shadmehr, J Neurosci 2006

6. Measured sensory input
Variance in reach errors indicates an integration of the predicted and actual sensory consequence of oculomotor commands
Estimate of target location
Measured sensory input
Integration
Sensory system
Motor commands
Predicted sensory consequences
Forward model
Vaziri, Diedrichsen, Shadmehr, J Neurosci 2006

7. Measured sensory consequences
What are internal models good for?
Improve ability to sense the world. By predicting the sensory consequences of motor commands, and then integrating it with the actual sensory feedback, the brain arrives at an estimate that is better than is possible from sensation alone.
muscles
force
Body part
State change
Motor commands
Sensory system
Proprioception
Vision
Audition
Measured sensory consequences
Integration
Bayesian mixture
Now you want to transition to adaptation of internal models.
Predicted sensory consequences
Forward model

8. Saccadic target jump experiments: gain reduction
30%
Eye X
Target
Equivalent to muscles being too strong
McLaughlin 1967

9. Kojima et al. (2004) J Neurosci 24:7531.
Savings: when adaptation is followed by de-adaptation, motor system still exhibits recall
Saccade gain =
Target displacement
Eye displacement
Kojima et al. (2004) J Neurosci 24:7531.
Result 1: After changes in gain, monkeys exhibit recall despite behavioral evidence for washout. + _

10. Kojima et al. (2004) J Neurosci 24:7531.
Offline learning: with passage of time and without explicit training, the motor system still appears to learn _ + +
Result 2: Following changes in gain and a period of darkness, monkeys exhibit a “jump” in memory.
Kojima et al. (2004) J Neurosci 24:7531.

11. Motor adaptation as concurrent learning in two systems:
A fast learning system that forgets quickly
A slow learning system that hardly forgets
prediction
Prediction error
Learning
Smith, Ghazizadeh, Shadmehr PLOS Biology, 2006

12. Savings: de-adaptation may not erase adaptation
Task reversal period
re-adaptation
Trial number
Smith, Ghazizadeh, Shadmehr PLOS Biology, 2006

13. The Bayesian learner’s interpretation of prediction error
Hidden states
Context
perturbation
Slow change A
fast change

14. Offline learning: Passage of time has asymmetric affects on the fast and slow systems
Smith, Ghazizadeh, Shadmehr PLOS Biology, 2006
Task reversal period
“dark” period
re-adaptation
Trial number
Slow state
Fast state -

15. The learner’s view about the cause of motor errors
1. Perturbations that can affect the motor plant have multiple time scales. Some perturbations are fast: muscles recover from fatigue quickly. Some perturbations are slow: recovery from disease may be slow.
Faster perturbations are more variable (have more noise).
Disturbances result in error, which can be observed, but with sensory noise.
The problem of learning is one of credit assignment: when I observe a disturbance, what is the time-scale of this perturbation?
To solve this problem, the brain must keep a measure of uncertainty about each possible timescale of perturbation.
Koerding, Tenenbaum, Shadmehr, unpublished

16. Savings: de-adaptation does not washout the adapted system
Spontaneous recovery
Simulation
Koerding, Tenenbaum, Shadmehr, unpublished

17. What prediction dissociates the two models?
Model 1 (Smith et al.): Error causes changes in multiple adaptive processes. Fast adaptive processes are highly responsive to error, but quickly forget. Slowly adaptive processes respond poorly to error, but retain their changes.
Prediction: When actions are performed with zero error, states of the adaptive processes decay, but at different rates.
Model 2 (Koerding et al.): Motor system is disturbed by processes that have various timescale (fatigue vs. disease). Credit assignment of error depends on uncertainty regarding what is the timescale of the disturbance.
Prediction: When there are actions but the sensory consequences cannot be observed, states decay at various rates, but uncertainty grows. Increased uncertainty encourages learning.

18. Adapting without uncertainty
Model 1: After a period of “darkness”, there will be spontaneous recovery, but rate of re-adaptation will be the same as initial learning.
Trial number
Slow state
Fast state
Task reversal period
“dark” period
re-adaptation -
Smith, Ghazizadeh, Shadmehr PLOS Biology 2006

19. Adapting with uncertainty
Model 2: After a period of “darkness”, there will be spontaneous recovery, but the rate of re-adaptation will be faster than initial learning.
Bayesian learner
Monkey data from Kojima et al. (2004). Simulations from Koerding, Tenenbaum, Shadmehr, unpublished

20. Sensory deprivation may increase uncertainty, resulting in faster learning
Monkeys were trained each day, but between training sessions they put on dark goggles, reducing their ability to sense consequences of their own motor commands. 1
1000
2000
3000
Saccade number
Darkness
Darkness
Robinson et al. J Neurophysiol, in press

21. Adapting with uncertainty: some predictions
Sensory deprivation  Faster subsequent rate of learning.
Example: A subject that spends a bit of time in the dark will subsequently learn faster than a subject that spends that time with the lights on.
Why: In the dark, uncertainty about state of the motor system increases.
Longer inter-stimulus interval  Better retention.
Example: A subject that trains on n trials with long ITI will show less forgetting than one that trains on the same n trials with short ITI.
Why: events that take place spaced in time will be interpreted as having a long timescale.

22. Summary
Joern Diedrichsen
Siavash Vaziri
By combining the predictions of internal models with sensory measurements, the brain ends up with less noisy estimates of the environment than is possible with either source of information alone.
A prediction error causes changes in multiple adaptive systems. Some are highly responsive to error, but rapidly forget. Others are poorly responsive to error but have high retention. This explains savings and spontaneous recovery.
Ali Ghazizadeh
Maurice Smith
Fast and slow adaptive processes arose because disturbances to the motor system have various timescales (fatigue vs. disease). When faced with error, the brain faces a credit assignment problem: what is the timescale of the disturbance? To solve this problem, the brain likely keeps a measure of uncertainty about the timescales.
Konrad Koerding