Motor adaptation and the timescales of memory

1
Motor adaptation and the timescales of
memory Reza Shadmehr Johns Hopkins School of
Medicine
Konrad Koerding
Maurice Smith
Haiyin Chen
Jun Izawa
Dave Zee
Wilsaan Joiner
Tushar Rane
2
(No Transcript)
3
The brain predicts the sensory consequences of
motor commands
Duhamel et al. Science 255, 90-92 (1992)
4
What we sense depends on what we
predicted Wolpert et al. (1995)
State change
force
Body part
muscles
Motor commands
Sensory system Proprioception Vision Audition
Measured sensory consequences
Predicted sensory consequences
Forward model
5
Saccade adaptation gain decrease
5
Eye Position (deg)
5
10
Eye Position (deg)
McLaughlin 1967
6
Saccade adaptation gain decrease
5
Eye Position (deg)
5
10
Eye Position (deg)
McLaughlin 1967
7
Savings when adaptation is followed by
de-adaptation, motor system still exhibits recall
Kojima et al. (2004) J Neurosci 247531.
8
Offline learning with passage of time and
without explicit training, the motor system still
appears to learn
_


Result 2 Following changes in gain and a period
of darkness, monkeys exhibit a jump in memory.
Kojima et al. (2004) J Neurosci 247531.
9
Adaptation as concurrent learning in multiple
systems A fast learning system that forgets
quickly A slow learning system that hardly forgets
prediction
Prediction error
Learning
Smith et al. PLOS Biology, 2006
10
Savings de-adaptation may not erase adaptation
11
Offline learning Passage of time has asymmetric
affects on the fast and slow systems
12
Spontaneous recovery is also observed in reach
adaptation
Errors clamped to zero
Smith et al. PLOS Biology 2006
13
The learners view about the cause of motor errors

• 1. Perturbations that can affect the motor plant
  have multiple time scales.Some perturbations are
  fast muscles recover from fatigue quickly.Some
  perturbations are slow recovery from disease may
  be slow.
• Faster perturbations are more variable (have more
  noise).
• The error that we observe is due to a
  contribution from all possible perturbations.
• The problem of learning is one of credit
  assignment when I observe an error, what is the
  time-scale of this perturbation?

Koerding, Tenenbaum, Shadmehr, unpublished
14
The Bayesian learners interpretation of motor
error
15
Savings de-adaptation does not washout the
adapted system
Spontaneous recovery
Koerding, Tenenbaum, Shadmehr, unpublished
16
Characteristics of long-term motor memory
Data from Robinson et al. J Neurophysiol 2006
Bayesian Learner
Koerding, Tenenbaum, Shadmehr, unpublished
17
Adapting with uncertainty
Motor system is disturbed by processes that have
various timescale (fatigue vs. disease). Credit
assignment of error depends on uncertainty
regarding what is the timescale of the
disturbance. Prediction When there are actions
but the sensory consequences cannot be observed,
states decay at various rates, but uncertainty
grows. Increased uncertainty encourages learning.
18
Adapting with uncertainty two predictions
Sensory deprivation ? Faster subsequent rate of
learning. Example A subject that spends a bit of
time in the dark will subsequently learn faster
than a subject that spends that time with the
lights on. Why In the dark, uncertainty about
state of the motor system increases. Longer
inter-stimulus interval ? Better
retention. Example A subject that trains on n
trials with long ITI will show less forgetting
than one that trains on the same n trials with
short ITI. Why events that take place spaced in
time will be interpreted as having a long
timescale.
19
Summary
A prediction error causes changes in multiple
adaptive systems. Some are highly responsive to
error, but rapidly forget. Others are poorly
responsive to error but have high retention.
This explains savings and spontaneous recovery.
Maurice Smith
Fast and slow adaptive processes arose because
disturbances to the motor system have various
timescales (fatigue vs. disease). When faced
with error, the brain faces a credit assignment
problem what is the timescale of the
disturbance? To solve this problem, the brain
likely keeps a measure of uncertainty about the
timescales.
20
What are some of the holes in these ideas?

• Internal models are supposed to help us control
  our movements in real-time. What are these fast
  and slow systems learning and how does that
  learning affect real-time control of movements?
• Can we say anything about the neural structures
  that might be responsible for computing internal
  models?

21
Emo Todorov Motor command generator as an
optimal controller
State change
Goal selector
Motor command generator
Body environment
Belief about state of body and world
Predicted sensory consequences
Forward model
Integration
Sensory system Proprioception Vision Audition
Measured sensory consequences
22
Motor command generator as a stochastic optimal
controller Todorov (2005)
Actual state of the system (eye state, target
state, etc.)
Signal dependent motor noise
What we can observe about the state of the system
Signal dependent sensory noise
Cost to minimize
Feedback control policy
Belief about state
23
eye velocity
The mathematical framework allows one to produce
detailed trajectory of movements. In the target
jump paradigm, error is a difference between
predicted and actual sensory consequences of
oculomotor commands. Therefore, the forward model
must adapt. But if that adaptation is not
precisely matched by the motor command generator,
the result will be sub-optimal saccades.
deg/sec
Saccade size
5
10
15
30
40
50
Time (sec)
State change
Body environment
Goal specification
Motor command generator
Belief about state of body and world
Predicted sensory consequences
Forward model
Integration
Sensory system
Measured sensory consequences
24
The direct and indirect output pathways from the
superior colliculus (SC)

• Direct pathway
• SC?brainstem
• Indirect pathway
• SC?cerebellum?brainstem

25
Cross-axis saccade adaptation
Equal rates of learning in the controller and the
forward model saccades remain straight
Learning in the forward model only saccades
become curved
T2
fixation
T1
26
Cross-axis saccade adaptation Experiment
design (In complete darkness, with search coil
lenses on the eyes)
Chen, Joiner, Zee, Shadmehr (unpublished)
27
Characteristics of primary saccades during
adaptation
T2
5o
15o
T1
Chen, Joiner, Zee, Shadmehr (unpublished)
28
Curvature of primary saccades quantified through
chord slopes
Chen, Joiner, Zee, Shadmehr (unpublished)
29
Saccade curvature suggests that errors cause
rapid adaptation in the forward model
The observation that saccades become curved, and
therefore sub-optimal, is a reflection of a
neural system that adaptively computes sensory
consequences of motor commands, and corrects the
motor commands as they are produced. The
forward model (indirect pathway) appears to adapt
much more quickly than the controller (direct
pathway).
State change
Body environment
Goal specification
Motor command generator
Belief about state of body and world
Predicted sensory consequences
Forward model
Integration
Sensory system
30
Summary
In saccades and reaching, performance is guided
by internal models that adapt at multiple
timescales A fast learning system that has poor
retention. A slow learning system that hardly
forgets. The observation that saccades become
curved, and therefore sub-optimal, is a
reflection of a neural system that adaptively
computes sensory consequences of motor commands,
and corrects the motor commands as they are
produced. The forward model (indirect pathway)
appears to adapt much more quickly than the
controller (direct pathway).
Haiyin Chen
Dave Zee
Wilsaan Joiner
31
What are some of the holes in these ideas?

• If learning of forward models (indirect pathway)
  is faster than the controller (direct pathway),
  the result is a sub-optimal system. Most of our
  movements appear optimal. What guides learning
  in the direct pathway so that we eventually
  become optimal?
• If we learn as a Bayesian, we keep a measure of
  uncertainty about what we know. Does the
  uncertainty in the internal model affect our
  control policies (direct pathway)?

32
Learning in the direct pathway finding a better
control policy in the high jump task
33
(No Transcript)
34
The optimal control policy To maximize
probability of arriving at target in time, I
should minimize my motor commands near the end of
the movement. Over compensate for the forces
early, let the robot bring you back.
Izawa, Rane, Donchin, Shadmehr (unpublished)
35
Izawa, Rane, Donchin, Shadmehr (unpublished)
36
In performing an action, the motor commands that
we generate should depend on our confidence
(uncertainty) in our models.
37
Traditional stochastic optimal control
Stochastic optimal control with model uncertainty
38
Stochastic optimal control with model
uncertainty Predictions
Izawa, Rane, Donchin, Shadmehr (unpublished work)
39
People learn policies that depend on their model
uncertainty Overcompensate only if you are
certain of the world
N6
Izawa, Rane, Donchin, Shadmehr (unpublished work)
40
Overview Computational problem of motor control
Motor control is about solving two distinct
problems Learning a control policy (direct
pathway). Learning a forward model (indirect
pathway). Motor learning is at multiple
timescales A fast learning system that has poor
retention. A slow learning system that hardly
forgets. The forward model (indirect pathway)
adapts much more quickly than the controller
(direct pathway).
Jun Izawa
Maurice Smith
Haiyin Chen
41
What are some of the holes in these ideas?

• In saccade adaptation, nothing happened to the
  body it was the target that was behaving
  strangely. When there is error, how does the
  brain distinguish between changes in the body vs.
  changes in the world? This is a second credit
  assignment problem.
• What is the error signal that guides learning of
  control policies?
• Are the direct and indirect pathways
  computational pathways or neural pathways?

42
Reversible disruption of cerebellar pathways in
humans
Motor cortex
Corticospinal tract
Sherwin Hua
43
Deep Brain Stimulation
1.5 mm electrode is implanted in the thalamus and
connected via subcutaneous wires to a stimulator.
The subcutaneous stimulator and battery.
Parameter settings can be adjusted via an
external device.
Fred Lenz
44
Stimulation of VL thalamus improves tremor but
impairs adaptation
Chen et al. Cerebral Cortex, 2006
45
EMG patterns during reach adaptation
Movement onset
Thoroughman Shadmehr, J Neurosci, 1999
46
Neural correlates of motor learning in the VL
thalamus
47
(No Transcript)
48
Behavioral performance
Adaptation level was low
49
Recording sites and neural responses

• Sites attempted recording .. 105
• Sites successfully recorded units . 58 (55)
• Units with more than 60 trials 61
• Vim.35
• Vim-Vop border12
• Voa/Vop 14
• Single units .. 16 (26)
• Movement related units . 36 (59)
• Vim.21
• Vim-Vop border5
• Voa/Vop10
• Units showed direction selectivity . 18 (50)
• Vim.11
• Vim-Vop border1
• Voa/Vop.6

50
Adaptation induces change in firing pattern
before movement onset
target
Vmax
stop
hold/wait
51
Conclusion and speculations

• The cerebellum appears to be a critical structure
  for motor adaptation. Is this the place where
  forward models are formed?
• Speculation cerebellar cortex may represent the
  fast system, with the cerebellar nuclei
  representing the slow system. Prediction
  cerebellar patients may learn slowly, but they
  will also forget slowly.
• Learning control policies depends on reward
  prediction errors.Is the basal ganglia the
  structure crucial for learning control policies?
• Challenge ahead To look for behavior and
  neural signatures of control policies and forward
  models in healthy individuals and patients with
  motor disorders.

52
The neural basis of motor adaptation
Cerebellar degeneration impaired adaptation of
reaching
Huntingtons disease (HD) patients showed no
deficit in adaptation
Smith and Shadmehr, J Neurophysiology 2005
53
Visual rotation adaptation