Image%20Stabilization%20by%20Bayesian%20Dynamics

1
Image Stabilization by Bayesian Dynamics

• Yoram Burak
• Sloan-Swartz annual meeting, July 2009

2
What does neural activity represent?
In Bayesian models probabilities
Accumulated evidence in area LIP Shadlen and
Newsome (2001)
Direction of motion single, static variable
3
What does neural activity represent?
In Bayesian models probabilities
Accumulated evidence in area LIP Shadlen and
Newsome (2001)
Direction of motion single, static variable
What about multi-dimensional, dynamic quantities?
4
Foveal vision and fixational drift
5
Foveal vision and fixational drift
Fixational drift is large in the fovea
- between micro-saccades - 20 receptive fields
- between spikes (100 Hz) - 2-4 receptive fields
!
Image from X. Pitkow
cone separation 0.5 arcmin
6
Foveal vision and fixational drift
Fixational drift is large in the fovea
- between micro-saccades - 20 receptive fields
- between spikes (100 Hz) - 2-4 receptive fields
!
Image from X. Pitkow
cone separation 0.5 arcmin
Downstream areas require knowledge of trajectory
to interpret spikes
7
Joint decoding of image and position
Bayesian
vs.
N x 2 probabilities
Discrimination task
X. Pitkow et al, Plos Biology (2007)
positions
8
Joint decoding of image and position
Bayesian
vs.
N x 2 probabilities
Discrimination task
X. Pitkow et al, Plos Biology (2007)
positions
30 x 30 binary pixels
Unconstrained image
N x 2900 probabilities
9
Joint decoding of image and position
Bayesian
vs.
N x 2 probabilities
Discrimination task
X. Pitkow et al, Plos Biology (2007)
positions
30 x 30 binary pixels
Unconstrained image
N x 2900 probabilities
Can the brain apply a Bayesian approach to this
problem?
10
Can the brain apply a Bayesian approach to this
problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
11
Can the brain apply a Bayesian approach to this
problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
12
Decoding strategy
Factorized representation
Discards information about correlations
13
Decoding strategy
Factorized representation
Discards information about correlations
Update dynamics
minimize DKL
evidence, diffusion
Exact if trajectory is known.
14
Decoding strategy
Factorized representation
Discards information about correlations
Update dynamics
minimize DKL
evidence, diffusion
Exact if trajectory is known.
Retinal encoding model
evidence - Poisson spiking (rate ?1 for on
pixels, ?0 for off) diffusion - Random walk
(diffusion coefficient D)
15
Decoding strategy
Factorized representation
Discards information about correlations
Neural Implementation -
Two populations where , what
For 30 x 30 pixels N 2900 ? N 900
quantities.
16
Update rules
Update of what neurons
Ganglion cells
multiplicative gating
What
nonlinearity
Where
17
Update rules
Update of what neurons
Ganglion cells
multiplicative gating
What
nonlinearity
Where
Update of where neurons
Ganglion cells
multiplicative gating
Where
diffusion
What
18
Demo
m x m binary pixels
image
retina
2d diffusion (D)
Poisson spikes 100 Hz (on), 10 Hz (off)
Decoder
19
Demo
20
Can the brain apply a Bayesian approach to this
problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
21
Performance
Performance degrades with larger D (and smaller
?)
accuracy
Convergence time s
D
D
22
Performance
Faster and more accurate for larger images
m 5, 10, 30, 50, 100
accuracy
Convergence time s
D
D
23
Demo
24
Performance
Faster and more accurate for larger images
m 5, 10, 30, 50, 100
accuracy
Convergence time s
D
D
25
Performance
Faster and more accurate for larger images
m 5, 10, 30, 50, 100
accuracy
Convergence time s
D
D
26
Performance
Faster and more accurate for larger images
m 5, 10, 30, 50, 100
accuracy
Convergence time s
D
D
27
Performance
scales with linear image size m
accuracy
Convergence time s
D/m
D/m
m x m pixels
28
Performance
scales with linear image size m
D
accuracy
Convergence time s
D/m
D/m
m x m pixels
Analytical scaling
29
Performance
Performance improves with image size. Success
for images 10 x 10 or larger Prediction for
psychophysics Degradation in high acuity tasks
when visual scene contains little background
detail.
30
Temporal response of Ganglion cells
Common view fixational motion important to
activate cells, due to biphasic response
f(t)
50 ms
t
Temporal response makes decoding much more
difficult.
Non-Markovian
Need history
31
Temporal response of Ganglion cells
Approach Choose decoder that is Bayes optimal if
the trajectory is known.
Ganglion
accuracy
Convergence time s
Where
What
D
D
history dependent decoder / naive decoder
filtered trajectory
32
Temporal response of Ganglion cells
Is fixational motion beneficial?
Known trajectory , perfect inhibitory balance
Convergence time s
D
Optimal D - order of magnitude smaller than
biological value
33
Can the brain apply a Bayesian approach to this
problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
34
Network architecture
Each ganglion cell innervates multiple what
where cells (spread 10 arcmin)
Reciprocal, multiplicative gating
Ganglion
Where
What
35
Activity
What neurons
Slow dynamics, evidence accumulation
Where neurons
Fewer. Highly dynamic activity Tonic, sparse in
retinal stabilization conditions.
36
Activity
What neurons
Slow dynamics, evidence accumulation
Where neurons
Fewer. Highly dynamic activity Tonic, sparse in
retinal stabilization conditions.
Where in the brain?
Monocular
If so, suggests LGN or V1
LGN?
Modulatory inputs to relay cells (gating?)
V1?
Lateral connectivity in where network, Increase
in number of neurons.
37
Summary
Strategy for stabilization of foveal vision
Factorized Bayesian approach to multi-dimensional
inference
38
Summary
Strategy for stabilization of foveal vision
Factorized Bayesian approach to multi-dimensional
inference
Explicit representation of stabilized image
What and where populations
39
Summary
Strategy for stabilization of foveal vision
Factorized Bayesian approach to multi-dimensional
inference
Explicit representation of stabilized image
What and where populations
Good performance at 1 arcmin resolution Problem
is easier for large images, for coarser
reconstruction
40
Summary
Strategy for stabilization of foveal vision
Factorized Bayesian approach to multi-dimensional
inference
Explicit representation of stabilized image
What and where populations
Good performance at 1 arcmin resolution Problem
is easier for large images, for coarser
reconstruction
Network architecture Many-to-one inputs from
retina, multiplicative gating (what/where)
41
Acknowledgments
Uri Rokni Haim Sompolinsky Markus Meister
Special thanks - the Swartz foundation