Maz Fallah

1
Object-based competition in attention and rivalry
Maz Fallah Jude Mitchell Gene Stoner John
Reynolds
2
(No Transcript)
3
Talk Outline Object-based attention as a tool
to probe visual processing Attentive selection
of an object (a surface) superimposed on an
unattended surface human psychophysics,
ERPs Object-based selection and interocular
rivalry competitive selection across levels of
processing Object-based selection in macaque V4
4
Talk Outline Object-based attention as a tool
to probe visual processing Attentive selection
of an object (a surface) superimposed on an
unattended surface human psychophysics,
ERPs Object-based selection and interocular
rivalry competitive selection across levels of
processing Object-based selection in macaque V4
5
(No Transcript)
6
Working definition of object-based
attention Selection of a stimulus that persists
under conditions that exclude 1) selection by
a purely spatial mechanism
7
Working definition of object-based
attention Selection of a stimulus that persists
under conditions that exclude 1) selection by
a purely spatial mechanism and 2) selection
by a purely feature-based mechanism. where
feature is understood to mean a stimulus
property that is explicitly coded by neurons
selective for that property.
8
Working definition of object-based
attention Selection of a stimulus that persists
under conditions that exclude 1) selection by
a purely spatial mechanism and 2) selection
by a purely feature-based mechanism. where
feature is understood to mean a stimulus
property that is explicitly coded by neurons
selective for that property. Example
Valdes-Sosas superimposed surface paradigm
9
Talk Outline Object-based attention as a tool
to probe visual processing Attentive selection
of an object (a surface) superimposed on an
unattended surface human psychophysics,
ERPs Object-based selection and interocular
rivalry competitive selection across levels of
processing Object-based selection in macaque V4
10
Valdes-Sosa, Cobo and Pinilla, 1998
11
Valdes-Sosa, Cobo and Pinilla, 1998
12
Valdes-Sosa, Cobo and Pinilla, 1998
13
Valdes-Sosa, Cobo and Pinilla, 1998
14
Valdes-Sosa, Cobo and Pinilla, 1998
15
Endogenous Cue
Valdes-Sosa, Cobo and Pinilla, 1998
16
Valdes-Sosa, Cobo and Pinilla, 1998 Reynolds,
Alborzian and Stoner, 2004
17
Endogenous Cue
Valdes-Sosa, Cobo and Pinilla, 1998 Reynolds,
Alborzian and Stoner, 2004
18
Endogenous Cue Eliminated
Valdes-Sosa, Cobo and Pinilla, 1998 Reynolds,
Alborzian and Stoner, 2004
19
Valdes-Sosa, Cobo and Pinilla, 1998 Reynolds,
Alborzian and Stoner, 2004
20
Conclusions from Expt 1 Consistent with
Valdes-Sosas findings, attention can select one
of two superimposed surfaces defined by
transparent motion. However, the endogenous
cue is not necessary. The first translation
acts as an exogenous cue that suppresses
processing of the other surface.
21
Experiment 2 ERPs during exogenous
surface-based selection.
22
ERPs During Exogenous Surface Selection
800
350 ms Rotation
800
800
350 ms Rotation
800
Valdes-Sosa et al 2000 Khoe, Mitchell, Reynolds
Hillyard, under review
23
ERPs During Exogenous Surface Selection
ERPs
Difference Wave
Scalp Topographies
24
Conclusions from Expt 2 Exogenous surface-based
cueing causes surface-dependent modulations N1
and C1 ERP components, with stronger responses
evoked by the cued surface. This suggests
surface based modulation of MT and, surprisingly,
V1.
25
Working definition of object-based
attention Selection of a stimulus that persists
under conditions that exclude 1) selection by
a purely spatial mechanism and 2) selection
by a purely feature-based mechanism? where
feature is understood to mean a stimulus
property that is explicitly coded by neurons
selective for that property.
26
Experiment 3 Ruling out feature-based
attention. One constant feature that
distinguishes the surfaces is color. Could we be
seeing a modulation of color-gain, as though
viewing the stimuli through a colored
filter? Methods Identical to Experiment 1, but
on half the trials, selected at random, the
surfaces were the same color.
27
(No Transcript)
28

TIME
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
Reynolds, Alborzian and Stoner, 2003 Vision
Research Mitchell, Stoner and Reynolds, 2004
Vision Research
29
(No Transcript)
30
Conclusions from Expt 3 Removal of the color
difference had no measurable effect on the cueing
effect. Selection cannot be explained as
resulting from a spatial or feature-based
mechanism, and is thus object-based.
31
Experiment 4 ERPs during exogenous
surface-based selection using identical colors.
32
(No Transcript)
33
Conclusions from Expt 4 Removal of the color
difference had no measurable effect on the ERP
modulation. Thus, suprisingly, we find evidence
of object-based selection in V1.
34
Talk Outline Object-based attention as a tool
to probe visual processing Attentive selection
of an object (a surface) superimposed on an
unattended surface human psychophysics,
ERPs Object-based selection and interocular
rivalry competitive selection across levels of
processing Object-based selection in macaque V4
35
Jude Mitchell
36
(No Transcript)
37
(No Transcript)
38
What does rivalry reflect?
Left eye
Right eye
39
What does rivalry reflect Interocular
competition?
40
What does rivalry reflect Interocular
competition?
Interocular Competition
41
What does rivalry reflect Interocular
competition?
X
Interocular Competition
42
What does rivalry reflect Interocular
competition?
X
Interocular Competition
Lee, Blake and Heeger, Journal of Vision, 2003
Blake, R., Westendorf, D, and Overton, R., 1979
What is suppressed during binocular rivalry?
Perception 9, 223 231.
43
What does rivalry reflect Interocular
competition?
Competition between stimulus representations
later in visual processing?
5
10
15
20
5
X
10
15
20
Interocular Competition
5
10
15
20
Competition in extrastriate areas that have lost
eye of origin information
44
Cue surface before Rivalry
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
?
Which Dominates?
45
Cue surface before Rivalry
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
?
Which Dominates?
46
Cue surface before Rivalry
Both Eyes, Same Image
Note Cue is binocular.
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
?
Which Dominates?
47
Cue surface before Rivalry
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
?
Which Dominates?
48
Switch to rivalry
Remove one image from each eye.
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
?
Which Dominates?
49
Switch to rivalry
Remove one image from each eye.
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
?
Which Dominates?
50
Switch to rivalry
Remove one image from each eye.
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
Variable period of rivalrous viewing.
?
Which Dominates?
51
Switch to rivalry
Remove one image from each eye.
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
Variable period of rivalrous viewing.
?
Which Dominates?
52
Surface Reported as Dominant
Switch to rivalry
100
90
Both Eyes, Same Image
80
70
60
Reported
50
Cue One Surface
40
30
Delay (150ms)
20
10
Rivalry Between Surfaces
0
0
500
1000
1500
2000
Dichoptic Viewing Period (ms)
?
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
53
Surface Reported as Dominant
Switch to rivalry
100
90
Both Eyes, Same Image
Cued Surface
80
70
60
Reported
50
Neither
Cue One Surface
40
30
Delay (150ms)
20
10
Uncued Surface
Rivalry Between Surfaces
0
0
500
1000
1500
2000
Dichoptic Viewing Period (ms)
?
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
54
Surface Reported as Dominant
Switch to rivalry
100
90
Both Eyes, Same Image
Cued Surface
80
70
60
Reported
50
Neither
Cue One Surface
40
30
Delay (150ms)
20
10
Uncued Surface
Rivalry Between Surfaces
0
0
500
1000
1500
2000
Dichoptic Viewing Period (ms)
?
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
55
Surface Reported as Dominant
Switch to rivalry
100
90
Both Eyes, Same Image
Cued Surface
80
70
60
Reported
50
Neither
Cue One Surface
40
30
Delay (150ms)
20
10
Uncued Surface
Rivalry Between Surfaces
0
0
500
1000
1500
2000
Dichoptic Viewing Period (ms)
?
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
56
Surface Reported as Dominant
Switch to rivalry
100
90
Both Eyes, Same Image
Cued Surface
80
70
60
Reported
50
Neither
Cue One Surface
40
30
Delay (150ms)
20
10
Uncued Surface
Rivalry Between Surfaces
0
0
500
1000
1500
2000
Dichoptic Viewing Period (ms)
?
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
57
Double-Translation Task during rivalry
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
Second Translation Judgment
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
58
Second Judgment Accuracy Rivalrous Surfaces
Double-Translation Task during rivalry
100
90
Both Eyes, Same Image
Cued Surface
80
70
60
Correct
50
Cue One Surface
40
30
Uncued Surface
Delay (150ms)
20
10
Rivalry Between Surfaces
0
0
500
1000
1500
2000
ISI (ms)
Second Translation Judgment
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
59
Second Judgment Accuracy Rivalrous Surfaces
Double-Translation Task during rivalry
100
90
Both Eyes, Same Image
Cued Surface
80
70
60
Correct
50
Cue One Surface
40
30
Uncued Surface
Delay (150ms)
20
10
Rivalry Between Surfaces
0
1500
2000
0
500
1000
ISI (ms)
Second Translation Judgment
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
60
Second Judgment Accuracy Rivalrous Surfaces
Double-Translation Task during rivalry
100
90
Both Eyes, Same Image
Cued Surface
80
70
60
Correct
50
Cue One Surface
40
30
Uncued Surface
Delay (150ms)
20
10
Rivalry Between Surfaces
0
0
500
1000
1500
2000
ISI (ms)
Second Translation Judgment
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
61
Double-Translation Task during rivalry
So, using a binocular cue, we have bypassed
interocular competition to cause the cued
surface to be dominant during rivalry. But,
interocular competition presumably was activated
by the switch to dichoptic viewing. Did it
influence the selection at that point?
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
Rivalry Between Surfaces
Second Translation Judgment
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
62
Double-Translation Task
in Monocular Transparency
So, using a binocular cue, we have bypassed
interocular competition to cause the cued
surface to be dominant during rivalry. But,
interocular competition presumably was activated
by the switch to dichoptic viewing. Did it
influence the selection at that point?
Both Eyes, Same Image
Cue One Surface
Delay (150ms)
Transient, but NO RIVALRY
Second Translation Judgment
?
Mitchell, Stoner and Reynolds 2004 Nature
Which Dominates?
63
Second Judgment Accuracy Transparent Surfaces
Double-Translation Task
in Monocular Transparency
100
90
Both Eyes, Same Image
Cued Surface
80
70
60
Correct
50
Cue One Surface
40
30
Uncued Surface
Delay (150ms)
20
10
Transient, but NO RIVALRY
0
0
500
1000
1500
2000
ISI (ms)
Second Translation Judgment
?
Which Dominates?
64
Second Judgment Accuracy Transparent Surfaces
Double-Translation Task
in Monocular Transparency
100
90
Both Eyes, Same Image
Cued Surface
80
70
60
Correct
50
Cue One Surface
40
30
Uncued Surface
Delay (150ms)
20
10
Transient, but NO RIVALRY
0
0
500
1000
1500
2000
ISI (ms)
Second Translation Judgment
?
Which Dominates?
65
Second Judgment Accuracy Transparent Surfaces
100
90
Cued Surface
80
70
60
Correct
50
40
30
Uncued Surface
20
10
0
0
500
1000
1500
2000
ISI (ms)
66
Second Judgment Accuracy Transparent Surfaces
100
Although both trials were identical up the the
switch to rivalry or monocular transparency, the
effect of the cue is delayed and longer lasting
in rivalry.
90
Cued Surface
80
70
60
Correct
50
40
30
Uncued Surface
20
10
0
0
500
1000
1500
2000
ISI (ms)
67
Second Judgment Accuracy Transparent Surfaces
100
90
Cued Surface
80
70
60
Correct
50
40
30
Uncued Surface
20
10
Sole difference whether both images were deleted
from one eye or one from each.
0
0
500
1000
1500
2000
ISI (ms)
68
Second Judgment Accuracy Transparent Surfaces
100
90
Cued Surface
80
70
60
Correct
50
40
30
Uncued Surface
20
10
This information is only available to neurons
with eye- of-origin, which mediate interocular
competition.
0
0
500
1000
1500
2000
ISI (ms)
69
Second Judgment Accuracy Transparent Surfaces
100
Timing difference must involve neurons with eye
of origin Information presumably the same
neurons that mediate interocular competition.
90
Cued Surface
80
70
60
Correct
50
40
30
Uncued Surface
20
10
This information is only available to neurons
with eye- of-origin information.
0
0
500
1000
1500
2000
ISI (ms)
70
Talk Outline Object-based attention as a tool
to probe visual processing Attentive selection
of an object (a surface) superimposed on an
unattended surface human psychophysics,
ERPs Object-based selection and interocular
rivalry competitive selection across levels of
processing Object-based selection in macaque V4
71
Maz Fallah
72
c.f. Yantis and Jonides
73
c.f. Yantis and Jonides
74
Delayed Onset Paradigm
First Surface Appears
TIME
75
Delayed Onset Paradigm
Second (cued) Surface Appears
(ISI)
TIME
76
Delayed Onset Paradigm
Cued Surface Translates
(ISI)
TIME
77
Delayed Onset Paradigm
Masking Rotation
(ISI)
TIME
78
Delayed Onset Paradigm
Cued Surface Translates
TIME
Uncued Surface Translates
79
Delayed Onset Response
16
60 Neurons
14
Effect of Cueing
12
10
Spike Rate (sp/s)
8
6
4
2
0
0
100
200
300
400
500
600
700
Time (ms)
80
Strength of the Cueing Effect
P
NP
Cued Surface
Uncued Surface
81
Cueing then Shifting the Stimuli into the
Receptive Field
10 º/s
10 º
Fixation Point
RF
Delayed Onset Trial
82
Cueing then Shifting the Stimuli into the
Receptive Field
10 º/s
10 º
Fixation Point
RF
Delayed Onset Trial
83
Cueing then Shifting the Stimuli into the
Receptive Field
10 º/s
10 º
Fixation Point
RF
Delayed Onset Trial
84
Selection Occurs Outside the RFand persists as
stimuli move into RF
3
Mean 0.51
Median 0.28
2
Num of Neurons
1
0
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Cued Surface
Uncued Surface
Normalized C.I.
85
Experiment Summary Delayed onset of one of
two surfaces biases responses in favor of the
delayed surface, and this advantage is
maintained, by as yet mysterious mechanisms, as
the two superimposed surfaces move through space,
successively activating populations of neurons.
86
Overall Summary Talk 1 The visual system
is modular, with more detailed retinotopy at
early stages. Neuronal response properties
often follow from the response properties of the
inputs (parasol/midget -gt M P pathways -gt
motion and color selectivity) However,
wholistic object properties can be found in
single-unit responses (e.g. MT bistable
cylinder) The mechanisms by which the visual
system reconstructs object properties are largely
unknown.
87
Overall Summary Talk 2 The visual system
is limited in capacity Therefore, neural
mechanisms must exist to select out behaviorally
relevant stimuli Lesion studies demonstrate that
extrastriate cortex is critically involved in
selecting behaviorally relevant stimuli from
among distracters. Single-unit recording
studies show that competitive mechanisms in
extrastriate cortex mediate selection of relevant
stimuli.
88
Overall Summary Talk 2 Attentional
modulation is similar in several important ways
to contrast-modulation in primary visual cortex
(multiplicative scaling of tuning, biased
competitive interactions, increased contrast
sensitivity with attention) This suggests that
the circuits in primary visual cortex are echoed
at later stages and play a role in attentional
selection. These circuits can be modulated by
feedback from the fronto-parietal control network
described by Dr. Yantis (FEF microstimulation) Exo
genous cueing causes selection which is neither
space-based nor feature based, and is thus
object-based.
89
Overall Summary Talk 3 Exogenous cueing
causes selection which is neither space-based nor
feature based, and is thus object-based. This
selection modulates the N1 and C1 components of
the ERP, suggesting modulation of MT and V1.
Surface-based selection modulates interocular
competition during rivalry.
90
Overall Summary Talk 3 Delayed onset of
one of two superimposed surfaces biases V4
neurons to respond preferentally to of the
delayed surface (cf Yantis and Jonides) This
advantage is maintained, by as yet mysterious
mechanisms, as the two superimposed surfaces move
through space, successively activating
populations of neurons. The local mechanisms by
which feature-selective neurons interact to
encode a coherent object are unknown. Studies
of object-based attention in psychophysics,
imaging, and at the single-neuron level hold the
promise of elucidating this integrative process.