Magnetoencephalogram (MEG) correlated with perceptual binding of color and motion

1
Magnetoencephalogram (MEG) correlated with
perceptual binding of color and motion

• Kaoru Amano1, Shinya Nishida2, and Tsunehiro
  Takeda1

1. University of Tokyo 2. NTT Communication
Science Laboratories
2
Color motion asynchrony
Perception
Stimulus
Stimulus synchrony (SOA 0 ms)
Perceptual asynchrony (When color reversals and
motion direction reversals are physically in
phase, they are not perceived to be in synchrony.)
250 ms
Stimulus asynchrony (SOA 100 ms)
Perceptual synchrony (When motion directions
reversal precedes color reversals by about 100
ms, they are perceived to be in synchrony)
250 ms
SOA
Moutoussis Zeki (1997)
3
Purpose

• To find the neural activities correlated with
  perceptual synchrony and binding of color and
  motion
• Comparison of MEGs between the condition of
  perceptual synchrony (SOA100 ms) and that of
  perceptual asynchrony (SOA0 ms)
• Averaged waveforms
• Wavelet analysis of time-frequency domain
• Psychophysical experiments of the perceptual
  synchrony and binding of color and motion
• To test psychophysically-proposed models, by
  seeing
• The difference in latency of motion response and
  color response
• The effect of alternation rate on MEG responses
• The effect of temporal structure on MEG responses

4
Visual stimulus

• Color
• Test (4 s) Color was reversed between red and
  green at 2Hz
• Pretest (1 s) a green pattern
• Always stationary
• Motion
• Test (4 s) Motion direction was reversed between
  expansion and contraction at 2Hz
• Pretest (1 s) a stationary pattern
• Always green
• Color Motion (SOA 0 ms)
• Test (4 s) Color and motion direction were
  reversed at 2Hz with no physical delay
  (perceptual asynchrony)
• Pretest (1 s) a stationary green pattern
• Color Motion (SOA 100 ms)
• Test (4 s) Color and motion direction were
  reversed at 2Hz with a color delay of 100ms
  (perceptual synchrony)
• Pretest (1 s) a stationary green pattern

color motion
SOA

stationary
Cont.

Exp.
Exp.
0.5 s
1 s
MEG measurement
5 s
trigger

• Concentric half rings
• SF 1.1 cycle/deg, contrast50
• Red or Green
• Stationary or Expantion/Contraction
• velocity 3 deg/s

5
MEG recording and analysis

• Ten male subjects were employed.
• MEG recording
• Whole head MEG system (Yokogawa, PQ244OR) with
  230 axial-z sensors and 70 x 3 vector sensors
• Sampling rate 625 Hz, Filter 0.3-200 Hz
• The number of average 100 times
• Pre-trigger 1000 ms, post-trigger 4000 ms
• Wavelet analysis
• Sensor output in each trial was convolved with
  complex Morlet wavelets and was averaged across
  all trials (Tallon-Baudry, 1996).
• The time varying energy for each frequency was
  corrected by the pre-trigger interval of 500 ms.
• The results of right or left occipital channels
  were averaged (34 channels for each).
• Dipole source localization
• 230 axial-z sensors over the whole head were used
  for the estimation.
• Analysis was conducted for averaged MEG filtered
  at gamma band (20-60 Hz).
• Goodness of Fit (GOF) should be larger than 80 .

6
Psychophysical experiment
Synchrony task
Binding task

Identify the color presented during expanding
motion
Tell whether color and motion were perceived to
be in phase
Motion fast
Color fast
Motion fast
Color fast
In-phase response ()
Red response ()
Averaged across all subjects (n10)
Color delay SOA (ms)
Color delay SOA (ms)
More successful binding of color and motion for
SOA100 ms than for SOA0 ms.
More synchronous perception of color and motion
for SOA100 ms than for SOA0 ms.
7
Interactions of averaged waveforms
Waveform interaction, defined by
ColorMotion-(ColorMotion), averaged
between 1000 and 3750 ms and across occipital 68
sensors and subjects (n10)
Color
Motion
MEG amplitude (fT)
ColorMotion (SOA 0 ms)
MEG amplitudes (fT)
ColorMotion (SOA 100 ms)
Latency (ms)

• Peaked response to each stimulus change (every
  250 ms) were found except for motion condition.
  
• There was no difference in the magnitude of
  waveform interaction between the condition of
  perceptual asynchrony (SOA0ms) and the condition
  of perceptual synchrony (SOA100 ms).

8
Interactions in time-frequency domain
Frequency domain interactions, defined by
colormotion-(motioncolor), averaged
between 1000 and 3750 ms and across occipital 34
left sensors and subjects (n10)
Color
Motion
Frequency (Hz)
ColorMotion (SOA 0 ms)
TF Energy
ColorMotion (SOA 100 ms)
20-25
30-35
40-45
50-55
25-30
35-40
45-50
55-60
Latency (ms)
Frequency band (Hz)

• Interactions in 20-40Hz were larger under the
  condition of perceptual synchrony (SOA100ms)
  than under the condition of perceptual asynchrony
  (SOA0ms).
• The effect was significant for 30-35 Hz in left
  hemisphere (p0.034)

9
Dipole source localization
Color,Color Motion (without filter)
Color Motion (20-60Hz)
168 ms
3120 ms

• Dipoles were estimated in the similar visual area
  for Color and ColorMotion with both SOAs.
• Dipole estimation for motion direction reversal
  was not successful because of the low S/N.

• Magnetic field maps of 6/10 subjects indicated
  the activities in the occipital areas
• Their dipoles were mainly estimated around
  calcarine sulcus.
• The dipole amplitudes were generally larger under
  the condition of perceptual synchrony than under
  the condition of perceptual asynchrony.

10
Dependency of temporal structure
Enhanced 20-40Hz responses might be correlated
not with perceptual synchrony but with physical
asynchrony

• Perceptual asynchrony depends on the temporal
  structure of the stimuli (first-order change
  versus second-order change) rather than the
  attribute type (color versus motion).
• C1P2 Motion delay (color motion asynchrony)
• C1P1 or C2P2 No delay
• C2P1 Color delay

Nishida and Johnston, 2002
Using C1P1 or C2P2 stimulus will make it clear
whether enhanced gamma responses are correlated
with perceptual synchrony or physical asynchrony
11
First- and second-order change of color and
position
Nishida and Johnston, 2002
First-order change (transitions)
C1P1 (no delay)
color
Defined by two points in time
Position
C1P2 (motion delay)
C2P1 (color delay)
Second-order change (turning point)
color
Defined by three points in time
C2P2 (no delay)
Position (Motion)
12
Psychophysical experiment (C1P1)
Synchrony task
Binding task

Identify the color when the rings were at the
outward position
Tell whether color change and position change
were perceived to be in phase
Motion fast
Color fast
Motion fast
Color fast
Red response ()
In-phase response ()
Color delay SOA (ms)
Color delay SOA (ms)
Averaged across all subjects (n6)
More synchronous perception of color and motion
for SOA0 ms than for SOA100ms.
More successful binding of color and motion for
SOA0ms than for SOA100ms.
13
Interactions in waveforms and time-frequency
domain (C1P1)
Waveform interactions, defined by
C1P1-(C1P1), averaged between 1000 and
3750 ms and across occipital 68 sensors and
subjects (n6)
Frequency domain interactions, defined by
C1P1-(C1P1), averaged between 1000 and
3750 ms and across occipital 68 sensors and
subjects (n6)
TF Energy
MEG amplitudes (fT)
20-25
30-35
40-45
50-55
25-30
35-40
45-50
55-60
Frequency band (Hz)

• Interactions of averaged waveforms were very
  similar between the two SOA conditions.
• Interactions in 20-30Hz were larger under the
  condition of perceptual synchrony (SOA 0 ms).

14
Psychophysical experiment (C2P2)
Synchrony task
Binding task
Identify the color increasing during expanding
motion
Tell whether color change and motion were
perceived to be in phase
Motion fast
Color fast
Motion fast
Color fast
Green response ()
In-phase response ()
Averaged across all subjects (n6)
Color delay SOA (ms)
Color delay SOA (ms)
More synchronous perception of color and motion
for SOA0 ms than for SOA100ms.
More successful binding of color and motion for
SOA0ms than for SOA100ms.
15
Interactions in waveforms and time-frequency
domain (C2P2)
Waveform interactions, defined by
C2P2-(P2C2), averaged between 1000 and
3750 ms and across occipital 68 sensors and
subjects (n6)
Frequency domain interactions, defined by
C2P2-(C2P2), averaged between 1000 and
3750 ms and across occipital 68 sensors and
subjects (n6)
TF Energy
MEG amplitudes (fT)
20-25
30-35
40-45
50-55
25-30
35-40
45-50
55-60
Frequency band (Hz)

• Interactions of averaged waveforms were very
  similar between the two SOA conditions.
• Interactions in 20-30Hz were larger under the
  condition of perceptual synchrony (SOA 0 ms).
• The effect was significant for 20-25 Hz in right
  hemisphere and 20-25, 25-30 Hz in left hemisphere
  (p0.049, p0.032, p0.0065 respectively).

16
Discussions
Previous studies on the binding of visual
attributes
20-40 Hz responses were enhanced when C1 and P2
were perceived to be in synchrony (SOA100ms),
while 20-30 Hz responses were enhanced when C1
and P1 or C2 and P2 were perceived to be in
synchrony (SOA0ms).

• Electrophysiological studies on animals
• When two superimposed gratings that differ in
  orientation and drift in different directions are
  perceived as a single pattern (pattern motion),
  synchronization between neurons sensitive to each
  motion direction was increased (Castelo-Branco et
  al., 2000)
• Under the condition of binocular rivalry, cells
  driven by the winning eye show a significant
  correlation (Fries et al., 1997)
• EEG or MEG studies on human
• Gamma band responses were enhanced when subjects
  perceived illusory (Kaniza) triangle
  (Tallon-Baudry et al., 1996)
• Gamma band responses were enhanced when subjects
  perceived Dalmatian dog hidden in black blobs
  (Tallon-Baudry et al., 1997)

20-30Hz responses, slightly lower than the gamma
band (gt30Hz), might be correlated with perceptual
binding of color and motion
17
Models of color motion asynchrony

• Psychophysically-proposed models of color motion
  asynchrony
• Latency difference model (Moutoussis Zeki,
  1997)
• Color and motion are processed in different
  areas, and the processing time difference between
  color and motion leads to perceptual asynchrony
• Time marker model (Nishida Johnston, 2002)
• Temporal localisation judgements are made by
  using temporal markers assigned to salient
  changes
• Hybrid model (Clifford et al, 2003 Bedell et
  al., 2003)
• Separate mechanisms for temporal synchrony and
  binding

Give some suggestions on these models by
analyzing MEG amplitudes and latencies
18
Time marker model
Nishida Johnston, 2002

• Perceptual motion delay occurs only for rapid
  alternations
• Reaction times are not different between color
  and motion

Latency model can not account for color motion
asynchrony
Temporal localisation judgements are made by
using temporal markers assigned to salient changes

• Accurate temporal order judgement for slow
  alternations result from matching between first
  order changes (color change) and second order
  changes (motion direction reversal)
• Color motion asynchrony at rapid alternations
  result from matching between first-order changes
  (color change and position change)

• Color change, position change (motion)
• First-order change
• Time marker can be allocated even at rapid
  alternations
• Motion direction reversal
• Second-order change
• Time marker is difficult to be allocated at rapid
  alternations

19
Verification of the models

• The effect of alternation frequency on the color
  change (C1) response and motion direction change
  (P2) response.
• According to the time marker model, the response
  to motion reversal becomes less salient for
  higher frequencies while the saliency of color
  change does not change largely.
• Comparison of MEG amplitudes and latencies
  between color change (C1), color direction change
  (C2), position change (P1), and motion direction
  change (P2) at 2 Hz
• Are there any difference in latency between MEG
  to P2 and C1
• Does the temporal structure of stimuli affect MEG
  amplitudes and/or latency?

20
Dependency on alternation rate
NS
0.25 Hz

RMS (fT)
Peak RMS (fT)
2 Hz
0.25Hz
2Hz
0.25Hz
2Hz
RMS (fT)
color
motion
Average of peak RMS values at 250-500, 500-750 ms.
time (ms)

• Increasing alternation rate did not decrease
  color response.
• Motion response reduced at higher alternation
  rate (2Hz).

Consistent with the time marker model
21
Dependency on temporal structure Latency

• MEG latency was apparently shorter (but in fact
  longer, considering the short stimulus cycles)
  for color change (C1) than for motion direction
  change (P2).
• MEG latency was affected not by stimulus
  attribute but by temporal structure

Alternation rate 2Hz
Latency (ms)

• The latency difference between C1 and P2 is
  consistent with the latency model, and is also
  consistent with the time marker model given MEG
  primarily reflects the response to first-order
  temporal changes.
• Similarity in latency found between C1 and P1,
  and between C2 and P2 is consistent with time
  marker model, but the latency model cannot
  predict these results.

22
Dependency on temporal structure Amplitude

• MEG amplitude was smaller for motion direction
  change (P2) than for position change (P1), and
  for color direction change (C2) than for color
  change (C1).

Alternation rate 2Hz
Amplitudes (fT)
Compatible with the hypothesis that MEG primarily
reflects the response to the first-order temporal
change and the first-order change in P2 and C2
were less salient than that in P1 and C1.
23
Suggestions on models

• P2 response reduced at higher alternation rate
  (2Hz), though increasing alternation rate did not
  decrease C1 response.
• MEG latency was affected not by stimulus
  attribute (color vs. motion) but by temporal
  structure (first-order vs. second-order).

The MEG responses support the time marker model
(and hybrid models) more than the latency model.
24
Summary

• MEG responses for simultaneous alternations in
  color and motion direction were measured.
• 20-40 Hz responses were enhanced when C1 and P2
  were perceived to be in synchrony (SOA100ms),
  while 20-30 Hz responses were enhanced when C1
  and P1 or C2 and P2 were perceived to be in
  synchrony (SOA0ms).
• MEG responses were compatible with the time
  marker model of color-motion asynchrony (Nishida
  and Johnston, 2002) indicating matching of
  salient events.