Decoding Seen and Attended Edge Orientation and Motion Direction from the Human Brain Activity Measured by functional Magnetic Resonance Imaging (fMRI)

1
Decoding Seen and Attended Edge Orientation and
Motion Direction from the Human Brain Activity
Measured by functional Magnetic Resonance Imaging
(fMRI)

• Presented by Arash Ashari

• Kamitani, Y., and Tong, F. (2005). Decoding the
  visual and subjective contents of the human
  brain. Nat. Neurosci. 8, 679685.
• Kamitani, Y., Tong, F. (2006). Decoding seen and
  attended motion directions from activity in the
  human visual cortex. Current Biology, 16
  1096-1102.

2
Outline

• functional Magnetic Resonance Imaging (fMRI)
• Visual Cortex
• Orientation Decoder
• Direction Decoder
• Data Analysis
• fMRI- Accomplishments and Future works

3
Outline

• functional Magnetic Resonance Imaging (fMRI)
• Visual Cortex
• Orientation Decoder
• Direction Decoder
• Data Analysis
• fMRI- Accomplishments and Future works

4
MRI Magnetic Resonance Imaging

• Using Nuclear Magnetic Resonance (NMR)
  technology, magnetic field influence the nucleus
  of hydrogen.
• The MRI transmit Radio Frequency (RF) wave to the
  nucleus, and measures changes in magnetic field.
  
• The RF changes according to the chemical
  structure of the tissue.
• The computerized images give a detailed
  anatomical view of the organ.

• High Spatial Resolution
• Used in brain structure research and
  localization of brain-tumor

http//www.bm.technion.ac.il/courses/335014/projec
ts04/mid_term_presentations_05/Mapping_visual_cort
ex.ppt
5
fMRI Functional MRI

• The new neuroimaging method for probing the
  intact human brain.
• The technique is based on
• In neural activity an additional supply of
  oxygenated blood is delivered.
• Oxygenated Hemoglobin (Hb) is magnetically
  transparent (diamagnetic).
• Deoxygenated Hb is not transparent
  (paramagnetic).
• The change in the ratio of oxygenated to
  deoxygenated can be detected in the MR signal.
• This signaling mechanism is the Blood Oxygen
  Level-Dependent (BOLD)

6
fMRI-Characteristics

• It measures changes in the subject in real time,
  without external Indicator.
• The exam can be repeated many times, without any
  harm.
• High Spatial Resolution mm
• Relatively High Temporal Resolution Sec
• High SNR allows to measure the size of
  differences, not just their presence or absence.

7
Outline

• functional Magnetic Resonance Imaging (fMRI)
• Visual Cortex
• Orientation Decoder
• Direction Decoder
• Data Analysis
• fMRI- Accomplishments and Future works

8
Visual Cortex

• The term visual cortex refers to the primary
  visual cortex/V1 and extra-striate visual
  cortical areas such as V2, V3, V4, and V5/MT.
• The primary visual cortex, V1, receives
  information directly from the lateral geniculate
  nucleus. V1 transmits information to two primary
  pathways, called the dorsal stream and the
  ventral stream.

9
Primary Visual Pathways

• The dorsal stream begins with V1, goes through
  Visual area V2, then to the dorsomedial area and
  Visual area MT and to the posterior parietal
  cortex. The dorsal stream, sometimes called the
  "Where Pathway", is associated with motion,
  representation of object locations, and control
  of the eyes and arms.
• The ventral stream begins with V1, goes through
  visual area V2, then through visual area V4, and
  to the inferior temporal cortex. The ventral
  stream, sometimes called the "What Pathway", is
  associated with form recognition and object
  representation. It is also associated with
  storage of long-term memory.

http//en.wikipedia.org/wiki/Visual_cortex
10
Visual Perception

• It is commonly assumed that human visual
  perception is based on the neural coding of
  fundamental features, such as Orientation, Color,
  Motion and so forth.
• So it can be hypothesized that functional
  neuroimaging (fMRI) identifies brain areas that
  show robust responses to visual orientation and
  motion.

11
Outline

• functional Magnetic Resonance Imaging (fMRI)
• Visual Cortex
• Orientation Decoder
• Direction Decoder
• Data Analysis
• fMRI- Accomplishments and Future works

12
Orientation Decoder

• Subjects views one of eight possible stimulus
  orientations while activity is monitored in early
  visual areas (V1-V4 and MT) using standard fMRI
  procedures (3T MRI scanner, spatial resolution
  333 mm). For each 16-s 'trial' or stimulus
  block, a square-wave annular grating is presented
  at the specified orientation (0, 22.5, ...,
  157.5), and flashes on and off every 250 ms with
  a randomized spatial phase to ensure that there
  is no mutual information between orientation and
  local pixel intensity.

13
Orientation decoding accuracy

• fMRI activity patterns in the human visual cortex
  are sufficiently reliable to predict what
  stimulus orientation the subject is viewing on
  individual trials.
• Ensemble fMRI activity in areas V1/V2 led to
  precise decoding of which of the eight
  orientations the subject saw on individual
  stimulus trials.
• Root mean squared error (RMSE) between the true
  and the predicted orientations
• 17.9, 21.0, 22.2 and 31.2, respectively for
  subjects S1-S4

14
Orientation decoding accuracy across visual areas

• The ability to extract robust orientation
  information from ensemble fMRI activity allows us
  to compare orientation selectivity across
  different human visual areas. Orientation
  selectivity is most pronounced in early areas V1
  and V2, and declines in progressively higher
  visual areas. Unlike areas V1 through V4, human
  area MT showed no evidence of orientation
  selectivity consistent with the idea that this
  region is more sensitive to motion than to
  stimulus form.

15
Source of orientation Information

• The orientation preference of individual voxels
  on the flattened surface of left ventral V1 and
  V2 for subjects S2 and S3.
• Voxel colors depict the orientation detector for
  which each voxel provides the largest weight.

16
Mind-Reading of Attended Orientation

• The robust effects found in V1 and V2 suggest
  that top-down voluntary attention acts very early
  in the processing stream to bias
  orientation-selective signals when two competing
  stimuli are entirely overlapping.

17
Outline

• functional Magnetic Resonance Imaging (fMRI)
• Visual Cortex
• Orientation Decoder
• Direction Decoder
• Data Analysis
• fMRI- Accomplishments and Future works

18
Direction decoder

1. The decoder receives fMRI voxel intensities,
   averaged for each 16-s stimulus block, as inputs.
2. The next layer consisting of linear ensemble
   direction detectors calculates the weighed sum
   of voxel inputs.
3. Voxel weights are optimized using a statistical
   learning algorithm applied to independent
   training data, so that each detectors output
   become larger for its direction than for the
   others.
4. The direction of the most active detector is used
   as the prediction of the decoder.

19
Comparison of Orientation and Direction
Selectivity
20
Mind-Reading of Attended Direction

• Feature-based attention can alter the strength
  of direction-selective responses throughout the
  visual pathway, with top-down bias effects
  emerging at very early stages of visual
  processing. ( Attention should bias the pattern
  of neural activity to more closely resemble the
  activity pattern that would be induced by the
  attended feature alone.)

21
Outline

• functional Magnetic Resonance Imaging (fMRI)
• Visual Cortex
• Orientation Decoder
• Direction Decoder
• Data Analysis
• fMRI- Accomplishments and Future works

22
Data Analysis

1. Three dimensional (3D) motion correction
2. Linear trend removal (using automated image
   registration software)
3. Sorting the voxels according to the responses to
   the visual areas.
4. Shifting the data 4 s to account for the
   hemodynamic delay
5. Averaging the fMRI signal intensity of each voxel
   for each 16 s stimulus trail.
6. Normalization relative to average of the entire
   time course within each run
7. Labeling the activity patterns according to
   corresponding stimulus Orientation or Direction.
8. Classification (using cross-validation for train
   and test)

23
Classification

• The classification is done by a linear ensemble
  Orientation/Direction detector
• ?k preferred Orientation/direction
• x(x1, x2,..., xd) voxel inputs
• Linear detector function
• where wi is the weight of voxel i, and w0 is
  the bias

24
Classification (con)

• Linear Support Vector Machine (SVM) is used to
  calculate a linear discriminant function for the
  pairs of all orientation/direction
• Then the pairwise discriminant functions
  comparing ?k and the other directions are simply
  added to yield the linear detector function

25
Outline

• functional Magnetic Resonance Imaging (fMRI)
• Visual Cortex
• Orientation Decoder
• Direction Decoder
• Data Analysis
• fMRI- Accomplishments and Future works

26
fMRI- Accomplishments and Future works

• Identification of the position of several
  retinotopically organized visual areas.
• Measurement of the retinotopic organization
  within these areas.
• Identification of the location of
  orientation/motion-sensitive region.
• Measurement of the responses associated with
  contrast, color and motion.
• Measurement of localized deficits in activity in
  subjects with cortical damage.
• Measurement of the effects of attentional
  modulation on visually evoked responses.

High inter-subject variability of functional
activity and sheer dimensionality of fMRI data
are the two major problems in fMRI data
classification.
27
Another Active Project on Mind-Reading

• Classifying cognitive processes based on fMRI
  images How can we use statistical machine
  learning methods to classify the hidden cognitive
  process of a human subject, based on their
  observed fMRI data?
• In this fMRI study Mitchll et al trained their
  algorithms to decode whether the words being read
  by a human subject are about tools, buildings,
  food, or several other semantic categories. The
  trained classifier is 90 accurate, for example,
  discriminating whether the subject is reading
  words about tools or buildings.

http//www.cs.cmu.edu/afs/cs/project/theo-73/www/i
ndex.html
28
Thank you

• Any Questions?

Much remains to be learned about how the human
brain represents the basic attributes of visual
experiences.