Single-Trail Analysis of EEG during Rapid Visual Discrimination: Enabling Cortically Coupled Computer Vision

1
Single-Trail Analysis of EEG during Rapid Visual
Discrimination Enabling Cortically Coupled
Computer Vision

• Authors
• Paul Sajda, Adam D. Gerson, Marios G.
  Philiastides
• Dept. of Biomedical Engineering, Columbia
  University, USA
• Lucas Parra
• Dept. of Biomedical Engineering
• City College of New Work
• USA

2
Outlines

• Introduction
• Linear Methods for Single-Trail Analysis
• EEG Correlates of Perceptual Decision Making
• Identifying Cortical Process leading to Response
  Time Variability
• EEG-Based Image Triage
• Conclusion

3
Introduction

• Linear discrimination of multichannel
  electroencephalography (EEG) has used for
  single-trail detection of neural signatures of
  visual recognition events.
• Demonstrate relating neural variability to
  response variability
• Studies for response accuracy
• Response latency during visual target detection
• Example Running in the park with your head
  phones on, listening to your favorite tune,
  concentrating on your stride, you look up see a
  face that you immediately recognize as a high
  school friend.

4
Introduction (Con..)

• She is wearing a hat, glasses has aged fifteen
  years since you last saw her.
• You she are running in opposite directions so
  you only see her for a fleeting moment, yet you
  are sure it was her.
• Your visual system has just effortlessly
  accomplished a feat that has thus far baffled
  best computer vision systems.
• Such ability for rapid processing of visual
  information is even more impressive in light of
  the fact that neurons are relatively slow
  processing elements to digital computers, where
  individual transistors can switch a million times
  faster than a neuron can spike.

5
Introduction (Con..)

• Noninvasive neuroimaging has provided a means to
  peer into brain during rapid visual object
  recognition.
• Analysis of trial-averaged event-related
  potentials in EEG has enabled us to assess speed
  of visual recognition discrimination in terms
  of timing of underlying neural process.
• Recent work has used single-trial analysis of EEG
  to characterize neural activity directly
  correlated with behavioral variability during
  tasks involving rapid visual discrimination.
• Components extracted from EEG can capture neural
  correlates of visual recognition
  decision-making processes on a trail-by-trail
  basis.

6
Introduction (con..)

• Noninvasive BCI paradigms
• Having a subject consciously modulate brain
  rhythms
• Having a subject consciously generate a motor
  plan/visual imagery
• Directly modulating subjects cortical activity by
  stimulus frequency (SSVEP)
• Exploiting specific ERPs (novelty/oddball P300)
• Focus Single trail detection of ERPs
  relationship to visual discrimination/recognition
• Cortically coupled computer vision visual
  processor performs perception recognition
• EEG interface detects result (decision).

7
Linear Methods for Single-Trail Analysis LDA

• Conventional paradigm of evoked response
  considers neuronal activity following
  presentation of a stimulus.
• Analyzing EEG activity of multiple electrodes
  following presentation of an image.
• Aim identify one type of event, visual target
  recognition, Differentiate from other
• Binary classification based on temporal spatial
  profile of potential evoked following stimulus
  presentation.

8
Linear Methods for Single-Trail Analysis (Con..)

• EEG activity following each stimulus is recorded
  as DxT values.
• D of channels
• T of samples
• Record 1000 Hz, 64 channels
• ½ sec time window one word acquire 32000
  samples
• Train classifier Larger feature vector than N
  100 trials
• Low SNR
• Brute-force classification fail (32000-dim
  feature vector)

9
Linear Methods for Single-Trail Analysis (Con..)

• Exploit prior info
• S1 Reduce trial-to-trail variability by
  filtering (60 Hz ) slow drifts (lt0.5 Hz), no
  info carry
• S2 Reduce dimensionality by grouping signal into
  L sample blocks does not change within time
  window.
• S3 Increase of trails by L redundant
  samples/window
• L50 signal of interest at 10 Hzgtgt signal noise
• Transform original data for each trail with TD
  dimensions into L trails TD/L
• L50, N100, training exam5000
• train classifier with 640-dimensional feature
  vector at 10 Hz

10
Linear Methods for Single-Trail Analysis LDA
(Con..)

• linear classifier TD/L dimension, good results
• Single training window LT, D-dimensional
  feature vector
• Linear Classification
• Feature vector x is projected onto an orientation
  defined by vector w
• Projection, y wTx, optimally differentiates
  betn two classes.

11
(No Transcript)
12
Linear Methods for Single-Trail Analysis (Con..)

• Adv
• linear voltage combination-immediate
  interpretation (current)
• Coefficients that coupled current with observed
  voltages given for linear model by altxTygt/lty2gt
• Angular bracket indicate average over trials
  samples.
• Coefficients a describe coupling ( correlation)
  of discrimination component y with sensor
  activity x.
• a x D-dim vector(row/column)
• Strong coupling low attenuation visualized
  intensity map-sensor projections

• Easy to implement
• Fast
• Permit real-time operation
• Disadv
• does not capture synchronized activity gt10 Hz
• Neither does it capture activity that is not a
  fixed distance in time from stimulus
• Only phase-locked activity detection

13
EEG Correlates of Perceptual Decision Making

• Single-trail LDA to identify cortical correlates
  of decision-making during rapid discrimination of
  images.
• Experiment Psychophysical performance is
  measured for several subjects during RSVP task
• A series of target (faces) and non-target (cars)
  trials presented in rapid succession (Fig. 25.2a)
• Simultaneously recording neuronal activity from
  64-channel EEG electrode array.
• Stimulus evidence varied with phase coherence
  (Fig. 25.2b)

14
EEG Correlates of Perceptual Decision Making
(Con..)

• Within trail block face car images over a
  range of phase coherence presented random order.
• 12 face 12 car, grayscale images (512x512, 8
  bits/pixel)
• Equal of front side views
• Equal spatial frequency, luminance contrast
• Subjects are required to discriminate
• type of image (face/car)
• report decision by pressing a button.

15
EEG Correlates of Perceptual Decision Making
(Con..)

• EEG data acquired in Electrosatically shielded
  room
• Used Sensorium EPA-6 Elecrophysiological Amp.
  from 60 Ag/Agcl scalp electrodes, 3 periocular
  electrodes placed below left eye, left right
  outer canthi
• Sample rate 1000Hz, 0.01-300 Hz passband, 12
  dB/octave HPF 8th order elliptic LPF.
• Software-based 0.5 Hz HPF (remove dc drifts)
• 60/120 Hz notch filter (minimize line noise)
• Record EOG signals, remove motion blink
• Motor response stimulus events recorded on
  separate channels

16
EEG Correlates of Perceptual Decision Making
(Con..)

• Identify EEG components that maximally
  discriminate betn 2 conditions
• Identify 2 time window for discrimination
• Identify early (?170 ms following stimulus)
  late component (gt300 ms)
• LD trained by integrating data across both time
  windows (2D-feature vector)
• Performance improved (higher Az)
• Fig. 25.3 comparison of psychometric
  neurometric functions for 1 sub in dataset.
• All subjects a single function can fit behavioral
  neuronal datasets 2 sep functions.

17
EEG Correlates of Perceptual Decision Making
(Con..)

• Data analyzed both stimulus response-locked
  conditions
• Both face selective components appear to be more
  correlated with onset of visual stimulation than
  response for one subject (Fig. 25.4)
• Discriminate activity significant down to 30
  phase coherence (early late component)
• Randomize Az significant level of plt0.01.
• Results neural correlates of perceptual decision
  making identified using high-spatial density EEG
  corresponding component activities

18
EEG Correlates of Perceptual Decision Making
(Con..)
19
Identifying Cortical Process leading to Response
Time Variability

• Significant variability in response time observed
  across trials in many visual discrimination
  recognition.
• Factors difficulty in discriminating an object
  on any given trial
• Trial-by-trial variability of subjects
  engagement
• Intrinsic variability of neural processing
• Identifying neural activity correlated with
  response time variability may shed light on the
  underlying cortical networks responsible
• for perceptual decision-making processes
  processing latencies that these networks may
  introduce for a given task

20
Identifying Cortical Process leading to Response
Time Variability (Con..)

• Visual target detection use RSVP single trial
  spatial integration of high-density EEG
• to identify time course cortical origins
  leading to response time variability.
• RSVP paradigm (Fig 25.5)
• Varied scale, pose position of target objects
  (people) requires subjects to recognize rather
  than low-level features
• Continuous sequence of natural scenes
• 4 blocks (50 sequences) rest period no more than
  5 mins.

21
Identifying Cortical Process leading to Response
Time Variability (Con..)

• Each sequences 50 images, 50 chance containing
  one target image with one or more people in a
  natural scene.
• Target appear middle 30 images/50 sequences
• Detractor image remaining natural scene without
  a person
• Each image 100 ms
• Fixation cross 2s betn sequences.
• Instructed to press left button of a 3-button
  mouse with right index finger while fixation
  cross present, release as soon as recognize
  target image.
• Used LD to determine spatial weighting
  coefficients that optimally discriminate betn
  EEG resulting from different RSVP task conditions
  (target vs. distractor) over specific temporal
  window betn stimulus response.

22
Identifying Cortical Process leading to Response
Time Variability (Con..)

• Integration across sensors enhances signal
  quality without loss of temporal precision common
  to trail averaging in ERP.
• Resulting discrimination components describe
  activity specific to target recognition
  subsequent response for individual trails.
• Intertrial variability estimated by extracting
  feature from discrimination components.
• Peaks of spatially integrated discriminating
  components found by fitting a parametric function
  to extracted component y(t)

23
Identifying Cortical Process leading to Response
Time Variability (Con..)

• Gaussion Profile parameterized by its height ?,
  width?, delay ? and baseline offset ?
• Response-locking of discriminating components
  determined by computing linear regression
  coefficients that predict latency of component
  activity, measured by ? from response time given
  by r.
• ?j ?rj b ?j-peak latencies, rj-response
  time for j-th trial, b-offset term for
  regression (Fig. 25.6)
• Proportionality factor from response time peak
  latency regression (?) degree of
  response-locking () for each com.
• Quantify extent to which component is correlated
  with response across trails.

24

• 0 pure stimulus
• 100 pure response lock
• ?100 slow responses late activity
• fast responses
• early activity
• ?0 timing activity does not change with
  response time therefore stimulus locked.

25
Identifying Cortical Process leading to Response
Time Variability (Con..)

• Group results discriminating component activity
  across 9 participants (Fig. 25.7)
• Scalp projections normalized prior averaging
• A shift of activity from frontal to parietal
  regions over the course of 200 ms.
• Scaled response times and component peak time are
  concatenated across subjects
• These registered group response times then
  projected onto scaled component peak times to
  estimate degree of response-locking across
  subjects.
• Group response lock increased from 28 (200ms) to
  78 (50 ms) after response.

26

27
Identifying Cortical Process leading to Response
Time Variability (Con..)

• Significant processing delay introduced by early
  processing stages
• Within 200 ms prior to response (250 ms following
  stimulus), activity is already, on average, betn
  25 35 response-locked
• It is possible that some of this early
  response-locking may be due to early visual
  processes (0-250 ms post stimulus
• For 9 subjects correlation Analysis reveal that
  Discrimination Component activity progressively
  becomes more response-locked with subsequent
  Processing stages

28
EEG-Based Image Triage

• EEG system capable of using neural signatures
  detected during RSVP to triage sequences of
  images, reordering them so target images are
  placed near the beginning of sequence.
• Called Cortically coupled computer vision
• Robust recognition capabilities of human visual
  system (invariance to pose, lighting, scale)
• use a noninvasive cortical interface (EEG) to
  intercept signatures of recognition events-
• visual processor performs perception
  recognition EEG interface detects the result
  (decision) of processing

29
Image Triage (Con..)

• Experimentation Series of self-paced feedback
  slides were presented indicating position of
  target images within sequence before after EEG
  triage.
• Participants completed 2 blocks (50 sequences)
  with brief rest period lasting no more than 5
  minutes betn blocks
• During 2nd block participants were instructed to
  quickly press left button of 3-button mouse with
  right index finger as soon as they recognized
  target images.
• Button press twice if one target immediately
  followed other.
• Not respond with a button press during 1st block.
• Classify EEG online used Fisher linear
  discriminator
• to estimate a spatial weighting vector that
  maximally discriminate between sensor array
  signals evoked by target non-target images.

30
Image Triage (Con..)

• 5000 images presented to subject in sequences of
  100 images (with without motor response)
• Training 2500 images (50 targets, 2450
  non-target)
• Training window 400-500 ms following stimulus
  onset was used to extract training data
• Weights updated adaptively with each trail during
  training
• Classification threshold adjusted to give optimum
  performance for observed prevalence (Class-prior)
• These weights threshold were fixed at end of
  training period applied to subsequent testing
  dataset (images 2501-5000)
• Offline triage after experiment multiple
  classifier with different training window onsets
  were used.

31
Image Triage (Con..)

• Training window 0900 ms in steps of 50 ms
• Duration 50 ms
• After trained, optimal weighting outputs found
  using logistic regression to discriminate target
  non-target images.
• EEG data evoked by 2500 images to train find
  inter-classifier weights.
• applied to testing dataset evoked by 2nd set 2500
  images (25015000)
• Following experiments all image sequences were
  concatenate to create training testing
  sequences that each contain 2500 images (target
  50, non-targets 2450).
• Image sequences are restored according to output
  of classifier with multiple training windows for
  EEG evoked by every image.

32
Image Triage (Con..)

• Comparison Sequence triaged based on button
  response.
• Image restored
• RT-onset of button response (occurs within1 sec)
• P(target/RT)0, no response
• Priors P(target) 0.02
• P(non-target)0.98
• P(RT/target) Gaussian distributions with mean
  variance determined from response times from
  training sequences
• If target appeared 1st in sequence 2 button
  response occurred within 1 sec-assigned to target
  images
• 2nd response to 2nd target image

33
Image Triage (Con..)

• Testing sequences if 2 or more within 1 sec,
  response with greatest p(target/RT) assigned to
  image.
• P(RT/non-target) is a mixture of 13 Gaussians
  (same variance as p(RT/target))
• Means shifting mean from p(RT/target) 600 ms in
  past to 700 ms in future, increments of 100 ms,
  exclude actual mean of p(RT/target)
• Triage results (subject 2) Fig. 25.8
• of targets as a function of of distracter
  images both before after triage based button
  press EEG.
• area under curve generated by plotting fraction
  of targets as a function of fraction of
  distracter images presented is used to quantify
  triage performance.

34
(No Transcript)
35
Image Triage (Con..)

• Area is 0.50 for all unsorted image sequences
  since target images are randomly distributed
  throughout sequences.
• Ideal 1.00
• No significant diff in performance (button-based
  EEG-based) triage (0.93?0.06, 0.92 ? 0.03,
  p0.69, N5)
• EEG (motor no-motor)(0.92?0.03, 0.91 ? 0.02,
  p0.81, N5)
• position of target images (black squares)
  non-target images (white squares) in images
  sequences (Raster 25.8b-f)
• Performance (5 subjects) Table 25.1

36
Image Triage (Con..)

• Performance (5 subjects) Table 25.1
• High-level performance
• Button-based triage performance begins to fail,
  when subjects do not consistently respond to
  target stimuli (response timegt1 sec), subject 2
  only 74
• EEG (motor button) increase triage performance

37
Conclusion

• Invasive noninvasive EEG recording obtained
  during RSVP of natural image stimuli have shed on
  speed, variability spatiotemporal dynamics of
  visual processing.
• Future issues learning/priming/habituation,
  effect of subject expertise, image type
  category
• Application level intercepting neural correlates
  of visual discrimination recognition events
  that effectively bypass slow noisy motor
  response loop.

38
Thanks