Learning fMRI-Based Classifiers for Cognitive States

1
Learning fMRI-Based Classifiers for Cognitive
States

• Stefan Niculescu
• Carnegie Mellon University
• April, 2003

Our Group Tom Mitchell, Luis Barrios, Rebecca
Hutchinson, Marcel Just, Francisco Pereira,
Xuerui Wang
2
fMRI and Cognitive Modeling

• Have
• First generative models
• Task ? Cognitive state seq. ? average fMRIROI
• Predict subject-independent, gross anatomical
  regions
• Miss subject-subject variation, trial-trial
  variation
• Want
• Much greater precision, reverse the prediction
• ltfMRI, behavioral data, stimulusgt of single
  subject, single trial ? Cognitive state seq.

3
Cognitive task
Cognitive state sequence
4
Cognitive task
Cognitive state sequence
Virtual sensors of cognitive state
5

1. Does fMRI contain enough information?
2. Can we devise learning algorithms to construct
   such virtual sensors?

Cognitive task
Cognitive state sequence
Virtual sensors of cognitive state
6
(No Transcript)
7

8
Preliminary Experiments Learning Virtual Sensors

• Machine learning approach train classifiers
• fMRI(t, t d) ? CognitiveState
• Fixed set of possible states
• Trained per subject, per experiment
• Time interval specified

9
Approach

• Learn fMRI(t,,tk) ? CognitiveState
• Classifiers
• Gaussian Naïve Bayes, SVM, kNN
• Feature selection/abstraction
• Select subset of voxels (by signal, by anatomy)
• Select subinterval of time
• Average activities over space, time
• Normalize voxel activities

10
Study 1 Pictures and Sentences
Xuerui Wang and Stefan Niculescu

• Trial read sentence, view picture, answer
  whether sentence describes picture
• Picture presented first in half of trials,
  sentence first in other half

• Image every 500 msec
• 12 normal subjects
• Three possible objects star, dollar, plus
• Collected by Just et al.

11

• It is true that the star is above the plus?

12
(No Transcript)
13
---
14
(No Transcript)
15
Is Subject Viewing Picture or Sentence?

• Learn fMRI(t, , t15) ? Picture, Sentence
• 40 training trials (40 pictures and 40 sentences)
• 7 ROIs
• Training methods
• K Nearest Neighbor
• Support Vector Machine
• Naïve Bayes

16
Is Subject Viewing Picture or Sentence?

• SVMs and GNB worked better than kNN
• Results (leave one out) on picture-then-sentence,
  sentence-then-picture data and combined
• Random guess 50 accuracy
• SVM using pair of time slices at 5.0,5.5 sec
  after stimulus 91 accuracy

17
Error for Single-Subject Classifiers
 
 

• 95 confidence intervals are 10 - 15 large
• Accuracy of default classifier is 50

18
Can We Train Subject-Indep Classifiers?
19
Training Cross-Subject Classifiers

• Approach define supervoxels based on
  anatomically defined regions of interest
• Normalize per voxel activity for each subject
• Each value scaled now in 0,1
• Abstract to seven brain region supervoxels
• 16 snapshots for each supervoxel
• Train on n-1 subjects, test on nth
• Leave one subject out cross validation

20
Error for Cross Subject Classifiers

• 95 confidence intervals approximately 5 large
• Accuracy of default classifier is 50

21
Study 2 Word Categories
Francisco Pereira

• Family members
• Occupations
• Tools
• Kitchen items
• Dwellings
• Building parts

• 4 legged animals
• Fish
• Trees
• Flowers
• Fruits
• Vegetables

22
Word Categories Study

• Ten neurologically normal subjects
• Stimulus
• 12 blocks of words
• Category name (2 sec)
• Word (400 msec), Blank screen (1200 msec) answer
• Word (400 msec), Blank screen (1200 msec) answer
• Subject answers whether each word in category
• 32 words per block, nearly all in category
• Category blocks interspersed with 5 fixation
  blocks

23
Training Classifier for Word Categories

• Learn fMRI(t) ? word-category(t)
• fMRI(t) 8470 to 11,136 voxels, depending on
  subject
• Training methods
• train ten single-subect classifiers
• kNN (k 1,3,5)
• Gaussian Naïve Bayes ? P(fMRI(t) word-category)

24
Study 2 Results

• Classifier outputs ranked list of classes
• Evaluate by the fraction of classes ranked ahead
  of true class
• 0perfect, 0.5random, 1.0 unbelievably poor

 
 
25
Study 3 Syntactic Ambiguity
Rebecca Hutchinson

• Is subject reading ambiguous or unambiguous
  sentence?
• The experienced soldiers warned about the
  dangers conducted the midnight raid.
• The experienced solders spoke about the dangers
  before the midnight raid.

26
Study 3 Results

• 10 examples, 4 subjects
• Almost random results if no feature selection
  used
• With feature selection
• SVM - 77 accuracy
• GNB - 75 accuracy
• 5NN 72 accuracy

27
Feature Selection

• Five feature selection methods
• All (all voxels available)
• Active (n most active available voxels according
  to a t-test)
• RoiActive (n most active voxels in each ROI)
• RoiActiveAvg (average of the n most active
  voxels in each ROI)
• Disc (n most discriminating voxels according to
  a trained classifier)
• Active works best

28
Feature Selection
 
   
29
Summary

• Successful training of classifiers for
  instantaneous cognitive state in three studies
• Cross subject classifiers trained by abstracting
  to anatomically defined ROIs
• Feature selection and abstraction are essential

30
Research Opportunities

• Learning temporal models
• HMMs, Temporal Bayes nets,
• Discovering useful data abstractions
• ICA, PCA, hidden layers,
• Linking cognitive states to cognitive models
• ACT-R, CAPS
• Merging data from multiple sources
• fMRI, ERP, reaction times,

31
End of talk