Project

1
Project 7Emotion Detection in the Loop from
Brain Signals and Facial Images

• Guillame Chanel, Koray Ciftci, Javier Cruz Mota,
  Arman Savran, Luong Hong Viet, Lale Akarun, Alice
  Caplier, Michele Rombaut and Bulent Sankur

2
Objective

• Determine the feasibility of fusing different
  modalities for emotion detection!
• 1) Is subject excited or not?
• 2) If excited, positive or negative emotion?

Emotions Positive Happiness Negative
Disgust Neutral
3
The Modalities

• Video recordings
• functional Near-Infrared Spectroscopy (fNIRS)
• Electroencephalography EEG
• Physiological data Galvanic Skin Response (GSR),
  blood pressure, respiration
• First step, preparation of a multimodal database
  of emotions!
• Difficulties
• Two group of modalities

4
fNIRS video
fNIRS acquisition
trigger
fNIRS device
stimuli
Video acquisition
5
Protocol for Video and fNIRS

• What kind of stimulus?
• Decided to work with videos of facial emotion and
  stimulation images
• negative, neutral, positive
• disgust, neutral, happy

6
Video Analysis
No eyebrows!
7
Video Analysis

• Facial Feature Extraction
• 1) Active Contour Based Approach
• 2) Active Appearance Models

8
Video Analysis

• AAM Result

9
fNIRS EEG
fNIRS acquisition
trigger
stimuli
EEG periph. response device
EEG acquisition
10
Protocol for EEG and fNIRS

• What kind of stimulus?
• International Affective Picture System (IAPS)
• Selection of pictures based on values of valence
  and arousal
• Calm, exciting positive, exciting negative
• Protocol design

Trial 
Session 
Experminent  
11
An example fNIRS signal during positive
stimulus
12
For the EEG
Fs 1024 Hz 54 electrodes 12.5 seconds
4-45Hz filtering Laplacian computation
FFT for each signal (electrode)
13
For peripheral signals

• GSR
• Mean, variance, derivative
• Respiration
• Frequency analysis
• Blood volume pressure
• compute heart rate

14
Results

• Video and fNIRS database
• 16 participants (7 are recorded twice) ? 23
  recordings
• 5 stimuli per class (neutral, happiness, disgust)
  
• 1h20 of emotional reactions (5h of recordings)

15
Results

• Video DataBase Structure

16
Results

• EEG, peripheral signals and fNIRS database
• 5 participant over 3 sessions
• 30 stimuli per class (calm, positive, negative)
• 1h30 of emotional reactions

17
To do,

• Video
• Facial contours detection
• Classification
• TBM (Transferable Belief Model)
• Other algorithms (HMM, )
• Analysis of dynamics aspects of facial
  expressions
• fNIRS, EEG and peripheral signals
• Feature extraction
• Classification (Bayes, SVM, KNN)

18
To do,

• Fusion
• Do facial expressions correspond to internal
  state of the user?
• Feature level
• combination or concatenation of features
• Decision level
• Majority voting, probabilistic