Project 10 Facial Emotion Recognition Based On Mouth Analysis

1
Project 10Facial Emotion Recognition Based On
Mouth Analysis

• SSIP 08, Vienna

http//www.we-hope-project10-will-win.info
2
The Project

• Objective To recognize emotional state /
  expression using mouth information
• Input Mouth images (no make-up)
• Output Emotional State/ Expression
• Happy, Neutral, Sad

http//www.we-hope-project10-will-win.info
3
The Team
Péter Web programmer
Sofia programmer
Kornél programmer
Naiem researcher
Kamal programmer
http//www.we-hope-project10-will-win.info
4
The Tasks

• Create facial expressions photographic database
• Segment the mouth in the input image
• Use suitable features for expression
  characterization
• Design a reliable classifier to distinguish
  between different mouth expressions

http//www.we-hope-project10-will-win.info
5
SSIP Lips database

• Happy, Neutral and Sad Photos of SSIP students
  and lecturers
• Thank you all!!!

Happy Neutral Sad
http//www.we-hope-project10-will-win.info
6
Mouth Segmentation
http//www.we-hope-project10-will-win.info
7
Segmentation Results
And Segmentation Problems
http//www.we-hope-project10-will-win.info
8
Lips Features Extraction

• Detect the leftmost and rightmost lip points
• Normalize images (rotation, translation and
  scaling)
• Calculate features
• Eccentricity
• Convex Area
• Minor Axis
• Ratio of Upper to Lower Lip

http//www.we-hope-project10-will-win.info
9
Expression Classification

• SVM Classifier
• Two Stage Classification

?
?
Mouth Features
http//www.we-hope-project10-will-win.info
10
Results 1

• Differences between different classes were found
  to be statistically significant (plt0.01)
• Classification Accuracy
• Stage 1 (Sad / Not Sad) ? 88
• Stage 2 (Happy/ Neutral) ?62

http//www.we-hope-project10-will-win.info
11
Results 2
http//www.we-hope-project10-will-win.info
12
Conclusion

• Mouth information is often insufficient for
  recognizing facial expression / emotional state
• Other face features such as eyes and eyebrows can
  contribute in emotional state recognition

Future Work

• Acquire larger database for training and testing
• Test different facial expressions (such as anger
  and disgust)
• Other classifiers NN, FIS

http//www.we-hope-project10-will-win.info
13
GUI
http//www.we-hope-project10-will-win.info
14
References

• M. Gordan, C. Kotropoulos, I. Pitas,
  Pseudoautomatic Lip Contour Detection Based on
  Edge Direction Patterns
• J. Kim, S. Na, R. Cole, Lip Detection Using
  Confidence-Based Adaptive Thresholding
• F. Tang, Facial Expression Recognition using AAM
  and Local Facial Features
• M. Pantic, M. Tomc, L. Rothkrantz , A Hybrid
  Approcah to Mouth Features Detection

http//www.we-hope-project10-will-win.info
15

• Thank you for your attention!!!

http//www.we-hope-project10-will-win.info