Emotion Recognition

1
Emotion Recognition

• Iris Bass
• Thao Nguyen

Supervised by Dr. Ishwar K. Sethi
2
(No Transcript)
3
Objective

• Human computer interaction
• Voice Stream
• Discriminate between emotions

4
Why do we need emotion recognition?

• Interactive Voice Response (IVR)
• Robotics
• Computer/video games

5
(No Transcript)
6
Features

• Pitch
• Energy
• Speaking Rate
• Formants their Bandwidths
• Mel Frequency
• Cepstral
• Coefficients
• (MFCC )

7
(No Transcript)
8
SUSAS

• Composed of 32 speakers (13 female, 19 male)
• Access of 16,000 utterances
• Five domains
• Neutral sample
• Angry sample
• Five-fold cross validation. We split our database
  into two parts testing and training.

9
(No Transcript)
10
Classifiers

• Decision Tree
• Linear Discriminant Analysis (LDA)
• Self Organizing Map ( SOM )
• Support Vector Machines (SVM)

11
Decision Tree

• Binary Tree
• Nodes, branches
• Split criteria to determine the effectiveness of
  splitting on training data

12
Decision tree( cont )
lt 0.53276
13
Classifiers

• Decision Tree
• Linear Discriminant Analysis (LDA)
• Self Organizing Map ( SOM )
• Support Vector Machines (SVM)

14
Linear Discriminant Analysis

• Classification problems
• Linearly separates two groups
• Discriminant analysis classifies the instance to
  the largest conditional probability in order to
  make the smallest expected number or
  misclassifications

15
Classifiers

• Decision Tree
• Linear Discriminant Analysis (LDA).
• Self Organizing Map ( SOM )
• Support Vector Machines (SVM)

16
What is SOM?

• Neural network-based method for unsupervised
  learning.
• Maps high dimensional data on a low dimensional
  grid

17
Classifiers

• Decision Tree
• Linear Discriminant Analysis (LDA).
• Self Organizing Map ( SOM )
• Support Vector Machines (SVM)

18
Support Vector Machine

• Map data into a high dimensional space where they
  are linearly separable
• Find the maximum margin linear classifier in that
  space

19
SVM( cont)

• 2 class classifier
• Solve our problem
• - construct 4 SVM classifiers, each
  classifier for each emotion.
• - choose the maximum output score among 4
  output scores from the SVMs

20
(No Transcript)
21
Results

• From SVM
• From decision tree
• From minimum misclassification method
• From maximum accuracy method
• From dominant class method
• From SOM LDA

22
Results from SVM
Poly 3
Poly 5
Rbf 2
Rbf 4
Rbf 6
Sig 2
Poly 2
Poly 4
Poly 6
Rbf 3
Rbf 5
Sig 1
23
Results

• From SVM
• From decision tree
• From minimum misclassification method
• From maximum accuracy method
• From dominant class method
• From SOM LDA

24
Results from decision tree
Best decision tree
25
Confusion matricies
26
Results

• From SVM
• From decision tree
• From minimum misclassification method
• From maximum accuracy method
• From dominant class method
• From SOM LDA

27
Min misclassification

• Classify testing data by using both
• - SVM ( polynomial , exp 2 )
• - Decision tree( DT )
• Construct confusion matrices
• Compare the predicted class labels, ESVM and EDT
  , for each testing sample
• - if the same, choose that common class
  label
• - if different, find PESVM ( EDT )
• PEDT(
  ESVM )
• minProb
  min(PESVM, PEDT)
• choose
  class label from minProb

28
Confusion matricies
29
Min misclassification
30
Results

• From SVM
• From decision tree
• From minimum misclassification method
• From maximum accuracy method
• From dominant class method
• From SOM LDA

31
Maximum accuracy

• Classify testing data by using both
• - SVM ( polynomial , exp 2 )
• - Decision tree( DT )
• Construct confusion matrices
• Compare the predicted class labels, ESVM and EDT
  , for each testing sample
• - if the same, choose that common class
  label
• - if different, find PESVM ( ESVM
  )
• PEDT(
  EDT )
• maxProb
  max(PESVM, PEDT )
• choose
  class label from maxProb

32
Confusion matricies
33
Maximum accuracy
34
Confusion matricies
35
Confusion matricies
36
Results

• From SVM
• From decision tree
• From minimum misclassification method
• From maximum accuracy method
• From dominant class method
• From SOM LDA

37
Dominant class

• Classify testing data by using both
• - SVM ( polynomial , exp 2 )
• - Decision tree( DT )
• Construct confusion matrices
• Compare the predicted class labels, ESVM and EDT
  , for each testing sample
• - if the same, choose that common class
  label
• - if different,
• 1) if either ESVM or EDT is
  neutral, choose neutral
• 2) none of them is neutral,
  
• use the algorithm
  of the first combination

38
Dominant class
39
Process of improvement

• Enjoy our work !!!

40
Process of improvement

• Enjoy our work !

41
Comparision
42
Results

• From SVM
• From decision tree
• From minimum misclassification method
• From maximum accuracy method
• From dominant class method
• From SOM LDA

43
Hierachy
44
First layer
45
Color map with Hit and Purity
46
Component Analysis
47
Results from SOMLDA
48
Conclusion

• Overview
• Achieved our results

49
Future Work

• Perform feature elimination
• Work with a database that has whole sentences.
• Try other classifier combinations.
• Improve the accuracy
• Combine all of the different parts of speech
  recognition to produce the voice streaming
  system.

50
Acknowledgements

• Dr. Sethi
• Intelligent Information Laboratory
• REU Faculty
• REU Participants
• NSF

51
References

• R. Kent, and C. Read, Acoustic Analysis of
  Speech Second Edition, Thomas Learning, 2002
• Rabiner, L. and Schafer, R. Digital Processing of
  Speech Signals, Prentice-Hall, 1978
• J. Deller, J. Proakis, and J.Hansen,
  Discrete-Time Processing of Speech Signals,
  Macmillian, 1993
• T. Mitchell, Machine Learning, McGraw-Hill 1997
• M. Schröder (2003). Emotional speech synthesis
  for emotionally-rich virtual worlds. Proc. of
  Workshop on emotionally rich virtual worlds with
  emotion synthesis at the 8th International
  Conference on 3D Web Technology (Web3D), 10.
  March 2003, St.Malo, France.
• O.-W. Kwon, K. Chan, J. Hao, and T.-W. Lee,
  Emotion Recognition by speech Signals," Proc.
  EUROSPEECH 2003, Geneva, Switzerland, Sept. 2003.
  
• Mingkun Li, Support Vector Machine, Intelligent
  Information Engineering Lab, Oakland University,
  2003
• Dr. Ishwar K. Sethi, Language Identification for
  a Voice Stream Portizoration System, Intelligent
  Information Engineering Lab, Oakland University,
  2003
• http//www.fbckenner.org/.../ images/future.gif
  future

52
References Continued

• A. Nogueriras, A. Moren, A. Bonafonte, and J.
  Marino Speech Emotion Recognition Using Hidden
  Markov Models Proc. EUROSPEECH, Scandinavia,
  2001
• Valery A. Petrushin Emotion Recognition in
  Speech Signal Experimental Study, Development,
  and Application, Accenture, April 2001
• Feng Yu,Eric Chang,Ying-Qing Xu, Heung-Yeung Shum
  . Emotion Detection from Speech to Enrich
  Multimedia Content. The Second IEEE Pacific-Rim
  Conference on Multimedia, Beijing,October 24-26,
  2001
• S. Yacoub, S. Simske, X. Lin, J. Burns
  HPL-2003-136 Recognition of Emotions in
  Interactive Voice Response Systems -2003
• http//www.asel.udel.edu/speech/tutorials/acoustic
  s/freq_domain.html Sound in the Frequency
  Domain 1996
• http//www.asel.udel.edu/speech/tutorials/acoustic
  s/time_domain.html Sound in the Time Domain
  1996
• http//mi.eng.cam.ac.ukkkc21/thesis_main.html
  The Formulation of Support Vector Machine 1998
• www.robotoys.com/.../ st_prod.html?p_prodid151
  I Cybie
• N. Sebe, I. Cohen, A. Garg, M.S. Lew, T.S. Huang,
  International Conference on Pattern Recognition
  (ICPR'02), vol I, pp. 17-20, Quebec City, Canada,
  August 2002

53
(No Transcript)