Estimating Local Optimums in EM Algorithm over GMM

1
Estimating Local Optimums in EM Algorithm over GMM

• Zhenjie Zhang, Bing Tian Dai, Anthony K.H. Tung

2
Outline

• Introduction
• Local Trapping Property
• Maximal Region and Its Verification
• Experimental Results
• Future Work and Conclusion

3
Gaussian Mixture Model

• Basics of GMM
• There are k components following Gaussian
  distributions
• The probability of observing x from component i
• The probability of observing x

4
Gaussian Mixture Model

• Find the configuration to maximize the
  probability of the whole data set
• Equivalent to maximize the log likelihood

5
Expectation-Maximization

• Initialization
• Randomly selects the initial configuration
• E-Step
• Calculate the cluster membership probability
• M-Step
• Update the parameters of component
• Convergence Condition
• No change on membership probability

6
Multiple-Run EM

• No guarantee on the convergence to Global Optimum
• Solution run multiple EM with different initial
  configuration, and return the best result
• Effect of Early Termination
• Efficiency
• More runs
• Better final result

7
Acceleration
8
Outline

• Introduction
• Local Trapping Property
• Maximal Region and Its Verification
• Experimental Results
• Future Work and Conclusion

9
Solution Space

• Solution Space
• Configuration move by EM Iteration

10
Local Trapping Property

• Local Trapping
• There is a path, any configuration on the path is
  a better solution than

11
Local Trapping Property

• Local Trapping
• No way to get out of the region, if every
  boundary configuration no better than

12
Outline

• Introduction
• Local Trapping Property
• Maximal Region and Its Verification
• Experimental Results
• Future Work and Conclusion

13
Maximal Region

• Maximal Region
• It covers the current configuration
• Every configuration on the boundary is no better
  than
• Local Optimum must be in Maximal Region
• By local trapping property

14
Maximal Region

• A special class of region
• Containing all configurations satisfying

15
Maximal Region

• Verification of maximal region
• O(n) time, where n is the number of points
• Derivation of an upper bound on the log
  likelihood of local optimum
• Constant time

16
Outline

• Introduction
• Local Trapping Property
• Maximal Region and Its Verification
• Experimental Results
• Future Work and Conclusion

17
Experiments

• On synthetic data set
• Points from some unknown random GMM

18
Experiments

• On real data set
• Spam size4601, dimensionality58

19
Future Work and Conclusion

• Future Work
• Stability analysis on Gaussian Mixture Model
• Distribution change detection

20
Future Work and Conclusion

• Conclusion
• Extension to the understanding of EM algorithm
  over GMM
• Acceleration of Multiple-Run EM algorithm
• Some Potentials in real applications

21
Question Answer