Practical results with PPM

1
Practical results with PPM

• Keith Vertanen
• COGAIN Camp
• June 1st, 2005
• Inference Group
• University of Cambridge
• kv227_at_cam.ac.uk

2
Practical results with PPM

• What model order should we use?
• What training text should we use?
• How much training text should we use?
• Can we utilize multiple models?
• What are some areas for improvement?

3
Model order

• Fixed number of prior characters are used to
  estimate next character

4
Model order
5
Memory consumption
6
Training text

• What text should we train PPM on?
• Nothing
• General text corpora
• Newswire text
• Enron email
• Books
• Users past writings
• Email
• Documents

7
Training text
8
Using lots of training data
9
Switching model

• Multiple models with separate training text
• Use model with lowest avg bits/symbol on previous
  window

Switching model
Enron model
User model
new
text
Users past writing
10
Switching model
Enron training, academic papers test
2.2
Original, 16K
Switching, 16K
Original, 128K
Switching, 128K
2.1
2
Bits per symbol
1.9
1.8
1.7
1.6
1e005
1e006
1e007
1e008
Number training symbols
11
Multilingual language model
12
Language modeling challenges

• Long range dependencies
• Is your answer no_
• The company reported _ ten point six percent
  drop.
• It started to rain, so to stay dry I opened my
  big green and white golf ______
• Local suboptimal behavior
• Steamed white _____
• I hate golf.

13
Questions?
http//www.inference.phy.cam.ac.uk/kv227/ kv227_at_c
am.ac.uk