Language-Model Based Text-Compression

1
Language-Model Based Text-Compression

• James Connor
• Antoine El Daher

2
Compressing with Structure

• Compression
• Huffman
• Arithmetic
• Lempel Ziv (LV78 LV77)
• Most popular compression tools based on LV77
• Exploiting structure
• Our goal incorporate prior knowledge about the
  structure of the input sequence

3
Perplexity and Entropy

• Compression ratio is bounded by the Entropy of
  the sequence to be compressed
• A low-perplexity language model is also a
  low-entropy distribution

4
Character N-grams

• Represent text as an nth order markov chain of
  characters
• Maintain counts of n-grams
• Build a library of huffman tables based on these
  counts

5
Compressing the file

• Training
• For each bigram in the training set, we keep a
  map of all the words that can follow it, along
  with their probabilities.
• E.g. to have ? (seen, 0.1), (been, 0.1),
  (UNK, 0.1), etc.
• Then for each bigram, we build a Huffman tree.

6
Compressing the File

• Compressing
• We go through the input file, using the Huffman
  trees from the training set to code each word
  based on the two preceding words.
• If the trigram is unknown, we code the UNK token,
  the revert to a unigram model (also coded using
  Huffman).
• If the unigram is unknown, we use a character
  level Huffman (trained on the training set) to
  code it.
• Decompression works similarily we mimic the same
  behavior

7
Extensions

• We have a sliding context window, so that
  whenever we are compressing a file, words that
  are seen there have their counts incremented when
  they enter the window (and decremented when they
  leave) this allows us to make better use of the
  local context in terms of trigrams/bigrams, and
  give more representative weights.

8
Results

• Competitive with Gzip