Title: Language Modeling experiences collected over the past evaluations SWB, MT
1Language Modeling experiences collectedover the
past evaluations (SWB, MT)
- Interactive Systems Labs
- Christian Fügen
- Pittsburgh, Aug. 25th, 2004
2Outline
- Corpus selection cleaning
- Some aspects of language modeling
- Length of n-grams and cut-offs
- Class based LMs
- Text-Merging vs. Interpolation
- Decoding with interpolated LMs
- Dictionary making
- Hints on using the SRILM-Toolkit
- Some evaluation results
3Corpus Selection
- The more data, the better? Not always!
- Corpora should match the demanded domain,if used
to form a single LM - Erroneous material should better be
removed,instead of using it for training ? LM
pollution - The more consistent the data, the better!
- Never use pre-filtered text material
- Rules which have been used to filter the text are
unknown - Interesting parts could be already filtered out,
e.g. noises - Unknown errors could have been inserted, e.g.
used abbreviation format, word substitutions,
compound handling - Merging different corpora could be difficult
4Corpus Cleaning I
- Noises, e.g. ltbreathgt
- Map noises only to a few classes
- SWB ltnoisegt BN ltbreathgt, ltfillergt, ltnoisegt
- Decision should be made on available
transcriptions - Hesitations, e.g. uh, uh-huh, mhm,
- Many different spelling variants
- Map all hesitations to a few (3) distinguishable
classes - Automatic clustering depending on the context
probably performs best - Abbreviations / Spellings, e.g. CNN ? C._N._N.
- Normalize all abbreviations to a common form
- Did not used the abbrproc script provided by LDC
? inserts errors - Instead wrote own abbreviation tagger based on a
background list and a set of rules - Numbers / Punctuation, e.g. 1993, 1st,
- Used numproc and punctproc provided by LDC
5Corpus Cleaning II
- Partial words, e.g. some-, s-
- Map all partial words with the same starting
phoneme to the same class - Add a few more classes for the most common
partials - Reductions, e.g. gonna ? going to
- Expand all reductions to their base form
- Not done for e.g. weve, youre
- Compounds, e.g. good-bye ? good bye
- Use a background dictionary to find a common form
- Sentence splitting
- Use output of punctproc together with a set of
own rules - Overall improvement about 0.5 - 1.0 absolutely
6N-Gram Length and Cut-Offs
- Depends on the corpus size
- SWB, 4.5M wordsbest 3-gram LM with cut-off of 2
for tri-grams - MTSWB, 5.5M wordsbest 3-gram LM with cut-off
of 2 for tri-grams - BN 140M wordsbest 4-gram LM with cut-off of 1
for bi-grams, 2 for tri-grams and 4 for
four-grams - Better way for pruning instead of using
cut-offs? - Remove n-grams if their removal causes (training
set) perplexity of the model increase by less
than threshold relative (SRILM)
MT corpus only
7Class Based LMs
- Clustering
- 500 - 1000 classes seems to be pratical
- Very time consuming 2 days for clustering 5M
word corpus with 800 classes and doing 30
iterations - Using the SRILM toolkit instead seems to be
better (1 day, see later) - Hints
- Best performance when interpolated together with
a standard n-gram LM on same corpus (see next
slide) - Due to introduced classes the n-gram length can
be increased - Also use cut-offs
means context dependent interpolation
8Text Merging vs. Interpolation
- Merging of text corpora
- No discounting already applied
- Reduces number of interpolated LMs
- Interpolation of LMs
- Interpolation weight can beautomatically
computed - Different n-gram lengths, cut-offs
- Context dependent interpolationis possible
- But time consuming during decoding
- Hints
- Merge text corpora, if they belong to the same
domain - Interpolate LMs if they represent different
types or domains - Do context dependent interpolation only, if
cross-validation corpus matches test corpus - Maximum number of interpolated LMs used so far
for eval system 3-fold
9Decoding with interpolated LMs
- Problems when using 3-fold interpolation with big
LMs - Decoding speed decreases
- Lattice generation takes long
- Rescoring over a matrix takes even longer
- Use a lookahead mapper
- Lookahead build over the first domain dependent
LM only - Exact scores only asked, when word identity is
known
Example Manual decoder initialization
10Dictionary Making I
- Filler-Words
- Models for silence and pause
- Noises
- SWB acoustic models for ltnoisegt, lthumangt,
ltbreathgt, ltlaughgt, ltthroatgt, ltsmackgt mapped to
one LM-class - BN acoustic models for ltbreathgt, ltnoisegt mapped
to themselves - Depends on the trained acoustic models therefore
- Hesitations
- SWB several special trained acoustic models
mapped to 3 LM classes - BN one acoustic model ltfillergt mapped to itself
- Depends on the trained acoustic models therefore
- Multi-Words
- Should be included
11Dictionary Making II
- Pronunciation variants
- SWB multiple pronunciation dictionary, 2.2
entries per word, 100k - BN single pronunciation dictionary, 1.1
entries per word, 54k - Decision depends highly on type of training
dictionary - Comparison can only be done between over several
iterations trained systems - Computing pronunciation probabilities
- Generate all pronunciation variants on the basis
of rules - Forced alignment on training data
- Count used rules during forced alignment
- Compute probabilities after some pruning
- Overall reduction in WER through LM-ing and
Dictionary making 3 absolutely
12Hints on using the SRILM-Toolkit I
- Discounting
- Modified Kneser-Ney
- kndiscountN for every Nkndiscount1
kndiscount2 kndiscount3 for tri-gram LM - Good-Touring
- Default
- Cut-offs for all discountings
- gtNmin Me.g. gt3min 3 for 32
- Default is gtNmin 1 for Ngt2
- Pruning
- prune P
- Same value for all n-grams
- Value has to be very small
13Hints on using the SRILM-Toolkit II
- Interpolation
- Only context independent
- Result can be written into one single LM
- Seems to be better than UKA/CMU context dependent
interpolation (only results for small domain) - Currently only good results for interpolating
standard n-gram LMs, i.e. w/o classes - Clustering
- Uses only bi-gram statistics
- Faster and as good as the tool from
UKA/CMUngram-class -vocab swb04mt04.vocab -text
swb04mt04.txt-numclasses 800 -classcounts
swb04mt04.counts-classes swb04mt04.classes gt
ngram-class.log - Interpolation with standard n-gram LMusing the
SRILM currently not working - For using LMs produced by SRILM-Toolkit
aseparate conversion script must be used,e.g.
convertSRI.tcl provided by me!!!
14RT-03 (SWB) Results I
- Vocabulary
- 41k vocabulary selected from SWB, BN, CNN
- CNN used for vocabulary selection but not for LM
training - Pronunciation Variants
- Rule derived dictionary expansion 95k entries
- Probabilities
- Based on frequencies (forced alignment)
- Viterbi decoding probabilities as
penalties (e.g. max 1) - Confusion networks real probabilities (e.g. sum
1) - Single vs. multi-pronunciation dictionary
15RT-03 Results II
- Better text processing, more data
- Removing inconsistencies 32.6 ? 32.4
- Adding CELL CTRAN transcripts 32.4 ? 31.6
- LM interpolation (context dependent)
- 3gram SWB
31.4 - 3gram SWB 5gram class SWB
31.0 - 3gram SWB 5gram class SWB 4gram BN
30.3 - 3gram SWB 5gram class SWB 4gram BN 4gram
CNN 30.5
16RT-04S (Meeting) Results I
- Existing 40k BN vocabulary expanded by
- Missing words from RT-03 SWB vocabulary
- Missing words from meetings (frequency gt 4)
- Missing abbreviations (frequency gt 5)
- 56 most common partial words plus breath, filler,
noise - 48k vocabulary with an OOV-Rate of 0.8 on RT-04S
devtest - Pronunciations derived from
- Existing SWB/Meeting dictionaries
- CMU dictionary
- Generated by Festival
- 55k dictionary
17RT-04S Results II
- LM sources
- SWB/CTRAN/CH/CELL
- Meetings
- BN
- 3-fold interpolation
- SWB Meeting 3-gram LM
- SWB Meeting 5-gram classbased LM (800
classes) - BN 4-gram LM
- Results on RT-04S devtest(SDM, ref.
segmentation) - Baseline 40k BN dictionary 3-gram BN LM
- All other experiments useexpanded dictionary
text merging interpolation (context ind.)
18 19Decoding alongContext Free Grammars with Ibis
- Interactive Systems Labs
- Christian Fügen
- Pittsburgh, Aug. 25th, 2004
20Outline
- Basics
- Language Model Interface in Ibis
- CFG Framework
- Some other Aspects when using grammar based
speech recognition - Spontaneous speech
- Sub-grammars / grammar domains
- Tight coupling of SR and DLM
- Other features
21Basics
- Context Free Grammars
- Chomsky Type 2
- A ? v, A ? V, v ? (V ? ?) e.g. anbn?
set of terminals, V set of non-terminals, (V??)
set of strings composes from symbols from V
and/or ? including the empty string - Some Speech Grammar Specifications
- SOUP Format of the SOUP-Parser, extension of the
CMU Phoenix-Parser format.greeting ( hello
) ( hi ) - JSGF The Java Speech Grammar Formatltgreetinggt
hello hi - SRGS XML Specification of JSGF from W3C
- A few others
- Grammar types
- Syntactical, semantical or mixed
- Statistical or non statistical
22Grammar based decoding
- Static network representation
- Faster, especially when used in FST-based
decoders - Transition scores can be computed on the basis of
any context lengthMostly only bi-/tri-grams are
practical - Memory requirements
- Time consuming static network compilation
- Dynamic network representation
- More flexible Adding of new parts on the fly is
easy - Less memory expensive
- Slower, due to dynamical expansion
- Only bi-gram statistics
23Language Model Interface in Ibis
- Linguistic Knowledge Source (LingKS)
- Common interface to all types of language models,
e.g. PhraseLM, Grammars, interpolation of LingKSs - Consists of 4 major functions
- lks.createLCT (lvX)creates an initial LCT with
an lvX - lks.extendLCT (lct, lvX)extends a given LCT with
an lvX - lks.scoreArray (lct)returns transition scores
for all lvX given an LCT - lks.score (lct, lvX)returns only the transition
score given an LCT to an lvX - Search Vocabulary Mapper (SVMap) defines mapping
between svX and lvX
lks lingistic knowledge source lct linguistic
context lvX language model index svX search
vocabulary index
24CFG Framework I
- CFG
- Context free grammar
- CFGSet
- Set of context free grammars
- ParseTree
- Stores the grammar states(LCTs) while
traversingthrough the network - RuleStack
- Stack for accessing andleaving the grammar
ruleswhile traversing through the network - Lexicon
- Stores terminal and non-terminal symbols
LingKS CFGSet
LingKS CFG
25CFG Framework II
- Supported file formats
- SOUP Grammar format of the SOUP-Parser(without
support of character level and speaker-side
rules) - JSGF Java Speech Grammar Format(without support
of import statements) - PFSG Probabilistic Finite State Graph format,
which is used by SRILM - FSM ATT FSM (finite state machine) text file
format - Basic initialization
- Can be used for decoding and basic parsing (word
skipping is not possible) - Hints for grammar writing
- Be as restrictive as possible
set SID(cfg,grammars) list list NAV
cfgPath/cfg.ka.nav cfgPath/cfg.base.nav cfgIni
t SID
ibisInit SID -lm cfgSetSID spassSID run
decoding cfgSetSID.data parseTree text svMap
svmapSID parsing
26CFG Framework III
- Transition probabilities
- Can be defined to have a fixed value (-1.0) or
equally distributed - Fixed value seems to work a little better
- Automatic dictionary selection
- A dictionary can beautomatically compiled
whilereading in the grammarsusing a large
backgrounddictionary - Outsourcing terminal classes
- Large lists of terminals,e.g. street names can
beoutsourced to the SVMapfor speeding up the
decoding
set SID(cfg,dict) dictPath/nav.dict set
SID(cfg,baseDict) dictPath/baseDict cfgInit
SID -makeDict 1 dictInit SID
-desc set SID(cfg,dict) ibisInit SID
-lm cfgSetSID
set SID(cfg,classes) list dictPath/nav.classe
s Class File Example, whereby _at_street has to be
a terminal in the CFG acherstrase
_at_street adalbert-stifter-strase
_at_street adenauerring _at_street
27Spontaneous Speech
- Coping with non-human and human noises
- Difficult to model in a grammar, because its not
known when they appear - Use filler words for modeling noises
- Filler words can occur between any two terminals
of the grammars - LM is not asked for their score, instead a filler
penalty is used therefore - Complete set of variable definition and
initialization may look like
set SID(cfg,grammars) list list NAV
cfgPath/cfg.ka.nav cfgPath/cfg.base.nav \
list SHARED
cfgPath/cfg.shared set SID(cfg,dict)
dictPath/nav.dict set SID(cfg,baseDict)
dictPath/baseDict set SID(cfg,classes)
list dictPath/nav.classes set
SID(cfg,fillers) list dictPath/nav.fillers
set dict set SID(cfg,dict) cfgInit
SID -makeDict 1 dictInit SID -desc
dict ibisInit SID -lm cfgSetSID \
-vocabDesc dict.v -mapDesc
dict.m svmapSID configure -filPen 60
Example filler file click interjection inter
jection(ah) pause
28Sub-Grammars / Grammar Domains
- A CFGSet is a set of single sub-grammars
- Each grammar can be seen to model a single
domain, e.g. hotel reservation, navigation, - Tags can be given to sub-grammars to identify the
domains - Sub-grammars can be activated / deactivated
during run-time - Common, domain independent concepts can be
defined in a special shared sub-grammar - Better to maintain and better handling together
with dialogue systems - Can be used e.g. for multilingual decoding
- Multilingual acoustic model
- Each sub-grammarrepresents a differentlanguage
- Common, multilingualsearch and language model
vocabulary - Decoding is working in parallel
29Tight Coupling of SR and DLM
- Sub-Grammars and single rules can be
- activated / deactivated during runtime
- given a weight for e.g. penalizing them
- Mechanism can be used for a tight coupling of SR
and DLM - Therefore, the same grammar description files
should be used in both, SR and DLM - Parsed output of SR can be directly used by the
DLM to determine the user intention - Together with the dialoguecontext, it is now
possibleto predict the context of thenext user
answer - Rules within that contextcan be preferred
byadjusting the rule weightsin the SR
30Other features
- Grammars can be expanded on the fly
- Starting over during decoding one sentence can be
activated - It is possible to define all rules as top-level
rules - Visualization is possible by using the FSM output
format together with the ATT FSM toolkit and
dot - Upcoming features
- Training of the grammar transitions by parsing a
training corpus - Collecting bi-gram transition statistics
- Hybrid Language Models / Unified Language Models
- Combinations of n-gram LMs and CFGs
- .
N-gram LM
Context Free Grammar
N-gram LM
N-gram LM
N-gram LM
CFG
CFG
CFG