The SPECIALIST Lexicon and NLP Tools

1
The SPECIALIST Lexicon and NLP Tools
August 19, 2007
Medinfo 2007
Allen Browne, Guy Divita, Anantha Bangalore,
Karen Thorn, Chris Lu, Susanne Humphrey
2
US National Library of MedicineLister Hill
Nation Center for Biomedical Communication

• Allen C. Browne
• Guy Divita
• Chris Lu
• Susanne Humphrey
• Anantha Bangalore
• Karen Thorn

3
Medinfo Tutorial S-023SPECIALIST NLP Tools for
UMLS Developers

• Introduction Allen Browne
• The Specialist Lexicon Allen Browne
• Lexical Tools Chris Lu
• Text Tools Guy Divita
• Text Categorization Lu Browne
• Knowledge Source Server Anantha Bangalore

4
Text processing
Lexical tools
SPECIALIST LEXICON
5
The SPECIALIST Lexicon

• A syntactic lexicon
• Biomedical and general English
• Over 330,000 records

6
Lexicon Growth
7
George A. Miller The Science of Words 1991
8
(No Transcript)
9
Frequency Spectrum Alice in Wonderland
Bayaan, 2001
10
The SPECIALIST Lexicon

• Morphology
• Inflection
• Derivation
• Orthography
• Spelling variants
• Syntax
• Complementation for verbs, nouns, and adjectives

11
Morphology

• Inflectional
• nucleus, nuclei
• cauterize, cauterizes, cauterized, cauterizing
• red, redder reddest
• Derivational
• laryngeal -- larynx
• transport -- transportation

12
Derivational Morphology
Dictionaryologyist
13
Inflectional Morphology

• octopus
• octopi
• octopuses

14
Orthography
Spelling Variation

• align -- aline
• Graves disease -- Gravess disease -- Graves
  disease
• anesthetize -- anesthetise
• Esophagus -- oesophagus
• foetus fetus
• centre -- center

15
Orthography
16
Syntax -- Verb Complements

• intran
• Ill treat.
• trannp
• He treated the patient.
• ditrannp,pphr(with,np)
• She treated the patient with the drug.

17
Syntax -- Verb Complements
basetreat entryE0061964 catverb variantsreg
intran trannp tranpphr(with,np) tranpphr(o
f,np) ditrannp,pphr(to,np) ditrannp,pphr(with,
np) ditrannp,pphr(for,np) cplxtrannp,advbl no
minalizationtreatmentnounE0061968
18
The SPECIALIST Lexicon
19
Miller -- 1991
20
Lexicon Unit Records
basechronic entryE0016869 catadj variantsin
v positionattrib(1) positionpred stative
baseKaposi's sarcoma spelling_variantKaposi
sarcoma entryE0003576 catnoun variantsuncount
variantsreg variantsglreg
baseaspirate entryE0010803 catverb
variantsreg trannp
nominalizationaspirationnounE0010804
basein entryE0033870 catprep
21
Orthographic vs. Lexicographic Word

• Why, for instance, if a two-word boy scout
  feels chilly on his one-word campground, does he
  pull up a two-word camp chair in front of his
  one-word campfire? Anyone who seeks a strictly
  logical answer to such questions is chasing
  will-o'-the-wisps (chargeable in telegrams as a
  single word, because of the hyphens) in a
  semantic bog.
• Louis Salomon, Semantics and Common Sense,
  Holt Rinehart and Winston, 1966.

22
UTF-8
basedeja vu spelling_variantdeja-vu
spelling_variantdéjà vu entryE0021340 catnoun
variantsuncount
baseresume spelling_variantrésumé
spelling_variantresumé entryE0053099 catnoun
variantsreg
baserole spelling_variantrôle entryE0053757 c
atnoun variantsreg
basecafe spelling_variantcafé
entryE0420690 catnoun variantsreg
23
Noun Variants

• Kaposis sarcoma
• Kaposis sarcomas
• Kaposis sarcomata
• Kaposi sarcoma
• Kaposi sarcomas
• Kaposi sarcomata

baseKaposi's sarcoma spelling_variantKaposi
sarcoma entryE0003576 catnoun variantsuncount
variantsreg variantsglreg
24
Regular Nouns
The plural suffix is s. y becomes ie following a
consonant before s. e is inserted before s if the
base ends in s, z, x, ch, or s
Leach Leaches Stomach Stomachs ? irregular
25
Greco-latin Regular nouns
26
Uncount Nouns(abstract or mass)
basesmallpox entryE0056359 catnoun variants
uncount basepotassium entryE0049387 catnoun
variantsuncount

• a smallpox
• two smallpoxes
• much smallpox
• a potassium
• two potassiums
• much potassium

This form does not occur
27
Fixed Plural Nouns
basescissors entryE0054633 catnoun
variantsplur
basepolice entryE0048616 catnoun variantspl
ur
28
Irregular Nouns
baselarynx entryE0036919 catnoun
variantsirreglarynges variantsreg
basecorpus entryE0019113 catnoun
variantsirregcorpora variantsreg
29
Regular Verbs

• The third person present tense suffix is s.
• y becomes ie following a consonant before s.
• e is inserted between z, x, ch, or sh and s.
• The past tense suffix is ed.
• y becomes ie following a consonant before ed.
• Final e is deleted before ed.

• The past participle is the same as the past tense.

• The present participle suffix is ing.
• - y becomes ie following a consonant before ing.
• - Final e is deleted before ing
• unless preceded by e, y or o.

30
Regular Verbs

• dismiss dismisses, dismissed, dismissing
• agree agrees agreed agreeing
• dry dries, dried, drying

31
Regular Doubling Verbs

• End in a CVC pattern
• Double the final consonant before ed and ing.
• Are otherwise regular
• variantsregd

control controls, controlled, controlling
32
Irregular Verbs

• basebite
• entryE0013219
• catverb
• variantsirregbitebitesbitbittenbiting
  
• intran
• trannp
• cplxtrannp,advbl

33
Ancillary Data Bases

• Synonymy
• sm.db
• Derivation
• dm.db, dm.rules
• Inflection
• im.rules
• Neoclassical compounds
• nc.db

34
Derivational Facts and Rules
dm.facts treatmentnountreatverb prohibitionno
unprohibitiveadj cell lineagenouncell
linenoun photochemotherapeuticadjphotochemother
apynoun pharmacotherapeuticadjpharmacotherapyn
oun
35
Derivational Facts and Rules
dm.rules e.g. alienationalienate ationnouna
teverb rationrate stationstate
36
Inflectional Facts and Rules
im.rules Noun rules (glreg) usnounsingular
inounplural antusanti manounsingularm
atanounplural anounsingularaenounplural
umnounsingularanounplural onnounsingular
anounplural sisnounsingularsesnounplura
l isnounsingularidesnounplural mennounsi
ngularminanounplural exnounsingularicesn
ounplural xnounsingularcesnounplural
37
Neoclassical compounds
nc.db abdomin(o)abdomenroot abaway
fromprefix acanth(o)prickleroot acar(o)mitero
ot acetabul(o)acetabulumroot adtowardsprefix a
gogueinducingterminal albumin(o)albuminroot si
sconditionterminal stomysurgical
openingterminal
38
PNEUMONOULTRAMICROSCOPICSILICOVOLCANOCONIOSIS
pneu.mo.no.ul.tra.mi.cro.scop.ic.sil.i.co.vol.ca.n
o.co.ni.o.sis \'n(y)u-m-(.)no--.l-tr-.mi--kr-
'skap-ik-'sil-i-(.)ko--(.)v\ n NL, fr. Gk
pneumo-n ISV ultramicroscopic NL silicon
al-'ka--no--.ko--ne--'o--ss ISV volcano Gk
konis dust a pneumoconiosis caused by the
inhalation of very fine silicate or quartz dust
-- Merriam Webster's 3rd International
Dictionary, page 1747.
39
The Protein of a tobacco mosaic virus, Dahlemense
strain

• acetylseryltyrosylserylisoleucylthreonylserylprol
  ylserylglutaminylphenylalanylvalylphenylalanylleuc
  ylserylserylvalyltryptophylalanylaspartylprolyliso
  leucylglutamylleucylleucyllasparaginylvalylcystein
  ylthreonylserylserylleucylglycllasparaginylglutami
  nylphenylalanylglutaminylthreonylglutaminylglutami
  nylalanylarginylthreonylthreonylglutaminylvalylglu
  taminylglutaminylphenylalanylserylglutaminylvalylt
  ryptophyllysylprolylphenylalanylprolylglutaminylse
  rylthreonylvalylarginylphenylalanylprolylglycylasp
  artylvalyltyrosyllsyslvalyltyrosylarginyltyrosylas
  paraginylalanylvalylleucylaspartylprolylleucylisol
  eucylthreonylalanylleucylleucylglycylthryonylpheny
  lalanylaspartylthreonylarginylasparaginylarginylis
  oleucylisoleucylglutamylvalylglutamylasparaginylgl
  utaminylglutaminylserylprolylthreonylthreonylalany
  lglutamylthreonylleucylaspartylalanylthreonylargin
  ylarginylvalylaspartylaspartylalanylthreonylvalyla
  lanylisoleucylarginylserylalanylasparaginylisoleuc
  ylasparaginylleucylvallasparaginylglutamylleucylva
  lylarginylglycylthreonylglycylleucultyrosylasparag
  inylglutaminylasparaginylthreonylphenylalanylgluta
  mylserylmethionylserylglycylleucylvalyltryptophylt
  hreonylserylalanylprolylalanylserine

40
Synonyms
sm.db alaradjwingnoun amygdalineadjtonsilno
un articularadjjointnoun bulbaradjmedulla
oblongatanoun fununcularadjboilnoun genicular
adjkneenoun hepatocellularadjliver
cellsnoun lazaradjleprosynoun lenticularadjc
rystalline lensnoun ypsiliformadjupsiloidadj w
olframnountungstennoun double
visionnoundiplopianoun
41
Text processing
Lexical tools
SPECIALIST LEXICON
42
Lexical Tools

• Wordind -- breaks strings into words
• Produces the Metathesaurus word indexes (MRXW)
• LVG -- performs various lexical transformations
• NORM -- a selection of LVG transformations,
• Used for Metathesaurus indexing
• Produces the Metathesaurus Normalized word and
  string indexes (MRXNW MRXNS)
• Used to access those indexes

43
Normalization

• Hodgkin Disease
• HODGKINS DISEASE
• Hodgkin's Disease
• Disease, Hodgkin's
• HODGKIN'S DISEASE
• Hodgkin's disease
• Hodgkins Disease
• Hodgkin's disease NOS
• Hodgkin's disease, NOS
• Disease, Hodgkins
• Diseases, Hodgkins
• Hodgkins Diseases
• Hodgkins disease
• hodgkin's disease
• DiseaseHodgkins
• Disease, Hodgkin

• disease hodgkin

44
The SPECIALIST Lexicon and NLP Tools
August 19, 2007
Medinfo 2007
Allen Browne, Guy Divita, Anantha Bangalore,
Chris Lu, Susanne Humphrey