Information Extraction

1
Information Extraction

• An Information Refinement Technology and its
  Implementation

2
Overview

• What is Information Extraction (IE)
• The Message Understanding Conferences (MUCs)
• Overview of the Proteus IE system
• IE walk-through

3
The Message Understanding Conferences (MUCs)

• Sponsored by the Defense Advanced Research
  Projects Agency (DARPA) 1991-1998.
• Developed methods for formal evaluation of IE
  systems
• In the form of a competition, where the
  participants compare their results with each
  other and against human annotators' key
  templates.
• Short system preparation time to stimulate
  portability to new extraction problems. Only 1
  month to adopt the system to the new scenario
  before the formal run.

4
Definitions
Broad definition Any method for filtering
information from large volumes of text
(including information retrieval, data mining
etc.) Narrow definition, MUC definition, de
facto definition "The identification of
instances of a particular class of events or
relationships in a natural language text, and
the extraction of the relevant arguments of
the event or relationship." It "involves the
creation of a structured representation (such as
a data base) of selected information drawn from
the text" (Grishman -97).
5
Events - Relations - Arguments
6
Scenario Template Information Extraction Task

• The identification of instances of predefined
  events or relationships in free text.
• The extraction of the arguments of these events
  or relationships.
• The creation of a structured representation of
  this information.

7
MUC-6 Scenario Template Task Example (Changes in
corporate executive management personnel)
8
F i l l e d T e m p l a t e
ltTEMPLATE-afv980225-1gt DOC_NR
"afv980225" CONTENT ltSUCCESSION_EVENT-afv980
225-1gt ltSUCCESSION_EVENT-afv980225-1gt
SUCCESSION_ORG ltORGANIZATION-afv980225-1gt
POST ordförande IN_AND_OUT
ltIN_AND_OUT-afv980225-1gt
ltIN_AND_OUT-afv980225-2gt
VACANCY_REASON REASSIGNMENT ltIN_AND_OUT-afv98022
5-1gt IO_PERSON ltPERSON-afv980225-1gt
NEW_STATUS OUT ON_THE_JOB UNCLEAR
OTHER_ORG ltORGANIZATION-afv980225-1gt
REL_OTHER_ORG SAME_ORG ltIN_AND_OUT-afv980225-2gt
IO_PERSON ltPERSON-afv980225-2gt
NEW_STATUS IN ON_THE_JOB UNCLEAR
OTHER_ORG ltORGANIZATION-afv980225-2gt
REL_OTHER_ORG OUTSIDE_ORG ltORGANIZATION-afv98022
5-1gt ORG_NAME "Karo Bio" ORG_TYPE
COMPANY ltORGANIZATION-afv980225-2gt
ORG_NAME "S-E-Bankens läkemedelsfonder"
ORG_TYPE COMPANY ltPERSON-afv980225-1gt
PER_NAME "Per-Olof Mårtensson" PER_ALIAS
"Mårtensson" ltPERSON-afv980225-2gt
PER_NAME "Bertil Hållsten"
9
Problem
Var är raketuppskjutningen?
PARIS A huge new rocket that had been intended
to put Europe further ahead of the United States
in the world commercial satellite-launching
market had to be blown up Tuesday, 40
seconds into its maiden flight, after it veered
from its trajectory. Stunned technicians and
invited guests at the European Space Agencys
launch center at Kourou, French Guiana, watched
pieces of the Ariane 5 rocket rain down in
flames from 13,120 feet, where the self-destruct
mechanism exploded it on a command from the
ground.
10
Vad är det för kul med IE?

• Limited text understanding, not full text
  understanding probably feasible
• Involves many different levels of language/text
  analysis
• name identification
• tagging - parsing
• semantics
• anaphora resolution
• discourse analysis
• world knowledge inferences etc...

11
Why?
An IE system can be seen as a frozen, reusable
query to be used when one wants stereotypical,
repetitive, information back rather than the
answer to one-off questions. (Yorick Wilks
-97) Most information is still to be found in
natural language media such as newspapers, radio
and TV broadcasts, not in structured databases.
To filter on the level of documents, with
Information Retrieval techniques, is not
sufficient. Automatic filtering of huge amounts
of text to find information about specific types
of events is helpful for many kinds of analyst
tasks, even if the result is not perfect.
12
MUC-7s five tasks

• Named Entity Task. Names of organizations,
  persons, locations. Mentions of dates, times,
  currency and percentage.
• Template Element Task. All organizations,
  persons, and vehicle artifacts mentioned in the
  texts, whether relevant to scenario or not.
• Template Relation Task. Domain-independent
  relationships between entities. Location_Of,
  Employee_Of, and Product_Of relations.
• Coreference Task. Capture information on
  coreferring expressions all mentions of a given
  entity. Nouns, Noun Phrases, Pronouns.
• Scenario Template Task.

13
Evaluation Metrics

• Systems are scored according to
• Recall (number of correctly filled slots)
• (täckningsgrad) (number of filled slots in the
  key)
• Precision (number of correctly filled slots)
• (number of filled slots)
• F-score (2 X precision X recall)
• (precision recall)

14
(No Transcript)
15
Textdatabas
Dokument
Preprocessningsmodul
Koncepthierarki
Lexikon
Taggning Disambiguering
Namndetektering
Sifferuttrycksanalys
Nomengruppsdetektering
Verbgruppsdetektering
Interaktiv exempelbaserad mönsterproduktion
Nominalfrasdetektering
Anaforaupplösning
Scenariospecifik mönsterigenkänning
Mönster matchnings motor
Sambandsledsanalys
Inferensregler
Mönster generalisering
Databasgenerering
Mönsterdatabas
Resultatdatabas
16
Information extraction walk-through
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
,
,
17
"ltingvargt" "ingvar" ltcgt ltSWEgt
ltFIRST_NAMEgt ltMASCgt N ? SG NOM "lterikssongt"
"eriksson" ltcgt ltSWEgt ltFAMILYgt N ? SG
NOM "lt\,gt" "\," CLB ltPUNCTgt "lttidigaregt"
"tidig" ltCMPgt A UTR/NEU DEF/INDEF SG/PL
NOM "ltvdgt" "vd" ABBR UTR/NEU INDEF SG
NOM "ltpågt" "på" PREP "ltassamgt"
"assam" ltcgt ltFOREIGNgt ltPROPgt ltPLACEgt N ? SG
NOM "ltpärksgt" "pärks" ltNON-SWETWOLgt N SG
NOM lt?gt "ltabgt" "ab" ltcgt ABBR UTR/NEU
INDEF SG NOM "ab" ABBR UTR/NEU INDEF SG
NOM "ab" ABBR UTR/NEU INDEF SG
NOM "lt\,gt" "\," CLB ltPUNCTgt "lthargt"
"ha" ltAUXgt ltSUPINEgt V ACT PRES "ltutsettsgt"
"utse" V PASS SUPINE "lttillgt" "till"
PREP "ltnygt" "ny" A UTR INDEF SG
NOM "ltordförandegt" "ordförande" N UTR
INDEF SG/PL NOM "ltigt" "i" PREP "lteckymgt"
"eckym" ltNON-SWETWOLgt N SG NOM
lt?gt "ltroposgt" "ropo" ltcgt ltFINgt ltFAMILYgt
N ? SG GEN "ltincgt" "inc" ltNON-SWETWOLgt N
SG NOM lt?gt "lt.gt" "." CLB ltPUNCTgt
18
lttextgt
ltCONSTIT CATN GENna DEFna NUMSG CASENOM
XN(RSS (HEAD a-firstname))gtltTOKENgtIngvarlt/TOKENgtlt
/CONSTITgt ltCONSTIT CATN GENna DEFna NUMSG
CASENOM XN(RSS (HEAD a-familyname))gtltTOKENgtEriks
sonlt/TOKENgtlt/CONSTITgt ltTOKENgt,lt/TOKENgt ltCONSTIT
CATA GENUTR-NEU DEFDEF-INDEF NUMSG-PL
CASENOM XN(RSS (HEAD tidig))gtltTOKENgttidigarelt/TO
KENgtlt/CONSTITgt ltCONSTIT CATABBR GENUTR-NEU
DEFINDEF NUMSG CASENOM XN(RSS (HEAD
VD))gtltTOKENgtVDlt/TOKENgtlt/CONSTITgt ltCONSTIT
CATPREP XN(RSS (HEAD på))gtltTOKENgtpålt/TOKENgtlt/CON
STITgt ltCONSTIT CATN GENna DEFna NUMSG
CASENOM XN(RSS (HEAD a-name))gtltTOKENgtAssamlt/TOKE
Ngtlt/CONSTITgt ltCONSTIT CATN GENna DEFna NUMSG
CASENOM XN(RSS (HEAD a-name))gtltTOKENgtPärkslt/TOKE
Ngtlt/CONSTITgt ltCONSTIT CATABBR GENUTR-NEU
DEFINDEF NUMSG CASENOM XN(RSS (HEAD
Ab))gt ltCONSTIT CATABBR GENUTR-NEU DEFINDEF
NUMSG CASENOM XN(RSS (HEAD Ab))gt ltCONSTIT
CATABBR GENUTR-NEU DEFINDEF NUMSG CASENOM
XN(RSS (HEAD AB))gtltTOKENgtABlt/TOKENgt lt/CONSTITgt lt/
CONSTITgt lt/CONSTITgt ltTOKENgt,lt/TOKENgt ltCONSTIT
CATV AUXY VOICEACT TENSEPRES XN(RSS (HEAD
ha))gtltTOKENgtharlt/TOKENgtlt/CONSTITgt ltCONSTIT CATV
AUXN VOICEPASS TENSESUPINE XN(RSS (HEAD
utse))gtltTOKENgtutsettslt/TOKENgtlt/CONSTITgt ltCONSTIT
CATPREP XN(RSS (HEAD till))gtltTOKENgttilllt/TOKENgtlt
/CONSTITgt ltCONSTIT CATA GENUTR DEFINDEF NUMSG
CASENOM XN(RSS (HEAD ny))gtltTOKENgtnylt/TOKENgtlt/CON
STITgt ltCONSTIT CATN GENUTR DEFINDEF NUMSG-PL
CASENOM XN(RSS (HEAD ordförande))gtltTOKENgtordföra
ndelt/TOKENgtlt/CONSTITgt ltCONSTIT CATPREP XN(RSS
(HEAD i))gtltTOKENgtilt/TOKENgtlt/CONSTITgt ltCONSTIT
CATN GENna DEFna NUMSG CASENOM XN(RSS (HEAD
a-name))gtltTOKENgtEckymlt/TOKENgtlt/CONSTITgt ltCONSTIT
CATN GENna DEFna NUMSG CASEGEN XN(RSS (HEAD
a-familyname))gtltTOKENgtRoposlt/TOKENgtlt/CONSTITgt ltCON
STIT CATN GENna DEFna NUMSG CASENOM XN(RSS
(HEAD a-name))gtltTOKENgtInclt/TOKENgtlt/CONSTITgt ltTOKEN
gt.lt/TOKENgt
lt/textgt
19
swedish-name-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
ltnamngt
lt/namngt
,
ltpositiongt
lt/positiongt
,
ltcompanygt
lt/companygt
ltpositiongt
lt/positiongt
ltcompanygt
lt/companygt
20
swedish-name-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
ltnamngt
lt/namngt
,
(HEAD a-person)
ltpositiongt
lt/positiongt
(HEAD a-position)
lt/companygt
,
ltcompanygt
(HEAD a-company)
ltpositiongt
lt/positiongt
(HEAD a-position)
ltcompanygt
lt/companygt
(HEAD a-company)
21
swedish-vg-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
ltnamngt
lt/namngt
,
(HEAD a-person)
ltpositiongt
lt/positiongt
(HEAD a-position)
lt/companygt
,
ltcompanygt
(HEAD a-company)
ltverb groupgt
lt/verb groupgt
ltpositiongt
lt/positiongt
(HEAD a-position)
ltcompanygt
lt/companygt
(HEAD a-company)
22
swedish-vg-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
ltnamngt
lt/namngt
,
(HEAD a-person)
ltpositiongt
lt/positiongt
(HEAD a-position)
lt/companygt
,
ltcompanygt
(HEAD a-company)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltpositiongt
lt/positiongt
(HEAD a-position)
ltcompanygt
lt/companygt
(HEAD a-company)
23
swedish-ng-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
ltnamngt
lt/namngt
,
(HEAD a-person)
ltpositiongt
lt/positiongt
(HEAD a-position)
lt/companygt
,
ltcompanygt
(HEAD a-company)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltpositiongt
lt/positiongt
(HEAD a-position)
ltcompanygt
lt/companygt
(HEAD a-company)
24
swedish-ng-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
ltnamngt
lt/namngt
,
(HEAD a-person)
ltnoun groupgt
(HEAD C-position)
lt/noun groupgt
lt/companygt
,
ltcompanygt
(HEAD a-company)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun groupgt
(HEAD C-position)
lt/noun groupgt
ltcompanygt
lt/companygt
(HEAD a-company)
25
swedish-ng-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
,
(HEAD a-person)
ltnoun groupgt
(HEAD C-position)
lt/noun groupgt
,
(HEAD a-company)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun groupgt
(HEAD C-position)
lt/noun groupgt
(HEAD a-company)
26
swedish-ng-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
,
(HEAD C-person)
ltnoun groupgt
(HEAD C-position)
lt/noun groupgt
,
(HEAD C-company)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun groupgt
(HEAD C-position)
lt/noun groupgt
(HEAD C-company)
27
swedish-entity-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
,
(HEAD C-person)
ltnoun groupgt
(HEAD C-position)
lt/noun groupgt
,
(HEAD C-company)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun groupgt
(HEAD C-position)
lt/noun groupgt
(HEAD C-company)
28
swedish-entity-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
,
(Entity C-person)
ltnoun groupgt
(Entity C-position)
lt/noun groupgt
,
(Entity C-company)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun groupgt
(Entity C-position)
lt/noun groupgt
(Entity C-company)
29
swedish-noun-complements
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
,
(Entity C-person)
ltnoun groupgt
(Entity C-position)
lt/noun groupgt
,
(Entity C-company)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun groupgt
(Entity C-position)
lt/noun groupgt
(Entity C-company)
30
swedish-noun-complements
(i.e. position-of-company)
lttextgt Ingvar Eriksson tidigare VD på Assam Pärks
AB har utsetts till ny ordförande i Eckym Ropos
Inc. lt/textgt
ltnoun groupgt
lt/noun groupgt
,
(Entity C-person)
ltnoun phrasegt
(Entity C-position)
lt/noun phrasegt
,
(Entity C-company)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun phrasegt
(Entity C-position)
lt/noun phrasegt
(Entity C-company)
31
swedish-noun-complements
swedish-person-position
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
ltnoun groupgt
lt/noun groupgt
,
(Entity C-person)
ltnoun phrasegt
(Entity C-position)
lt/noun phrasegt
,
(company attribute C-company.att)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun phrasegt
(Entity C-position)
lt/noun phrasegt
(company attribute C-company.att)
32
swedish-person-position
swedish-person-position
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
ltnoun phrasegt
,
(Entity C-person)
(Entity C-position)
lt/noun phrasegt
,
(company attribute C-company.att)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun phrasegt
(Entity C-position)
lt/noun phrasegt
(company attribute C-company.att)
33
swedish-person-position
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
ltnoun phrasegt
,
(Entity C-person)
(person attribute C-person.att)
(position attribute C-position.att)
lt/noun phrasegt
,
(company attribute C-company.att)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun phrasegt
(Entity C-position)
(position attribute C-position.att)
lt/noun phrasegt
(company attribute C-company.att)
34
swedish-event-patterns
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
ltnoun phrasegt
,
(Entity C-person)
(person attribute C-person.att)
(position attribute C-position.att)
lt/noun phrasegt
,
(company attribute C-company.att)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun phrasegt
(Entity C-position)
(position attribute C-position.att)
lt/noun phrasegt
(company attribute C-company.att)
35
swedish-event-patterns
(defpattern s-appoint "np-sem(C-Person)
infix-cl? vg(C-Appoint) 'till' np-sem(C-Position)
person-at1.attributes,
position-at5.attributes ")
C-Fire C-Hire
C-Appoint C-Become
...
utse
befordra
36
swedish-event-patterns
C-Person C-Appoint(voice pass) till
C-Position
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
ltnoun phrasegt
,
(Entity C-person)
(person attribute C-person.att)
(position attribute C-position.att)
lt/noun phrasegt
,
(company attribute C-company.att)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun phrasegt
(Entity C-position)
(position attribute C-position.att)
lt/noun phrasegt
(company attribute C-company.att)
37
swedish-event-patterns
C-Person C-Appoint(voice pass) till
C-Position
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
ltnoun phrasegt
,
(Entity C-person)
(person attribute C-person.att)
(position attribute C-position.att)
lt/noun phrasegt
,
(company attribute C-company.att)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun phrasegt
(Entity C-position)
(position attribute C-position.att)
lt/noun phrasegt
(company attribute C-company.att)
38
swedish-event-patterns
C-Person C-Appoint(voice pass) till
C-Position
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
ltnoun phrasegt
,
(Appoint Event)
(person attribute C-person.att)
(position attribute C-position1.att)
lt/noun phrasegt
,
(company attribute C-company1.att)
ltverb groupgt
lt/verb groupgt
(HEAD utse)
ltnoun phrasegt
(Entity C-position)
(position attribute C-position2.att)
lt/noun phrasegt
(company attribute C-company2.att)
39
inference rules
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
,
(Appoint Event)
(person attribute C-person.att)
(position attribute C-position1.att)
,
(company attribute C-company1.att)
(position attribute C-position2.att)
(company attribute C-company2.att)
40
inference rules
lttextgt Ingvar Eriksson tidigare VD på Assam
Pärks AB har utsetts till ny ordförande i Eckym
Ropos Inc. lt/textgt
,
(Template Event)
(Start-Job Event)
person C-person position C-position2 company
C-company2
(Leave-Job Event)
,
person C-person position C-position1 company
C-company1
(person attribute C-person.att)
(position attribute C-position1.att)
(company attribute C-company1.att)
(position attribute C-position2.att)
(company attribute C-company2.att)
41
template filling rules
ltTEMPLATEgt DOC_NR 1
CONTENT ltSUCCESSION_EVENT-1-1gt ltSUCCESSION_EVENT-
1-1gt SUCCESSION_ORG ltORGANISATION-1-1gt
POST ordförande
IN_AND_OUT ltIN_AND_OUTgt
VACANCY_REASON OTH_UNK ltIN_AND_OUTgt
IO_PERSON ltPERSON-1-1gt NEW_STATUS IN
ON_THE_JOB UNCLEAR
OTHER_ORG ltORGANIZATION-1-2gt
REL_OTHER_ORG OUTSIDE_ORG ltORGANIZATION-1-1gt
ORG_NAME Eckym Ropos
ORG_ALIAS ORG_DESCRIPTOR
ORG_TYPE COMPANY ORG_LOCALE
ORG_COUNTRY ltORGANIZATION-1-1gt
ORG_NAME Assam Pärks ORG_ALIAS
ORG_DESCRIPTOR ORG_TYPE COMPANY
ORG_LOCALE ORG_COUNTRY ltPERSO
N-1-1gt PER_NAME Ingvar Eriksson
PER_ALIAS PER_TITLE
(Template Event)
(Start-Job Event)
person C-person position C-position2 company
C-company2
(Leave-Job Event)
person C-person position C-position1 company
C-company1
(person attribute C-person.att)
(position attribute C-position1.att)
(company attribute C-company1.att)
(position attribute C-position2.att)
(company attribute C-company2.att)