Ancestry.com%20Data:%20DEG%20Results,%20Observations,%20

1
Ancestry.com DataDEG Results, Observations,
Future Directions
2
Acknowledgements
3
Extraction Ontologies

• EOs
• Conceptual models for some domain
• Data frames for each concept
• Instance recognizers IO converters
• Operations with recognizers and instance
  recognizers for parameters
• Used for information extraction, free-form query
  processing, information integration, semantic-web
  applications,
• But, for the Ancestry.com project, only instance
  recognition
• Regular-expression recognition rules
• Dictionary-based recognition rules
• Combinations
• Heuristic rules
• Disambiguation within context
• Specialized case sensitivity

4
EO Rules Names

• Dictionaries
• First last names
• From 1990 census (http//www.census.gov/genealogy/
  names/)
• Minus stopwords (http//www.lextek.com/manuals/oni
  x/stopwords2.html)
• Titles dictionary Mr, Mrs, Miss, Dr, Rev,
• Simple regular expressions
• Dictionary regular-expression combinations

5
EO Rules Putting it all together

• Baseline dictionary
• Two or more consecutive dictionary words
• Require capitalization
• Extraction Ontology (EO)
• Regular expressions
• Exclusions schools, addresses,
• Patterns (various)
• Based on document structure
• Not generally applicable elsewhere

6
Extraction Ontology (EO)

• \b(Title\s)0,1First\s(A-Z\s)0,1Last
  \b
• John Doe
• Mr John Q Doe
• \bTitle\sLast\b
• Mr Doe
• \bTitle( A-ZA-Za-z)1,3\b
• Mr Anythingcapitalized Upto Threewords
• \b(Last)(\sTitle)?(\sFirst\sA-Z)1,2
  \b
• Doe John
• Doe Miss Jane
• Doe Anything Capitalized
• Also exclusions

7
Numbers

• Using training data examples
• Should use dev-test numbers

8
EO Results Example 1
THE BLAKE FAMILY IN ENGLAND IN a Genealogical
History of William Blake of Dorchester published
in 1857 appears the statement that the emigrant
to New England was the son of Giles Blake of
Little Baddow Essex and the record of several
generations of the family is given The sub -
stance of this record is trustworthy as being a
copy from Morant ' s History of Essex but the
statement that the Dorchester settler was of
this family was unwarranted by any evidence
Subsequently the late H G Somerby Esq by request
of Stanton Blake Esq made extended researches in
England to determine the origin of the American
family He finally located it at Over Stcwey
Somerset and the results of his investigations
were published in 1881 by W H Whitmore Esq in A
Record of the Blakes of Somersetshire The
evidences upon which Mr Somerby based his
conclusions - were first the record of aN
baptism in 1594 at Over Stowey of a William
Blake
9
EO Results Example 1
Method Correct Precision Recall F1 Dictionary 5 /
13 55.56 38.46 45.45
Correct matches William Blake Giles Blake
William Blake Mr Blake William Blake Missed
entities H G Somerby Esq Stanton Blake Esq
W H Whitmore Esq Mr Somerby Rev Charles M
Blake William Blake Esq William Arthur Jones
Esq A M Edward J Blake Esq
False positives Stanton Blake Rev Charles
William Arthur Jones William Blake
Results measured on set-aside TRAINING data
10
EO Results Example 1
Method Correct Precision Recall F1 Dictionary 5 /
13 55.56 38.46 45.45 (Adjusted) 8 /
13 88.89 61.54 72.73
Correct matches William Blake Giles Blake
William Blake Mr Blake William Blake Missed
entities H G Somerby Esq Stanton Blake Esq
W H Whitmore Esq Mr Somerby Rev Charles M
Blake William Blake Esq William Arthur Jones
Esq A M Edward J Blake Esq
False positives Stanton Blake Rev Charles
William Arthur Jones William Blake
Results measured on set-aside TRAINING data
11
EO Results Example 1
Method Correct Precision Recall F1 Dictionary 5 /
13 55.56 38.46 45.45 (Adjusted) 8 /
13 88.89 61.54 72.73 EO 7 / 13 63.64 53.85 58
.33
Correct matches William Blake Giles Blake
Mr Somerby William Blake Rev Charles M Blake
Mr Blake William Blake Missed entities H G
Somerby Esq Stanton Blake Esq W H Whitmore
Esq William Blake Esq William Arthur Jones
Esq A M Edward J Blake Esq
False positives Stanton Blake William
Arthur William Blake Edward J Blake
Results measured on set-aside TRAINING data
12
EO Results Example 1
Method Correct Precision Recall F1 Dictionary 5 /
13 55.56 38.46 45.45 (Adjusted) 8 /
13 88.89 61.54 72.73 EO 7 / 13 63.64 53.85 58
.33 (Adjusted) 10 / 13 90.90 76.92 83.33
Correct matches William Blake Giles Blake
Mr Somerby William Blake Rev Charles M Blake
Mr Blake William Blake Missed entities H G
Somerby Esq Stanton Blake Esq W H Whitmore
Esq William Blake Esq William Arthur Jones
Esq A M Edward J Blake Esq
False positives Stanton Blake William
Arthur William Blake Edward J Blake
Results measured on set-aside TRAINING data
13
EO Results Example 2
msM ffi , 113 2lst Street . Wholesale and
Retail dealers in all grade of Cook and Heating
stoves . Also the celebra ted Jno , Vann , Wm .
Kiser and Char ter OakBangea . 1S4 K . L . POLK
CO . ' S Coker Henry W , dry goods and grocer
, Washington ave bet Hiekman and Maple , Irondale
Coke Newton , lab Aliee Furnace Coker Wesley T ,
elk W R Coker , res Washington ave cor Hiekman ,
Irondale Coker Wm R , dry good . and grocer ,
Washington ave , cor Hiekman , Irondale C ' ola
Carlo , lunch house . 19 20th s , res same Colby
Mips Alice , bds 280 ! ) 2d ave ColdirrU see
also . ' aldircll Coldwell K A , bds 22 50
4th ave Coldwell Wm ,
Birmingham City Directory
14
Example 2 - Dictionary
Method Correct Precision Recall F1 Dictionary 31
/ 78 51.67 39.74 44.93
Correct matches Coke Newton Coldwell Wm
Cole Alexander Cole Burt Cole Charles Wm
Hood Cole Frank Cole James Cole John Cole
John and others Missed entities Coker
Henry W Coker Wesley T Coker Coker Wm R C
' ola Carlo Colby Mips Alice Coldwell K A
Cole Artolphus and others
False positives Coker Henry Furnace Coker
Wesley Coker Wm Rolling Mill Peter Zins
Birmingham Rolling Mill Cleveland Cole Charles
Cole Franklin Cole James Cole John and
others
Results measured on set-aside TRAINING data
15
Example 2 - EO
Method Correct Precision Recall F1 Dictionary 31
/ 78 51.67 39.74 44.93 EO 47 /
78 72.31 60.26 65.73
Correct matches Coker Henry W Coke Newton
Coker Wesley T Coker Wm R Coldwell K A
Coldwell Wm Cole Alexander Cole Burt Cole
Charles Wm Hood and others Missed
entities Coker C ' ola Carlo Colby Mips
Alice Cole Artolphus Cole Cradford Cole
Charles H ) orter II Herxfcld Cole P W
and others
False positives Peter Zins Cleveland Cole
Charles Furnace Cole P Peter Zins Cole
Samuel I Alice Furnace Bains Cole Wm Cole
Win Bessemer Cole Wm Davis Coleman Bettie
and others
Exclusion patterns no effect
Results measured on set-aside TRAINING data
16
Can we do better?
What if we knew the record boundaries?
Manually add boundaries Use the knowledge that a
record usually starts with a name This will
measure the value of boundary information.
17
Example 2 Segmented
msM ffi , 113 2lst Street . Wholesale and
Retail dealers in all grade of Cook and Heating
stoves . Also the celebra ted Jno , Vann , Wm .
Kiser and Char ter OakBangea . 1S4 K . L . POLK
CO . ' S Coker Henry W , dry goods and
grocer , Washington ave bet Hiekman and Maple ,
Irondale Coke Newton , lab Aliee Furnace
Coker Wesley T , elk W R Coker , res
Washington ave cor Hiekman , Irondale Coker
Wm R , dry good . and grocer , Washington ave ,
cor Hiekman , Irondale C ' ola Carlo , lunch
house . 19 20th s , res same Colby Mips Alice
, bds 280 ! ) 2d ave ColdirrU see also .
' aldircll Coldwell K A , bds 22 50 4th ave
Coldwell Wm ,
Birmingham City Directory
18
Example 2 Segmented (Ideal)
Method Correct Precision Recall F1 Dictionary 31
/ 78 51.67 39.74 44.93 EO 47 /
78 72.31 60.26 65.73 Segmented 65 /
78 95.59 83.33 89.04
Correct matches Coker Henry W Coke Newton
Coker Wesley T Coker Coker Wm R C ' ola
Carlo Colby Mips Alice Coldwell K A and
others Missed entities Wm Hood ) orter
II Herxfcld ( t \ \ ' Bains j gt orter
Tompson Francis iV Cheiiovveth Cole Wm C
Cole J L Davis and others
False positives ColdirrU see also Cole
Wm C ( Reamer Cole Co )
Results measured on set-aside TRAINING data
19
Example 2 Segmented EO (Ideal)
Method Correct Precision Recall F1 Dictionary 31
/ 78 51.67 39.74 44.93 EO 47 /
78 72.31 60.26 65.73 Segmented 65 /
78 95.59 83.33 89.04 Seg.EO 66 /
78 86.84 84.62 85.71
Correct matches Coker Henry W Coke Newton
Coker Wesley T Coker Coker Wm R C ' ola
Carlo Colby Mips Alice Coldwell K A and
others Missed entities ) orter II
Herxfcld ( t \ \ ' Bains j gt orter Tompson
Francis iV Cheiiovveth Cole Wm C Cole J L
Davis E K Fulton and others
False positives ColdirrU see also Cole
Wm C ( Reamer Cole Co ) Peter Zins Peter
Zins Alice Furnace Whilden A Alice Furnace
Alice Furnace Loo M
Results measured on set-aside TRAINING data
20
EO Results Example 3
4 Index to the Precinct ReSisters of Tuolumne
County i Index to the Precinct Registers of
Tuolumne County 5 Name co Address No Name C
Address No Name ICo Address No Name Address No
Gore Asihford Gray lohn A G alt John Goss An lrew
Gerken Ilerman 11 Getchell Everett G Gallup Will
Seneca Gr ndl W illiam Ilarten J osep IIenderson
Robert Harper Edwin F Hall
21
EO Results Example 3
Method Correct Precision Recall F1 Dictionary 4 /
72 19.05 5.56 8.60
Correct matches Kelly Martin Leslie
Christopher Fred Madison Charles Albert
Marconi Frank Frederick Missed entities Gore
Asihford Gray lohn A G alt John Goss An
lrew Gerken Ilerman 11 Getchell Everett G
Gallup Will Seneca Gr ndl W illiam Ilarten J
osep IIenderson Robert and others
False positives John Goss Getchell Everett
Robert Harper Edwin Robert Barkley Thomas
Michael Thomas Douglas Frank Hayes William
George Jordan Joh Kelly Patrick Rufus
Clifton Kurr and others
Results measured on set-aside TRAINING data
22
EO Results Example 3
Method Correct Precision Recall F1 Dictionary 4 /
72 19.05 5.56 8.60 EO 10 /
72 41.67 13.89 20.83
Correct matches Getchell Everett G Harper
Edwin F Hayes William George Kelly Patrick V
Kelly Martin Kessler Peter Frederick Lang
Charles Lewis Leslie Christopher Fred Madison
Charles Albert Marconi Frank Frederick Missed
entities Gore Asihford Gray lohn A G alt
John Goss An lrew Gerken Ilerman 11 Gallup
Will Seneca Gr ndl W illiam Ilarten J osep
IIenderson Robert and others
False positives John Goss Robert Barkley
Thomas Michael Thomas Douglas George P Harp
Jordan Joh Rufus Clifton Forrest Lumsden
Paul B Christian F Morrison Robert David V
Meiser Frederick S John Felix
Results measured on set-aside TRAINING data
23
EO Results Example 3
Method Correct Precision Recall F1 Dictionary 4 /
72 19.05 5.56 8.60 EO 10 /
72 41.67 13.89 20.83 New lines 62 /
72 73.81 86.11 79.49
Correct matches Gore Asihford Gray lohn A
G alt John Goss An lrew Getchell Everett G
Gallup Will Seneca Gr ndl W illiam Ilarten J
osep and others Missed entities Gerken
Ilerman 11 HIarlp39r C harles Frank
IIughes Jolhn .1 liughes Charles .1 Haill Ed
ward .1 Jordan Joh nAlfred 1honalp I Lind say
Alexalnder and others
False positives Name co Address No Name C
Address No Name ICo Address No Name Address No
Gerken Ilerman HIarlp IIughes Jolhn liughes
Charles Haill Ed ward Jordan Joh nAlfred Bi
ak Flat and others
Pattern 1 Alphabetical data up to new lines, all
columns
Results measured on set-aside TRAINING data
24
EO Results Example 3
Method Correct Precision Recall F1 Dictionary 4 /
72 19.05 5.56 8.60 EO 10 /
72 41.67 13.89 20.83 New lines1 62 /
72 73.81 86.11 79.49 New lines2 62 /
72 87.32 86.11 86.71
Correct matches Gore Asihford Gray lohn A
G alt John Goss An lrew Getchell Everett G
Gallup Will Seneca Gr ndl W illiam Ilarten J
osep IIenderson Robert and others Missed
entities Gerken Ilerman 11 HIarlp39r C
harles Frank IIughes Jolhn .1 liughes Charles
.1 Haill Ed ward .1 Jordan Joh nAlfred
1honalp I Lind say Alexalnder and others
False positives Gerken Ilerman HIarlp
IIughes Jolhn liughes Charles Haill Ed ward
Jordan Joh nAlfred Lind say Alexalnder i C
urpliy PaI Mocalrtoe lhester lioujalin i
Pattern 2 Alphabetical data up to new
lines, selected columns
Results measured on set-aside TRAINING data
25
EO Results Example 3
Method Correct Precision Recall F1 Dictionary 4 /
72 19.05 5.56 8.60 EO 10 /
72 41.67 13.89 20.83 New lines1 62 /
72 73.81 86.11 79.49 New lines2 62 /
72 87.32 86.11 86.71 New lines3 45 /
72 84.91 62.5 72.00
Correct matches Gore Asihford G alt John
Getchell Everett G Gr ndl W illiam IIenderson
Robert Hall CharlesPs rry Ilartoen Thomas
Douglas Hayes William George and others
Missed entities Gray lohn A Goss An lrew
Gerken Ilerman 11 Gallup Will Seneca Ilarten
J osep Harper Edwin F liarlan Robert
Barkley and others
False positives Gerken Ilerman HIarlp'r C
harles Frank liughes Charles . Haill Ed ward
. Jordan Joh nAlfred Lind say Alexalnder i C
urpliy PaI'trick Mocalrtoe lhester lioujalin i
Pattern 3 All data up to new line or
digit, selected columns
Results measured on set-aside TRAINING data
26
Context Exclusion

• Name not part of something else
• Address Hollis Long Island N Y
• Schools Trenton State Normal School
• Companies K. L. Polk Co.
• Note It might be interesting to recognize these
  names too, but mark them as being part of
  something else.

27
EO Results Example 4
I GERTRUDE SMITH (Mrs William E Haines deceased)
Married shortly after graduation Died at age of
22 Was musician and taught piano lessons 1898
HOBART L BENEDICT Millburn Essex County N J
Graduated from Rutgers 1902 and from New York
Law School in 1904 with degrees of B Sc M Sc and
LL B Married April 9 1907 to Martha C Bunnell
One daughter Elizabeth Benedict Counsellor at
law with offices in Elizabeth and Millburn MARTHA
BUNNELL (Mrs Hobart L Benedict) Millburn Essex
County N J Married to Hobart L Benedict on date
above 1899 CORA SMITH (Mrs Louis Slingerland)
557 Third St South St Peters- burg Florida
Married
28
Example 4 - Dictionary
Method Correct Precision Recall F1 Dictionary 6 27
.27 28.57 27.91
Correct matches Elizabeth Benedict MARTHA
BUNNELL Mrs Louis Slingerland Louis
Slingerland Mrs Harry Engel Mrs Leslie
Ward Missed entities GERTRUDE SMITH Mrs
William E Haines HOBART L BENEDICT Martha C
Bunnell Mrs Hobart L Benedict Hobart L
Benedict CORA SMITH Mr Slingerland JENNIE
HAINES STELLA ILLSLEY WALTER BOSCHEN GEORGE
McQUAIDE MARGARET HAINES ABBY HEADLEY
CLARENCE GRIGGS
False positives Mrs William York Law School
Mrs Hobart High School Mr Slingerland Ave
Union Union School Trenton Looker School
Union Town Hollis Long Island Morris Ave
Union Battin High School Rutgers College
Ave Union Ave Union Trenton State School
Roselle
Results measured on TRAINING data
29
Example 4 - Tuned
Method Correct Precision Recall F1 Dictionary 6 27
.27 28.57 27.91 Tuned EO 18 94.74 85.71 90.00

Correct matches GERTRUDE SMITH Mrs William
E Haines Martha C Bunnell Elizabeth Benedict
MARTHA BUNNELL Mrs Hobart L Benedict CORA
SMITH Mrs Louis Slingerland Louis
Slingerland Mr Slingerland JENNIE HAINES
STELLA ILLSLEY Mrs Harry Engel GEORGE
McQUAIDE MARGARET HAINES ABBY HEADLEY Mrs
Leslie Ward CLARENCE GRIGGS
Missed entities HOBART L BENEDICT Hobart L
Benedict WALTER BOSCHEN False positives
Hollis Long

• Without exclusion 18 85.71 85.71 85.71
• Additional false positives
• Morris Ave
• Trenton State

Results measured on TRAINING data
30
Example 4 Broadened
Method Correct Precision Recall F1 Dictionary 6 27
.27 28.57 27.91 Tuned EO 18 94.74 85.71 90.00
Broadened 12 57.14 57.14 57.14
Correct matches Mrs William E Haines HOBART
L BENEDICT Martha C Bunnell Elizabeth
Benedict MARTHA BUNNELL Mrs Hobart L
Benedict Hobart L Benedict Mrs Louis
Slingerland Louis Slingerland Mr Slingerland
Mrs Harry Engel Mrs Leslie Ward
Missed entities GERTRUDE SMITH CORA SMITH
JENNIE HAINES STELLA ILLSLEY WALTER BOSCHEN
GEORGE McQUAIDE MARGARET HAINES ABBY HEADLEY
CLARENCE GRIGGS False positives Hollis Long
Island N Y Union N J Springfield N J Union
N J Elizabeth N J Newark N J Union N J
Newark N J
Candidates for exclusion (place names)
Results measured on TRAINING data
31
Example 4 Patterns
Method Correct Precision Recall F1 Dictionary 6 27
.27 28.57 27.91 Tuned EO 18 94.74 85.71 90.00
Broadened 12 57.14 57.14 57.14 Pattern 13 10
0.00 61.90 76.47
Correct matches GERTRUDE SMITH MARTHA
BUNNELL Mrs Hobart L Benedict CORA SMITH
Mrs Louis Slingerland JENNIE HAINES STELLA
ILLSLEY Mrs Harry Engel WALTER BOSCHEN
MARGARET HAINES ABBY HEADLEY Mrs Leslie Ward
CLARENCE GRIGGS
Missed entities Mrs William E Haines HOBART L
BENEDICT Martha C Bunnell Elizabeth Benedict
Hobart L Benedict Louis Slingerland Mr
Slingerland GEORGE McQUAIDE False
positives None
Patterns ALL CAPS, multiple words of multiple
letters Initial capitals inside of parentheses
Results measured on TRAINING data
32
Extraction Ontologies circa May 09

• May 09 version of the EO engine
• OCR problems reduce potential accuracy
• Can tune rules, potentially even for OCR, but
• Summer 09 observations
• Document patterns can help
• But must apply judiciously
• Discover Pattern (possibly more than one
  simultaneously)
• Discover Extent of Pattern

33
Future Work

• Test-set trials for EO
• Code backend conversion to work with Thomass
  evaluator
• Tune expressions, as needed, on training set,
• Run trials
• Revise EO engine for patterns
• Pattern Discovery
• Extract with dictionary
• Test for patterns
• Internal patterns such as all caps or name order
• External patterns such as within parens, bounded
  by
• Pattern-Extent Discovery
• Pattern sequence begin and end
• Multiple patterns within the same extent