A Linguistic Solution to Perfecting Search Technology

1
A Linguistic Solution to Perfecting Search
Technology
A new tool to improve the filtering options in
advanced searching
Fernando Moreno-Torres General Manager MTC Soft
2
The Problem (I) different meanings
European Commission concrete
Results 967.000
European Commission - Research Industrial
technologies - Robot ... Concrete repair The
permanence of our built environment can easily be
taken for granted. Materials such as concrete,
that are used to construct our roads,
... Concrete tasks or activities contained in
White Paper Financial ... European Commission End
2006. 46) Concrete actions in mortgage credit
(follow-up White Paper) European Commission
2007and ... UNCTAD XI European Commission
calls for concrete action to assist ... The
European Commission will call for concrete global
measures to support commodity dependent
developing countries and their producers at the
11 ... European Commission Information
roadshow on the European ... European
citizenship not just words, concrete rights! -
Information roadshow organised by the European
Union. European Commission. ...
3
The Problem (II) no relations between words
Tom Cruise bought house
Results 169.000
Doctors not fans of Tom Cruise's baby gift -
Women's health- msnbc.com 6 Dec 2005 ... Tom
Cruise bought a sonogram machine so that he and
girlfriend Katie Holmes can see their unborn ....
Video From 'House' to White House ... He shows
them the money - Los Angeles Times 6 May 2007 ...
Despite speculation that Tom Cruise and Katie
Holmes are having marital problems, he just
bought a Beverly Hills home for his family that
cost ... The one-story, 13000-square-foot house
was built on speculation this year Tom Cruise
Buys 35 Million Beverly Hills Mansion - Katie
Holmes ... 7 May 2007 ... ft. structure in 1937,
the house was expanded four years ago and ...
have bought a multi-million-dollar house only two
minutes away. They plan to move in July. Tom
Cruise's 35 million Beverly Hills mansion Photo
by Tom Cruise Bought Katie Holmes An
Engagement Ring Right After ... Tom Cruise bought
wife Katie Holmes an engagement ring after their
first date. .... Charlotte Church Kicks Her
Boyfriend Out Of House After Heated Argument ...
4
The problem
Several-word searches
Common words

• Too many results, most of then not related to the
  query

Why?
Search engines dont discriminate different
meanings of words
Words are not related in the texts
5
The solution
Parsing and tagging the texts
- Discovering the real meaning of every word in
every sentence - Discovering the relations
between words

• The search engines can use this information
• to filter the results
• to get accurate results

• With these enriched texts we can give users new
  possibilities to filter the search using
  additional options.

6
The advanced search query
Who? Tom Cruise The sentence subject The actor
What does he do? To buy The sentence verb The action
Concerning what? House The sentence direct object The object
Where? In Hollywood (optional) The place
When? Last Month (optional) The time
7
The new query and the new results
Tom Cruise is the subject of a sentence in which
buy is the verb and house is the direct object
This query would find

• - Tom Cruise has bought a house
• - Tom Cruise and Katie Holmes will buy a new
  house .
• - Yesterday Tom Cruise bought two dogs and a red
  house

This query would not find
- Tom Cruise bought a dog and visited a house in
- Tom Cruise is very famous. Madonna has bought
an expensive house - Tom Cruise doesnt house
any friends, because he has bought a new car and
8
How did we make it?

• A 10-year project

• Based on the linguistic analysis from a
  philologist and professional translator, working
  for the European Commission for more than 20 years

• Its not Statistical Natural Language Processing
• Its not Artificial Intelligence software
• Its not the work of software engineers

• Its the core of our new Automatic Translator (to
  be released next May 20th)

• Its a design of a philologist, constructed by
  software developers, MTC Soft, over the last
  three years

We perform Word Sense Disambiguation using
linguistic knowledge
9
Where it comes from
PHASE
PHASE
PHASE
PHASE
1
4
3
2
ENGLISH
SPANISH
Symbol identification
Grammatical Analysis
Syntactic Analysis
Translation
VICTOR Translator
Parsalyser

• The translator has 4 main phases or modules.

The first 3, in English, analyze and tag every
word of the text. This is the Parsalyser.
10
How does it work?
PHASE
PHASE
PHASE
1
3
2
Symbol identification
Grammatical Analysis
Syntactic Analysis
140 Processes
Original text
500 linguistic rules

• Germany is supporting the development of
  infrastructure (especially energy and water
  supplies), promotion of the economy and
  employment, advising the PISG on the
  restructuring of administrative structures in the
  education and vocational training area and the
  improvement of the general economic climate.

The text is analyzed in every process
Every process does its work
- Word identification
- Capital letter identification
In just a few seconds
- How gets special treatment
- Could gets special treatment
-
Tagged text

• - Disambiguation
• Verb Noun
• Adjective Pronoun
• Verb Preposition

Germany is supporting the development of
infrastructure (especially energy and water
supplies), promotion of the economy and
employment, advising the PISG on the
restructuring of administrative structures in the
education and vocational training area and the
improvement of the general economic climate.
140 Processes
11
Phase 1 Symbol identification
PHASE
PHASE
PHASE
3
2
1
Symbol identification
Grammatical Analysis
Syntactic Analysis

• Identification of words, symbols and expressions
• Identification of root words (using our
  dictionary)
• Assignation of possible grammar functions

Original text

• Tom Cruise has just bought a house that costs 35
  million.

Words identified
12
Phase 2 Grammatical Analysis
PHASE
PHASE
PHASE
3
2
1
Symbol identification
Grammatical Analysis
Syntactic Analysis

• Disambiguation
• Performing an exhaustive linguistic analysis

Original text

• Tom Cruise has just bought a house that costs 35
  million.

Text after this phase
13
Phase 3 Syntactic Analysis
PHASE
PHASE
PHASE
3
2
1
Symbol identification
Grammatical Analysis
Syntactic Analysis

• Identification of the syntagms
• Both simple and complex
• Both autonomous and subordinate
• Identification of syntactic and semantic
  relations between them

Original text

• Tom Cruise has just bought a house that costs 35
  million.

Syntactic functions
Subject
Verb
Object
14
An example
PHASE
PHASE
PHASE
1
3
2
Symbol identification
Grammatical Analysis
Syntactic Analysis
Original text

• On the basis of an annual report from the
  Commission, it shall also examine the effects of
  the special arrangements with regard to drugs,
  including the progress in the fight against drugs
  made by countries listed in the second annex and,
  if progress is insufficient, the Commission will
  consider taking any measures to suspend in whole
  or in part the application of Article 7, in
  accordance with the established procedure in
  Article 32 and after consulting the country
  concerned.

15
An example
PHASE
PHASE
PHASE
3
2
1
Symbol identification
Grammatical Analysis
Syntactic Analysis
After Phase 1 Words identified
16
An example
PHASE
PHASE
PHASE
3
2
1
Symbol identification
Grammatical Analysis
Syntactic Analysis
After Phase 2 Words disambiguated
17
An example
PHASE
PHASE
PHASE
3
2
1
Symbol identification
Grammatical Analysis
Syntactic Analysis
After Phase 3 Syntactic functions
18
The Dictionary
Facultad de Traducción e Interpretación Granada
University
In collaboration with

• Dictionary Main Statistics
• 106.634 English words
• 31.417 Fixed expressions
• 9.587 Open expressions
• 20.268 Idiomatic verbs
• Grammar functions
• 12.558 Adjectives
• 4.388 Adverbs
• 47.150 Substantives
• 8.558 Verbs (2.590 phrasal verbs)
• 2.136 acronyms
• 2.217 incidental phrases

7 professors 24 students 4 months
Página 18
19
The Dictionary

• 10 Dictionaries

• Every one has its type of record

• Every word has its data

20
State of the Beta
1
Key words Indexes
1
3
2
90
Parser Tagger Parsalyser
2
Advanced Search Interface
15
3
15
21
Indexes
Key words Indexes
2
(Simplified example. In the real table, Words are
codified using the dictionary)
Word Grammar Syntactic Sentence File
  Function Function    
Tom Cruise 1 1 1 123
Tom Cruise 1 1 3 234
Tom Cruise 1 1 12 345
Tom Cruise 1 3 2 12
Tom Cruise 1 3 23 134
Tom Cruise 1 3 2 567
Tom Cruise 1 3 12 1.234
Tom Cruise 1 3 33 2.345
buy 1 1 3 1.233
buy 1 1 124 3.432
buy 1 2 44 4.333
buy 3 2 3 234
buy 3 2 23 134
buy 3 2 2 345
buy 3 3 34 4.322
house 1 2 3 234
house 1 2 21 1.456
house 1 3 3 234
house 3 3 12 4.444
File 234 Sentence 3
1 2 3
Grammar function Substantive Adjective Verb

1 2 3
Syntactic function Subject Verb Direct Object

22
User Interface
Advanced Search Interface
3

• There is a lot of work done in this area.
• We has not developed the interface.
• Just a simple window to test the system.
• Even so, it is very powerful

A first approach
Tom Cruise bought house
Only texts with the words related
X
Only texts with the words related
Just a simple click to filter most of the
results
23
User Interface
Advanced Search Interface
3
Advanced searches

• For advanced users, a more powerful interface
• Analyses the sentence
• Proposes grammar functions
• Proposes syntactic functions
• Allows declined forms for every word

It works!
24
Other uses
Plain texts

• Germany is supporting the development of
  infrastructure (especially energy and water
  supplies), promotion of the economy and
  employment, advising the PISG on the
  restructuring of administrative structures in the
  education and vocational training area and the
  improvement of the general economic climate.

Search Engines
Semantic Web
Data Mining
Statistical Analysis

25
Other uses
Tagged text
Plain texts
Germany is supporting the development of
infrastructure (especially energy and water
supplies), promotion of the economy and
employment, advising the PISG on the
restructuring of administrative structures in the
education and vocational training area and the
improvement of the general economic climate.

• Germany is supporting the development of
  infrastructure (especially energy and water
  supplies), promotion of the economy and
  employment, advising the PISG on the
  restructuring of administrative structures in the
  education and vocational training area and the
  improvement of the general economic climate.

The parser tagger can be used for a lot of
Internet related tasks
Search Engines
Semantic Web
Data Mining
Statistical Analysis

26
A new Standard?
Tagged text
Germany is supporting the development of
infrastructure (especially energy and water
supplies), promotion of the economy and
employment, advising the PISG on the
restructuring of administrative structures in the
education and vocational training area and the
improvement of the general economic climate.
The question is

• Why use plain texts to start the work with, If
  you can use enriched texts?

• Change the raw material at the beginning.

• These enriched texts are
• created automatically (no human work)
• and have an universal format (no standard to
  discuss)

27
Thank you for attending

• Fernando Moreno-Torres
• direccion_at_mtcsoft.es
• 34.609.575.000
• 34.958.215.280
• C/ Concepción, 47
• 18009 Granada (Spain)