Practical Natural Language Processing

1
Practical Natural Language Processing

• CPSC 533 Artificial Intelligence
• Caroline Hrouda Marilena Rossi
• April 6, 2000

2
Contents

• Scaling Up the Grammar
• Ambiguity
• Discourse Understanding

3
Scaling up the Grammar

• To make sense of a real-life language it requires
  much more sophistication at every step of the
  language interpretation process
• Grammar - the study of the classes of words,
  their inflections, and their functions and
  relations in the sentence.

4
Nominal Compounds and Apposition

• Nominal Compounds - strings of nouns combining to
  form a larger unit that still can combine with an
  article to form an NP.
• For the larger noun unit we need a rule
• Rule ? Noun -gt NounNoun
• NounNounNoun POSTSCRIPT language code
  Noun input file
• Nominal Compound - file input
• Rule ? ?f ?i Input(i) /\ File(f) /\ NN(i,f)
• Achieved by the rule
• Rule ?
• Noun (?y ?x sem1(x) /\ sem2(y) /\ NN(x,y)) -gt
  Noun(sem1)Noun(sem2)

5

• Apposition - a construction of two noun phrases
  concatenated together in which both noun phrases
  refer to the same thing
• i.e.
• Restrictive Apposition - restricts the set of
  possible references, thus to make sure that one
  is not confused with something else
• i.e. David MacDonald the insane professor
• A simplified rule for apposition is
• Rule ? NP(q x sem1 /\ sem2) -gt NP(q x
  sem1)NP(q x sem2)

6
Adjective Phrases

• Adjective - serves as a modifier of a noun to
  denote a quality of the thing named, to indicate
  its quantity or extent, or to specify a thing as
  distinct from something else.
• Intersective Semantics - formed by a conjunction
  of the semantics contributed by the adjective and
  by the noun.
• the foot is stinky or the stinky foot
• ?w Stinky(w) /\ Foot(w)
• If all adjectives were with intersective
  semantics then
• Rule ? Noun (?x sem1(x) /\ sem2(x)) -gt
  Adjective(sem1)Noun(sem2)
• Note the semantic relation between adjective and
  noun is often more complicated than just
  intersection.

7
Determiners

• Determiner - a word belonging to a group of
  limiting noun modifiers characterized by
  occurrence before descriptive adjectives
  modifying the same noun.
• Articles are just one type of class determiner
  i.e. a, the
• A simple example
• Quasi-logical form 3x Sasquatch(x)
• Gives the following grammar rules
• Det(q) -gt Article(q)
• Det(q) -gt Number(q)
• NP(qx noun(x)) -gt Det(q) Noun(noun)

8
Noun Phrases

• Change Article to Determinant, include case
  information and agreement in person and number.
• Rule ?
• NP(case, Person(3), number, qx sem(x)) -gt
  Det(number, q) Noun(number, sem)
• - case variable is unbound - NP can be used in
  either subjective or objective case
• - number can be singular or plural, but rule
  says Det and Noun must have same number
• ( There are exceptions Det the or Noun
  sheep, can be singular or plural)
• To enforce subject / verb agreement
• Rule ? S(rel(obj)) -gt NP(Subject, person, number,
  obj) VP(person, number, rel)
• i.e. I am vs. We am

9

• It is possible to form a NP from a noun with no
  determiner
• Alice ate is ?e e ? Eat(Alice, Past)
• Since many things can be Alice
• ?e, x e ? Eat(? ! x Name(x) Alice, Past)
• Rule ?
• NP(case, Person(3), number, ? ! x Name(x)
  name)
• -gt Name(number, name) Name(Singular,
  Alice)-gtAlice

10
Clausal Complements

• All verbs have taken only noun phrases and
  prepositional phrases as complements, but some
  verbs accept clauses.
• Clause - a group of words containing a subject
  and predicate and functioning as a member of a
  complex or compound sentence.
• The same subcategorization mechanism from before
• VP(subcat) -gt VP(Ssubcat)S
• VP(subcat) -gt VP(VPsubcat)VP
• Verb(S) -gt believe ( I believe he has left
  )
• Verb(VP) -gt want ( I want to go there )
• Infinitive - a verb normally identical in
  English with the first person singular that
  performs some functions of a noun and at the same
  time displays some characteristics of a verb.

11
Relative Clauses

• Gap - _ symbol where it indicates the place where
  the head noun phrase (the person) would logically
  appear to complete the sentence the person that
  I saw _
• Filler - the head noun phrase is the filler of
  the gap
• the person that I looked pp at _
• Long distance dependency - filler gap relation
  that reaches down a potentially unbounded number
  of nodes into the parse tree
• the personi that s you said s you thought
  s I gave the book to _ i
• i on parse nodes used to show that there is an
  identity relationship
• person same as the recipient of the book

12

• Relative Clauses - an NP can be modified by
  following it with a relative clause. A relative
  clause consists of a relative pronoun followed by
  a sentence that contains a NP gap. the person
  that I saw _
• Rule ? NP(Gap) -gt NP(Gap) RelClause
• RelClause -gt Pronoun(Relative)S(Gap(NP))
• ? string comprises an NP with an NP Gap in it.
• Rule ? NP(Gap(NP)) -gt ?
• The Gap has to be passed along in the rest of
  the grammar
• i.e. S(Gap(Concat(g1 , g2))) -gt
  NP(Gap(g1))VP(Gap(g2))
• If g1 , g2 both are Gaps the S as a whole has no
  Gap

13
Questions? huh?

• In English there are two main types of questions
• 1) Yes / No - Did you do that?
• Subject - aux inversion - like a declarative
  sentence but it has an auxiliary verb that
  appears before the subject NP. Sinv is to denote
  a sentence that has it.
• Auxiliary - functioning in a subsidiary capacity
  of a verb, accompanying another verb and
  typically expressing person, number, mood, or
  tense.
• Rule ? S -gt Question
• Question -gt Sinv
• Sinv -gt Aux NP VP

14

• 2) Wh (gapped) - What did you see _?
• Will expect a noun phrase as an answer. It is an
  interrogative pronoun followed by a gapped Sinv
  (in the simplest case)
• Interrogative Pronoun - who, what, where, when,
  why and how
• Rule ? Question -gt Pronoun(Interrogative)
  Sinv(Gap(NP))
• Other question constructions, but less common
• 1) Echo - You did what?
• 2) Rising Intonation - You smell something?
• 3) Yes / No with be - Is it dead?
• 4) Wh Subject - When is this class over, Prof.
  Jacob?
• 5) Wh NP - What trashy novel did you read _
  ?
• 6) Wh PP - With what did you write it _ ?

15
Ambiguity

• Handling agrammatical strings
• Syntactic Evidence
• Lexical Evidence
• Semantic Evidence

16

• Difficult to find the correct interpretation,
  especially if one can only use lexical,
  syntactic, semantic rules.
• Try to use logical inference through
  probabilistic models, such as belief networks and
  hidden Markov models
• Belief networks help determine how to combine
  lexical, syntactic, semantic evidence.
• Difficulty lies in selecting the appropriate
  evidence and how to implement it.
• Thus, its very important to note the difference
  between the evidence and source of ambiguity.

17
Syntactic Evidence

• Source of ambiguity adverbs and prepositional
  phrases (a.k.a. modifiers) can be applied to many
  different heads.
• Adverb - a word belonging to one of the major
  form of classes, typically serving as a modifier
  of a verb, an adjective, another adverb, a
  preposition, a phrase, a clause , or a sentence
  and expressing some relation of manner or
  quality, place or time.
• Solution concluding that the modifier should be
  applied to the most recent head.
• i.e. I walked through the sludge near my house.
• near my house can be applied to both I and
  the sludge. According to the solution, apply
  it to the sludge.

18
Lexical Evidence

• Source of ambiguity syntax of prepositional
  phrases modify verb and nouns in sentence,
  changing the entire meaning of the phrase.
• I.e. Lee positioned the dress on the rack.
• Kim wanted the dress on the rack.
• On the rack determines the interpretation of
  the sentence in the first sentence, it affects
  the verb in the second sentence, it affects the
  noun dress.
• Solution preference of the verb for
  sub-categorisation.

19
Semantic Evidence

• Source of ambiguity lexical ambiguity where the
  favoured word sense alters the context of the
  sentence.
• i.e. Ball, diamond, bat, base implies baseball,
  but the words individually have a more common
  different meaning.

20
Semantic Evidence

• Another common ambiguity words associated with
  the word with, where with can have many
  different meanings and related noun phrases.
• Sentence Relation
• I ate macaroni with ketchup (ingredient of
  macaroni)
• I ate macaroni with dessert (side dish of
  macaroni)
• I ate macaroni with abandon (manner of macaroni
  eating)
• I ate macaroni with a chopstick (instrument of
  difficult eating)
• I ate macaroni with my dog (accompanier of
  macaroni eating)
• Solution have the interpretations that refer to
  the most likely events, although the correct
  interpretation must still be sought out.

21
Metonymy

• Metonymy - a figure of speech where one object is
  used to represent another.
• Its a frequent occurrence in spoken language and
  difficult to represent grammatically.
• i.e. Microsoft announced a loss of 17 billion
  dollars.
• We know that Microsoft really is a spokesperson
  for the company Microsoft.

22
Metonymy

• Solution need to represent a new level of
  ambiguity to represent the new semantics.
• Provide two objects for the semantic
  interpretation
• one for the object the phrase literally refers
  to
• one for the metonomic reference.
• Then state that there is a relation between the
  two objects.

23
Metonymy

• In current grammar
• ? x,e Microsoft(x) ? e ? Announce(x,Past).
• Needs to be altered to
• ? m,x,e Microsoft(x) ? Metonymy(m,x) ? e e
  Announce(m,Past).
• This is only a representation of the problem.
  Need to define constraints for the metonymy
  relation.

24
Metonymy

• Case 1 No Metonymy, where x and m are
  identical
• ? m, x (mx) --gt Metonymy (m,x)
• Case 2 Representational reference for an
  organisation
• ? m, x Organisation(x) ? Spokesperson(m,x) -gt
  Metonymy (x,m)

25
Metonymy

• Other examples
• referring to an author for his/her works
• referring to a producer instead of the product
• referring to a group name for the whole (i.e. a
  team)
• slang

26
Metaphor

• Metaphor - a figure of speech where a phrase with
  one literal meaning is used to suggest another
  meaning through analogy.
• Has a large part in everyday language, not just
  poetry.
• i.e. The system has crashed.

27
Metaphor

• Solution 1 define all known/common metaphors
  into lexicon, adding a new sense to the meaning
  of words (i.e. fallen, dipped refer to some other
  scale other than altitude).
• Note that this doesnt necessarily yield the
  correct interpretation of the sentence
• Solution 2 include explicit knowledge of common
  metaphors and use it to interpret a new sense to
  the sentence.

28
Discourse Understanding

• Discourse or Text - is any string of language,
  usually one that is more than one sentence long.
  Requires an understanding of text longer than one
  sentence.
• Easier to break down language into individual
  sentences, but need to grasp the relations
  between all the sentences in a given text.
• Discourse produced through three steps
• intention
• generation
• synthesis

29
Discourse Understanding

• Discourse understanding is done through
• perception
• analysis (semantic, syntactic, lexical)
• disambiguation
• incorporation
• This all depends on the amount of knowledge that
  an agent has. Two different knowledge bases will
  lead to two different understandings of a text.

30
Discourse Understanding

• General Discourse Equation
• KB Discourse Understanding (text, KB)
• The two versions of KB being
• KB knowledge base of the agent
• KB agents knowledge after understanding the
  text.

31
Discourse Understanding

• Six types of knowledge to achieve understanding
• 1) General knowledge about a word
• 2) General knowledge about the structure of
  coherent discourse
• 3) General knowledge about syntax and semantics
• 4) Specific knowledge about situation being
  discussed
• 5) Specific knowledge about beliefs of the
  characters
• 6) Specific knowledge about the beliefs of the
  speaker

32
Discourse Understanding

• Interpretation tends towards a priori knowledge
  of meaning.
• Lets look at an example discourse
• Pete went to a car race.
• He shouted very loudly.
• He had to leave at 6pm.
• There can be many interpretations to this
  discourse due to the hearers knowledge base.

33
Structure of Coherent Discourse

• First, conjunction is NOT commutative in natural
  languages
• For example
• Dr. Monroe went golfing. Dr.Monroe went golfing.
• It started to rain. He was struck by lightning.
• He was struck by lightning. It started to rain.
• temporal ordering important
• purpose is important

34
Structure of Coherent Discourse

• Segment - clause, complete sentence, group of
  consecutive sentences. Discourses are composed of
  segments.
• Coherence relation - each segment in a discourse
  is related to a previous segment and determines
  the role of each segment in the discourse.
• The hearer must discern the relations of
  segments, not just ascertain the ambiguities.
• Coherence relations constrain the possible
  meanings of each sentence (i.e. single sentences
  have many meanings, but together only a few).

35
Structure of Coherent Discourse

• Hobbs Theory - a speaker does four things to
  make discourse
• 1) Convey message
• 2) Has motivation or goal to do number one.
• 3) Wants to make it easy to understand the
  message.
• 4) Links information to what hearer already
  knows.
• A sentence is a coherent extension to discourse
  if it does one of the four points above.

36
Structure of Coherent Discourse

• 1) A funny thing happened yesterday
• 2) Wendy went to a fast food restaurant
• 3) Wendy is a sandwich
• 4) The clerk said we dont serve food here
• 5) Wendy was shocked and hurt
• 6) The clerk said theyd make an exception this
  time
• 7) She was very embarrassed by her forgetfulness
• Thus, 2 adjacent segments si and sj stand in the
  evaluation coherence relation if one can infer
  from si that sj is a step in the speakers plan
  to achieve a discourse goal.

37
Structure of Coherent Discourse

• Different types of Coherence Relations
• 1) Evaluation
• 2) Enablement
• 3) Causal
• These come from the speakers goal.
• Therefore, understanding has 2 levels of plan
  recognition
• the speakers plans and the characters plans in
  the discourse.

38
Structure of Coherent Discourse

• Other Coherent Relations
• elaboration, used by the speaker to make
  discourse easier to understand by saying
  something differently
• explanation, where the speaker adds new details
  to the hearers existing knowledge to help
  understand the discourse at hand.
• A more elaborate set of coherence relations was
  developed by Mann and Thompson solutionhood,
  evidence, justification, motivation, reason,
  sequence, enablement, elaboration, restatement,
  condition, circumstance, cause, concession,
  background, thesis-antithesis.

39
Structure of Coherent Discourse

• Grosz and Sidners theory notes the where the
  attention is focussed during a discourse by the
  speaker and the hearer.
• Attention/focus varies by what segment is added
  on (pushed) and removed (popped) from the stack
  and when.
• This alters the direction of the focus.

40
Structure of Coherent Discourse

• Discourse (A)
• I went to Edmonton
• I bought you Perogies
• Then I hitch-hiked home
• I went to K-Mart
• I bought some underwear

Discourse (B) I went to Edmonton Then I
hitch-hiked home I went to K-Mart I bought you
Perogies I bought some underwear
41
Chit - Chat with Chatterbot

• Go to Chatterbot!

42
Summary

• 1) NLP techniques make it practical to develop
  programs that make queries, extract information
  from texts, translate and so on.
• 2) It is possible to parse sentences efficiently
  using an algorithm.
• 3) There has been a shift from grammar to the
  lexicon.
• 4) Natural languages have a huge variety of
  syntactic forms.
• 5) Choosing the right interpretation requires
  evidence from many sources.
• 6) Interesting language comes from connected
  discourse rather than in isolated sentences.