The Emergentist Approach To Language As Embodied in Connectionist Networks

1
The Emergentist Approach To LanguageAs Embodied
in Connectionist Networks

• James L. McClelland
• Stanford University

2
Some Simple Sentences

• The man liked the book.
• The boy loved the sun.
• The woman hated the rain.

3
The Standard ApproachUnits and Rules

• Sentences
• Clauses and phrases
• Words
• Morphemes
• Phonemes

• S -gt NP VP
• VP-gt Tense V (NP)
• Tense V -gt Vpast
• V -gt like, love, hate
• N -gt man, boy, sun
• man -gt /m/ /ae/ /n/
• past -gt ed
• ed -gt /t/ or /d/ or /d/
• depends on context

4
What happens with exceptions?
5
Standard Approach to the Past Tense

• We form the past tense by using a (simple) rule.
• If an item is an exception, the rule is blocked.
• So we say took instead of taked
• If youve never seen an item before, you use the
  rule
• If an item is an exception, but you forget the
  exceptional past tense, you apply the rule
• Predictions
• Regular inflection of nonce forms
• This man is blinging. Yesterday he
• This girl is tupping. Yesterday she
• Over-regularization errors
• Goed, taked, bringed

6
The Emergentist Approach

• Language (like perception, etc) arises from the
  interactions of neurons, each of which operates
  according to a common set of simple principles of
  processing, representation and learning.
• Units and rules are useful to approximately
  describe what emerges from these interactions but
  have no mechanistic or explanatory role in
  language processing, language change, or language
  learning.

7
An Emergentist TheoryNatural Selection

• No grand design
• Organisms produce offspring with random
  differences (mating helps with this)
• Forces of nature favor those best suited to
  survive
• Survivors leave more offspring, so their traits
  are passed on
• The full range of the animal kingdom including
  all the capabilities of the human mind emerge
  from these very basic principles

8
An Emergentist/Connectionist Approach to the
Past Tense

• Knowledge is in connections
• Experience causes connections to change
• Sensitivity to regularities emerges

9
The RM Model

• Learns from verb root, past tense pairs
• Like, liked love, loved carry, carried
  take, took
• Present and past are represented as patterns of
  activation over units that stand for phonological
  features.

10
Examples of wickelfeatures in the verb baked

• Starts with a nasal followed by a vowel
• Has a long vowel preceded by a nasal and followed
  by a stop
• Ends with a dental stop preceded by a velar stop
• Ends with an unvoiced sound preceded by another
  unvoiced sound

11
(No Transcript)
12
A Pattern Associator Network
Pattern representing sound of the verbs past
tense
Matrix of connections
p(a1)
Pattern representing the sound of the verb root
13
Learning rule for the Pattern Associator network

• For each output unit
• Determine activity of the unit based on its
  input.
• If the unit is active when target is not
• Reduce each weight coming into the unit from each
  active input unit.
• If the unit is inactive when the target is
  active
• Increase the weight coming into the unit from
  each active input unit.
• Each connection weight adjustment is very small
• Learning is gradual and cumulative

14
Some Learning Experiments

• Learn a single item
• Test for generalization
• Learn from a set of regular items
• Test for generalization

15
(No Transcript)
16
Over-regularization errors in the RM network

• Most frequent past tenses in English
• Felt
• Had
• Made
• Got
• Gave
• Took
• Came
• Went
• Looked
• Needed

Heres where 400 more words were introduced
Trained with top ten words only.
17
Over-regularization simulation in the 78 net

• First learn one exception
• Then continue training the exception with all
  other forms
• What happens to the exception?

18
Questions?
19
Some features of the model

• Regulars co-exist with exceptions.
• The model produces the regular past for most
  unfamiliar test items.
• The model captures the different subtypes among
  the regulars
• like-liked
• love-loved
• hate-hated

• The model is sensitive to the no-change pattern
  among irregulars
• hit-hit
• cut-cut
• hide-hid

20
Additional characteristics

• The model exploits gangs of related exceptions.
• dig-dug
• cling-clung
• swing-swung

• The regular pattern infuses exceptions as well
  as regulars
• say-said, do-did
• have-had
• keep-kept, sleep-slept
• Burn-burnt
• Teach-taught

21
Key Features of the Past Tense model

• No lexical entries and no rules
• No problem of rule induction or grammar selection

22
Strengths and Weaknesses of the Models
23
Elmans Simple Recurrent Network

• Task is to predict the next element of a sequence
  on the output, given the current element on the
  input units.
• Each element is represented by a pattern of
  activation.
• Each box represents a set of units.
• Each dotted arrow represents all-to-all
  connections.
• The solid arrow indicates that the previous
  pattern on the hidden units is copied back to
  provide context for the next prediction.
• Learning occurs through connection weight
  adjustment using an extended version of the error
  correcting learning rule.

24
Results for Elman net trained with letter
sequences
25
(No Transcript)
26
Hidden Unit Patterns for Elman Net Trained on
Word Sequences
27
Key Features of the Both Models

• No lexical entries and no rules
• No problem of rule induction or grammar selection