Babies, Variables, and Relational Correlations

1
Babies, Variables, and Relational Correlations
Cognitive Science Conference, Aug. 14, 2000
Michael Gasser, Eliana Colunga Cognitive Science,
Computer Science Indiana University
2
Overview

• Two pattern-learning tasks (Saffran et al.,
  Marcus et al.)
• Regularity in patterns content-specific and
  relational correlations
• Hebbian learning of pattern regularity of both
  types

3
Saffran, Aslin Newport

• Training syllable sequence with repeated
  3-syllable subsequences (words)
• bi da ku pa do ti go la bu bi da ku go la bu pa
  do ti tu pi ro ...
• Test word or non-word syllable triples
• pa do ti g familiar
• do ti go g unfamiliar

4
Saffran et al. (2)

• The babies apparently learned syllable-to-syllable
  transition probabilities. They distinguished
• probabilities between syllables within words and
• probabilities between syllables across word
  boundaries.

5
Marcus, Vijayan, Bandi Rao Vishton

• Training 3-syllable sequences embodying a
  grammatical rule AAB, ABB, or ABA
• li li we de de ji je je wi
• Test sequences of novel syllables agreeing or
  not agreeing with training rule
• ga ga po g familiar
• ko ga ga g unfamiliar

6
Marcus et al. (2)

• According to Marcus et al., the babies are making
  use of explicit variables they recognize the
  rule independent of the specific content of the
  elements.
• On this account, superficial similarity of a test
  sequence to the training sequences does not
  matter.

7
Simulating Saffran et al.

• Content-specific regularities (correlations) in
  patterns
• Pattern elements and correlations as units in a
  settling network trained with unsupervised
  Contrastive Hebbian Learning (Movellan)

8
Patterns

• Saffran et al. and Marcus et al. are
  pattern-learning tasks
• Patterns embody regularity based on element
  features/categories

pa do ti go la bu pa do ti ...
li li we de de ji je je wi ...
9
Content-Specific Pattern Regularities

• Detecting co-occurrences of elements by their
  content
• Keeping track of the recurrence of co-occurrences

pa do ti go la bu pa do ti ...
10
Pattern Regularities and Connectionist Networks

• Units represent elements.
• Units joining co-occurring element units
  represent co-occurrences (correlations),
  permitting learning of higher-order correlations.
• Hebbian learning captures the regularities.

11
Content-Specific Correlations in a Network
12
Simulation of Saffran et al. Network

• Continuous Hopfield network
• PATTERN units window of 3 consecutive syllables,
  one unit per syllable.
• CORRELATION units pairwise co-occurrences of
  syllables.

13
Simulation of Saffran et al. Network
14
Simulation of Saffran et al. Training

• PATTERN window slides across syllables in
  sequence of words (180).
• For each input pattern,
• PATTERN units clamped. Network settles.
• PATTERN units unclamped. Network settles.
• Weights adjusted through unsupervised Contrastive
  Hebbian Learning.

15
Simulation of Saffran et al. Testing

• Sequence of 3 syllables clamped on PATTERN units,
  either a word or 3 syllables spanning a boundary
  between words.
• Network settles.
• PATTERN units unclamped, network settles again,
  change in activations recorded measure of
  unfamiliarity of pattern.

16
Simulation of Saffran et al. Testing
17
Simulation of Saffran et al. Results

• Words (pa do ti)
• Activation change (unfamiliarity) 0.614
• Non-words (do ti go, ti go la)
• Activation change (unfamiliarity) 1.114
• Like the babies, the network is sensitive to the
  within-word and between-word transition
  probabilities.

18
Simulating Marcus et al.

• Relational regularities (correlations)
• Grouping of pattern elements by similarity
• Grouping (segmentation) in connectionist networks
• Representation of relational correlations
• Element representation and generalization

19
Relational Pattern Regularities

• Detecting co-occurrences of elements by their
  similarity
• Keeping track of the recurrence of co-occurrences

li li we de de ji je je wi ...
li li we de de ji je je wi ...
20
Relational Regularities and Grouping

• Finding relational regularities in a pattern
  involves grouping elements by similarity.

li li we ...
21
Grouping and Segmentation

• Grouping is one form of segmentation.
• Segmentation is based on proximity, common fate,
  and within-region (group) similarity.

22
Segmentation by Similarity
23
Segmentation and Binding

• A mechanism for segmentation must solve the
  binding problem.
• Activation of a collection of connectionist units
  represents only the presence of a set of
  features, not their binding.

24
Binding Dimension

• A common solution to the binding problem in
  connectionist networks is a separate binding
  dimension along which units vary (Shastri, Hummel
  Biederman, etc.).
• Units aligned on the binding dimension represent
  the same object units out of alignment
  represent different objects.

25
Segmentation and Alignment
26
Binding and Relations in PLAYPEN

• Binding dimension simple units have angles in
  addition to activation.
• A pair of units affects each others activation
  and angle through a single weight.
• Relational correlations SAMENESS and DIFFERENCE
  relation units, which are activated to the extent
  that their inputs agree/disagree in angle.

27
Relational Correlations in a Connectionist
Network (PLAYPEN)
28
Element Representation and Generalization

• Generalization behavior depends on how elements
  are represented.
• Coarse-coded representations based on element
  features permit generalization by similarity.
• Different degrees of coarseness permit more and
  less abstract generalizations.

29
Local and Coarse-Coded Element Representation
30
Specific and Abstract Correlations
31
Simulation of Marcus et al. Network

• PATTERN units syllable sequences
• Distributed syllable representations
• PATTERN unit angles grouping of elements within
  sequences, done prior to presentation to network
• CORRELATION units pairwise relational
  correlations between pattern element units

32
Simulation of Marcus et al. Network
33
Simulation of Marcus et al. Training

• Each syllable sequence presented 3 times.
• For each pattern,
• PATTERN units clamped. Network settles.
• PATTERN units unclamped. Network settles.
• Weights adjusted through unsupervised Contrastive
  Hebbian Learning.

34
Simulation of Marcus et al. Testing (1)

• Four types of test syllable sequences
• Grammatical, familiar li li de
• Grammatical, partially familiar la la da
• Grammatical, unfamiliar ga ga po
• Ungrammatical li de li, ga po po

35
Simulation of Marcus et al. Testing (2)

• Syllable sequence clamped on PATTERN units.
• Network settles.
• PATTERN units unclamped, network settles again,
  change in activations recorded.

36
Simulation of Marcus et al. Results
37
Simulation of Marcus et al. Results

• As in original experiment, ungrammatical
  sequences are less familiar than grammatical
  sequences composed of relatively novel syllables.
• Within grammatical sequences, similarity of
  syllables to training syllables affects
  familiarity. Content still matters.

38
Summary

• Content-specific and relational regularities can
  both be extracted by an associative learning
  device.
• Recognizing relational regularities begins with
  grouping of elements within groups.
• Relational correlations take the form of explicit
  relation units.

39
Predictions

• The relational correlational account makes
  different predictions than the symbolic account
  using variables.

40
Prediction 1

• Performance should be affected by factors
  influencing grouping .

li li we de de ji je je wi ...
The rule is more difficult to learn in these cases
li liwe de deji je jewi ...
41
Prediction 2

• Similarity to the training set still matters.
  Relation units are not variables.

42
Outstanding Issues

• If grouping (segmentation) is the key, how is it
  learned?
• How does segmentation in the Saffran et al. task
  relate to within-subsequence grouping in the
  Marcus et al. task?
• How does the likelihood of detecting SAMENESS and
  DIFFERENCE depend on the whole pattern?