Anaphoric dependencies: A window into the architecture of the language system Eye tracking experimen

1
Anaphoric dependencies A window into the
architecture of the language system
Eye tracking experiments
Eric Reuland Frank Wijnen Arnout Koornneef
2
EYE-TRACKING
BY ARNOUT KOORNNEEF
3
OVERVIEW OF LECTURE

• Part 1 discussion of method
• - eye-tracking while reading
• Part 2 discussion of current research
• - eye-tracking experiments

4
THE EYE-TRACKER
5
THE EYE-TRACKER
6
THE EYE-TRACKER

• monitors eye-movements from millisecond to
  millisecond
• provides information about where people look and
  for how long

7
TWO MAIN EYE-TRACKING PARADIGMS

• eye-tracking while reading
• eye-tracking while listening
• (not discussed)

8
EYE-TRACKING WHILE READING
9
SOME FACTS ABOUT READING

• people do not read a text smoothly, but fixate
  a particular word (200 - 300 msec) and jump to
  the next
• a jump (or saccade) covers 7-9 letter spaces
• during a saccade visual input is reduced
• readers skip short words and words that are
  highly predictable (these words are identified in
  the parafoveal region)
• readers regress (look back)
• readers often undershoot on return sweeps (going
  from the end of a line to the next line)
• the perceptual span is asymmetrical to the right
  (to the left for languages like Hebrew)

10
SOME FACTS ABOUT READING

• people do not read a text smoothly, but fixate
  a particular word (200 - 300 msec) and jump to
  the next
• a jump (or saccade) covers 7-9 letter spaces
• during a saccade visual input is reduced
• readers skip short words and words that are
  highly predictable (these words are identified in
  the parafoveal region)
• readers regress (look back)
• readers often undershoot on return sweeps (going
  from the end of a line to the next line)
• the perceptual span is asymmetrical to the right
  (to the left for languages like Hebrew)

11
SOME FACTS ABOUT READING

• people do not read a text smoothly, but fixate
  a particular word (200 - 300 msec) and jump to
  the next
• a jump (or saccade) covers 7-9 letter spaces
• during a saccade visual input is reduced
• readers skip short words and words that are
  highly predictable (these words are identified in
  the parafoveal region)
• readers regress (look back)
• readers often undershoot on return sweeps (going
  from the end of a line to the next line)
• the perceptual span is asymmetrical to the right
  (to the left for languages like Hebrew)

12
SOME FACTS ABOUT READING

• people do not read a text smoothly, but fixate
  a particular word (200 - 300 msec) and jump to
  the next
• a jump (or saccade) covers 7-9 letter spaces
• during a saccade visual input is reduced
• readers skip short words and words that are
  highly predictable (these words are identified in
  the parafoveal region)
• readers regress (look back)
• readers often undershoot on return sweeps (going
  from the end of a line to the next line)
• the perceptual span is asymmetrical to the right
  (to the left for languages like Hebrew)

13
SOME FACTS ABOUT READING

• people do not read a text smoothly, but fixate
  a particular word (200 - 300 msec) and jump to
  the next
• a jump (or saccade) covers 7-9 letter spaces
• during a saccade visual input is reduced
• readers skip short words and words that are
  highly predictable (these words are identified in
  the parafoveal region)
• readers regress (look back)
• readers often undershoot on return sweeps (going
  from the end of a line to the next line)
• the perceptual span is asymmetrical to the right
  (to the left for languages like Hebrew)

14
SOME FACTS ABOUT READING

• people do not read a text smoothly, but fixate
  a particular word (200 - 300 msec) and jump to
  the next
• a jump (or saccade) covers 7-9 letter spaces
• during a saccade visual input is reduced
• readers skip short words and words that are
  highly predictable (these words are identified in
  the parafoveal region)
• readers regress (look back)
• readers often undershoot on return sweeps (going
  from the end of a line to the next line)
• the perceptual span is asymmetrical to the right
  (to the left for languages like Hebrew)

15
SOME FACTS ABOUT READING

• people do not read a text smoothly, but fixate
  a particular word (200 - 300 msec) and jump to
  the next
• a jump (or saccade) covers 7-9 letter spaces
• during a saccade visual input is reduced
• readers skip short words and words that are
  highly predictable (these words are identified in
  the parafoveal region)
• readers regress (look back)
• readers often undershoot on return sweeps (going
  from the end of a line to the next line)
• the perceptual span is asymmetrical to the right
  (to the left for languages like Hebrew)

16
SOME FACTS ABOUT READING

• people do not read a text smoothly, but fixate
  a particular word (200 - 300 msec) and jump to
  the next
• a jump (or saccade) covers 7-9 letter spaces
• during a saccade visual input is reduced
• readers skip short words and words that are
  highly predictable (these words are identified in
  the parafoveal region)
• readers regress (look back)
• readers often undershoot on return sweeps (going
  from the end of a line to the next line)
• the perceptual span is asymmetrical to the right
  (to the left for languages like Hebrew)

17
A TYPICAL READING EXPERIMENT

• Garden-path sentence
• Since Jay always jogs a mile seems like a short
  distance to him.
• Control sentence
• Since Jay always jogs a mile this seems like a
  short distance to him.

18
READING PATTERN (Garden-path sentence)
Since
Jay
always
jogs
a
mile
seems
like
a
short
distance
to
him.
19
READING PATTERN (Garden-path sentence)

• Since Jay always jogs a mile
• like a short distance to him.

seems
fixation after progressive saccade (first-pass)
fixation after regressive saccade
fixation after progressive saccade (second-pass)
20
READING PATTERN

• If readers experience some sort of trouble they
  may fixate the difficult region longer and the
  may even regress to earlier parts of the
  sentence/text.

21
HOW DO WE INTERPRET THE READING PATTERNS?
22
DIFFERENT MEASURES

• First fixation duration duration of first
  fixation in a region
• First-pass duration time spent in a region
  before moving on or looking back
• Regression path duration time from first
  entering a region until moving the eyes beyond
  that region, includes regression time
• Second-pass duration duration of re-fixations
• Total duration the sum of all fixations in a
  region
• Probability of a regression the percentage of
  regressive eye-movements out of a region

23
DIFFERENT MEASURES

• First fixation duration duration of first
  fixation in a region
• First-pass duration time spent in a region
  before moving on or looking back
• Regression path duration time from first
  entering a region until moving the eyes beyond
  that region, includes regression time
• Second-pass duration duration of re-fixations
• Total duration the sum of all fixations in a
  region
• Probability of a regression the percentage of
  regressive eye-movements out of a region

24
DIFFERENT MEASURES

• First fixation duration duration of first
  fixation in a region
• First-pass duration time spent in a region
  before moving on or looking back
• Regression path duration time from first
  entering a region until moving the eyes beyond
  that region, includes regression time
• Second-pass duration duration of re-fixations
• Total duration the sum of all fixations in a
  region
• Probability of a regression the percentage of
  regressive eye-movements out of a region

25
DIFFERENT MEASURES

• First fixation duration duration of first
  fixation in a region
• First-pass duration time spent in a region
  before moving on or looking back
• Regression path duration time from first
  entering a region until moving the eyes beyond
  that region, includes regression time
• Second-pass duration duration of re-fixations
• Total duration the sum of all fixations in a
  region
• Probability of a regression the percentage of
  regressive eye-movements out of a region

26
DIFFERENT MEASURES

• First fixation duration duration of first
  fixation in a region
• First-pass duration time spent in a region
  before moving on or looking back
• Regression path duration time from first
  entering a region until moving the eyes beyond
  that region, includes regression time
• Second-pass duration duration of re-fixations
• Total duration the sum of all fixations in a
  region
• Probability of a regression the percentage of
  regressive eye-movements out of a region

27
DIFFERENT MEASURES

• First fixation duration duration of first
  fixation in a region
• First-pass duration time spent in a region
  before moving on or looking back
• Regression path duration time from first
  entering a region until moving the eyes beyond
  that region, includes regression time
• Second-pass duration duration of re-fixations
• Total duration the sum of all fixations in a
  region
• Probability of a regression the percentage of
  regressive eye-movements out of a region

28
DIFFERENT MEASURES

• First fixation duration duration of first
  fixation in a region
• First-pass duration time spent in a region
  before moving on or looking back
• Regression path duration time from first
  entering a region until moving the eyes beyond
  that region, includes regression time
• Second-pass duration duration of re-fixations
• Total duration the sum of all fixations in a
  region
• Probability of a regression the percentage of
  regressive eye-movements out of a region

29
EXPLANATION OF DIFFERENT MEASURES
Bart annoyed Homer because
1
2
3
4
5
7
6
Reading Times for word 3 (Homer) First fixation
duration 3 First-pass duration 3
4 Regression Path duration 3 4
5 Second-pass duration 6 Total duration 3 4
6
30
DIFFERENT FIRST-PASS MEASURES
FIRST FIXATION DURATION
FIRST-PASS DURATION
REGRESSION PATH DURATION
TIME
31
HOW DO WE INTERPRET THE READING TIMES?
32
THE LINKING PROBLEM
eye
mind
33
EYE-MIND ASSUMPTION(JUST CARPENTER, 1980)

• Readers retain fixation on a word until
  processing is completed
• This includes processes like word recognition,
  syntactic parsing, semantic integration,
  referential integration

34
AN IDEAL WORLD(VAN BERKUM, 2004)
Snowwhite
kissed
a
dwarf
W
P
S
R
W
P
S
R
W
P
S
R
W
P
S
R
TIME
W word recognition P parsing S semantic
integration R referential integration
35
THE REAL WORLDIS A REAL MESS(VAN BERKUM, 2004)
Snowwhite
kissed
a
dwarf
W
P
S
R
W
P
S
R
W
P
S
R
W
P
S
R
TIME
W word recognition P parsing S semantic
integration R referential integration
36
THE REAL WORLDIS A REAL MESS

• In the real world the processing of word X
  continues while fixating word X 1 (and possibly
  while fixating word X 2 etc.)
• Thus, the effects of a manipulation are often
  visible more downstream (i.e. after the critical
  word or region). This is called spill-over.

37
THE REAL WORLDIS A REAL MESS

• In the real world the processing of word X
  continues while fixating word X 1 (and possibly
  while fixating word X 2 etc.)
• Thus, the effects of a manipulation are often
  visible more downstream (i.e. after the critical
  word or region). This is called spill-over.

38
LINKING ASSUMPTION(BOLAND, 2004)

• The eyes do not leave a word until it has been
  structurally integrated (tree building).
  Therefore, constraints that control
  structure-building influence first-pass reading
  time.
• other measures (e.g., regression path duration)
  are sensitive for higher level processes
  (semantic integration, discourse processes)

39
SOLUTION LINKING PROBLEM?

• Perhaps the different measures can provide
  information about what is happening? (this is an
  empirical question)

40
A DISADVANTAGE OF THE READING PARADIGM

• You can only use skilled readers (no children or
  language-disordered populations).
• This is possible in spoken language paradigms.

41
CAVEAT

• the eyes tell us that something is happening at a
  specific point in time, but not what that
  something is!

42
TO CONCLUDE

• Eye-tracking (while reading and listening) excels
  in the when question.
• Not really suited for what question (use
  EEG/MEG instead).

43
QUESTIONS?
44
PART 2 THEORY
45
CONSTRUCTING ANAPHORIC DEPENDENCIES

• VARIABLE BINDING vs. CO-REFERENCE

46
BASIC ARCHITECTURE PROCESSING SYSTEM

• distinct modules
• ? syntax, semantics, discourse
• economy principle cross-modular operations carry
  a cost (Reuland 2001)
• syntax first in time-course-model of processing
  phases (Friederici 1995-)
• ? syntax is cheaper than semantics
• ? semantics is cheaper than discourse

47
CROSS-MODULAR OPERATIONS

• Discourse storage (values) a ? a
• C-I objects (variables) x1 x2
• Syntactic objects (chains) C1 C2
• Basic expressions a ß
• Discourse storage (values) a
• C-I objects (variables) x1 ? x1
• Syntactic objects (chains) C1 C2
• Basic expressions a ß
• Discourse storage (values) a
• C-I objects (variables) x1
• Syntactic objects (chains) C1 ? C1
• Basic expressions a ß

48
TIME-COURSE SIMPLIFIED
SYNTAX
SEMANTICS
DISCOURSE
49
SEMANTICS VS. DISCOURSE

• variable binding (semantic dependency)
• ? Every clown thinks that he is not funny.
• co-reference (discourse dependency)
• ? The clown has a big problem.
• He is not funny.

50
PREDICTIONS OF THEORY

• variable binding is cheaper/faster than
  co-reference
• in an ambiguous situation variable binding has
  precedence over co-reference

51
END OF INTRO