Title: Anaphoric dependencies: A window into the architecture of the language system Eye tracking experimen
1Anaphoric dependencies A window into the
architecture of the language system
Eye tracking experiments
Eric Reuland Frank Wijnen Arnout Koornneef
2EYE-TRACKING
BY ARNOUT KOORNNEEF
3OVERVIEW OF LECTURE
- Part 1 discussion of method
- - eye-tracking while reading
- Part 2 discussion of current research
- - eye-tracking experiments
4THE EYE-TRACKER
5THE EYE-TRACKER
6THE EYE-TRACKER
- monitors eye-movements from millisecond to
millisecond - provides information about where people look and
for how long
7TWO MAIN EYE-TRACKING PARADIGMS
- eye-tracking while reading
- eye-tracking while listening
- (not discussed)
8EYE-TRACKING WHILE READING
9SOME FACTS ABOUT READING
- people do not read a text smoothly, but fixate
a particular word (200 - 300 msec) and jump to
the next - a jump (or saccade) covers 7-9 letter spaces
- during a saccade visual input is reduced
- readers skip short words and words that are
highly predictable (these words are identified in
the parafoveal region) - readers regress (look back)
- readers often undershoot on return sweeps (going
from the end of a line to the next line) - the perceptual span is asymmetrical to the right
(to the left for languages like Hebrew)
10SOME FACTS ABOUT READING
- people do not read a text smoothly, but fixate
a particular word (200 - 300 msec) and jump to
the next - a jump (or saccade) covers 7-9 letter spaces
- during a saccade visual input is reduced
- readers skip short words and words that are
highly predictable (these words are identified in
the parafoveal region) - readers regress (look back)
- readers often undershoot on return sweeps (going
from the end of a line to the next line) - the perceptual span is asymmetrical to the right
(to the left for languages like Hebrew)
11SOME FACTS ABOUT READING
- people do not read a text smoothly, but fixate
a particular word (200 - 300 msec) and jump to
the next - a jump (or saccade) covers 7-9 letter spaces
- during a saccade visual input is reduced
- readers skip short words and words that are
highly predictable (these words are identified in
the parafoveal region) - readers regress (look back)
- readers often undershoot on return sweeps (going
from the end of a line to the next line) - the perceptual span is asymmetrical to the right
(to the left for languages like Hebrew)
12SOME FACTS ABOUT READING
- people do not read a text smoothly, but fixate
a particular word (200 - 300 msec) and jump to
the next - a jump (or saccade) covers 7-9 letter spaces
- during a saccade visual input is reduced
- readers skip short words and words that are
highly predictable (these words are identified in
the parafoveal region) - readers regress (look back)
- readers often undershoot on return sweeps (going
from the end of a line to the next line) - the perceptual span is asymmetrical to the right
(to the left for languages like Hebrew)
13SOME FACTS ABOUT READING
- people do not read a text smoothly, but fixate
a particular word (200 - 300 msec) and jump to
the next - a jump (or saccade) covers 7-9 letter spaces
- during a saccade visual input is reduced
- readers skip short words and words that are
highly predictable (these words are identified in
the parafoveal region) - readers regress (look back)
- readers often undershoot on return sweeps (going
from the end of a line to the next line) - the perceptual span is asymmetrical to the right
(to the left for languages like Hebrew)
14SOME FACTS ABOUT READING
- people do not read a text smoothly, but fixate
a particular word (200 - 300 msec) and jump to
the next - a jump (or saccade) covers 7-9 letter spaces
- during a saccade visual input is reduced
- readers skip short words and words that are
highly predictable (these words are identified in
the parafoveal region) - readers regress (look back)
- readers often undershoot on return sweeps (going
from the end of a line to the next line) - the perceptual span is asymmetrical to the right
(to the left for languages like Hebrew)
15SOME FACTS ABOUT READING
- people do not read a text smoothly, but fixate
a particular word (200 - 300 msec) and jump to
the next - a jump (or saccade) covers 7-9 letter spaces
- during a saccade visual input is reduced
- readers skip short words and words that are
highly predictable (these words are identified in
the parafoveal region) - readers regress (look back)
- readers often undershoot on return sweeps (going
from the end of a line to the next line) - the perceptual span is asymmetrical to the right
(to the left for languages like Hebrew)
16SOME FACTS ABOUT READING
- people do not read a text smoothly, but fixate
a particular word (200 - 300 msec) and jump to
the next - a jump (or saccade) covers 7-9 letter spaces
- during a saccade visual input is reduced
- readers skip short words and words that are
highly predictable (these words are identified in
the parafoveal region) - readers regress (look back)
- readers often undershoot on return sweeps (going
from the end of a line to the next line) - the perceptual span is asymmetrical to the right
(to the left for languages like Hebrew)
17A TYPICAL READING EXPERIMENT
-
- Garden-path sentence
- Since Jay always jogs a mile seems like a short
distance to him. -
- Control sentence
- Since Jay always jogs a mile this seems like a
short distance to him.
18READING PATTERN (Garden-path sentence)
Since
Jay
always
jogs
a
mile
seems
like
a
short
distance
to
him.
19READING PATTERN (Garden-path sentence)
-
- Since Jay always jogs a mile
- like a short distance to him.
-
seems
fixation after progressive saccade (first-pass)
fixation after regressive saccade
fixation after progressive saccade (second-pass)
20READING PATTERN
- If readers experience some sort of trouble they
may fixate the difficult region longer and the
may even regress to earlier parts of the
sentence/text.
21HOW DO WE INTERPRET THE READING PATTERNS?
22DIFFERENT MEASURES
- First fixation duration duration of first
fixation in a region - First-pass duration time spent in a region
before moving on or looking back - Regression path duration time from first
entering a region until moving the eyes beyond
that region, includes regression time - Second-pass duration duration of re-fixations
- Total duration the sum of all fixations in a
region - Probability of a regression the percentage of
regressive eye-movements out of a region
23DIFFERENT MEASURES
- First fixation duration duration of first
fixation in a region - First-pass duration time spent in a region
before moving on or looking back - Regression path duration time from first
entering a region until moving the eyes beyond
that region, includes regression time - Second-pass duration duration of re-fixations
- Total duration the sum of all fixations in a
region - Probability of a regression the percentage of
regressive eye-movements out of a region
24DIFFERENT MEASURES
- First fixation duration duration of first
fixation in a region - First-pass duration time spent in a region
before moving on or looking back - Regression path duration time from first
entering a region until moving the eyes beyond
that region, includes regression time - Second-pass duration duration of re-fixations
- Total duration the sum of all fixations in a
region - Probability of a regression the percentage of
regressive eye-movements out of a region
25DIFFERENT MEASURES
- First fixation duration duration of first
fixation in a region - First-pass duration time spent in a region
before moving on or looking back - Regression path duration time from first
entering a region until moving the eyes beyond
that region, includes regression time - Second-pass duration duration of re-fixations
- Total duration the sum of all fixations in a
region - Probability of a regression the percentage of
regressive eye-movements out of a region
26DIFFERENT MEASURES
- First fixation duration duration of first
fixation in a region - First-pass duration time spent in a region
before moving on or looking back - Regression path duration time from first
entering a region until moving the eyes beyond
that region, includes regression time - Second-pass duration duration of re-fixations
- Total duration the sum of all fixations in a
region - Probability of a regression the percentage of
regressive eye-movements out of a region
27DIFFERENT MEASURES
- First fixation duration duration of first
fixation in a region - First-pass duration time spent in a region
before moving on or looking back - Regression path duration time from first
entering a region until moving the eyes beyond
that region, includes regression time - Second-pass duration duration of re-fixations
- Total duration the sum of all fixations in a
region - Probability of a regression the percentage of
regressive eye-movements out of a region
28DIFFERENT MEASURES
- First fixation duration duration of first
fixation in a region - First-pass duration time spent in a region
before moving on or looking back - Regression path duration time from first
entering a region until moving the eyes beyond
that region, includes regression time - Second-pass duration duration of re-fixations
- Total duration the sum of all fixations in a
region - Probability of a regression the percentage of
regressive eye-movements out of a region
29EXPLANATION OF DIFFERENT MEASURES
Bart annoyed Homer because
1
2
3
4
5
7
6
Reading Times for word 3 (Homer) First fixation
duration 3 First-pass duration 3
4 Regression Path duration 3 4
5 Second-pass duration 6 Total duration 3 4
6
30DIFFERENT FIRST-PASS MEASURES
FIRST FIXATION DURATION
FIRST-PASS DURATION
REGRESSION PATH DURATION
TIME
31HOW DO WE INTERPRET THE READING TIMES?
32THE LINKING PROBLEM
eye
mind
33EYE-MIND ASSUMPTION(JUST CARPENTER, 1980)
- Readers retain fixation on a word until
processing is completed - This includes processes like word recognition,
syntactic parsing, semantic integration,
referential integration
34AN IDEAL WORLD(VAN BERKUM, 2004)
Snowwhite
kissed
a
dwarf
W
P
S
R
W
P
S
R
W
P
S
R
W
P
S
R
TIME
W word recognition P parsing S semantic
integration R referential integration
35THE REAL WORLDIS A REAL MESS(VAN BERKUM, 2004)
Snowwhite
kissed
a
dwarf
W
P
S
R
W
P
S
R
W
P
S
R
W
P
S
R
TIME
W word recognition P parsing S semantic
integration R referential integration
36THE REAL WORLDIS A REAL MESS
- In the real world the processing of word X
continues while fixating word X 1 (and possibly
while fixating word X 2 etc.) - Thus, the effects of a manipulation are often
visible more downstream (i.e. after the critical
word or region). This is called spill-over.
37THE REAL WORLDIS A REAL MESS
- In the real world the processing of word X
continues while fixating word X 1 (and possibly
while fixating word X 2 etc.) - Thus, the effects of a manipulation are often
visible more downstream (i.e. after the critical
word or region). This is called spill-over.
38LINKING ASSUMPTION(BOLAND, 2004)
- The eyes do not leave a word until it has been
structurally integrated (tree building).
Therefore, constraints that control
structure-building influence first-pass reading
time. -
- other measures (e.g., regression path duration)
are sensitive for higher level processes
(semantic integration, discourse processes) -
39SOLUTION LINKING PROBLEM?
- Perhaps the different measures can provide
information about what is happening? (this is an
empirical question)
40A DISADVANTAGE OF THE READING PARADIGM
- You can only use skilled readers (no children or
language-disordered populations). - This is possible in spoken language paradigms.
41CAVEAT
- the eyes tell us that something is happening at a
specific point in time, but not what that
something is!
42TO CONCLUDE
- Eye-tracking (while reading and listening) excels
in the when question. - Not really suited for what question (use
EEG/MEG instead).
43QUESTIONS?
44PART 2 THEORY
45CONSTRUCTING ANAPHORIC DEPENDENCIES
- VARIABLE BINDING vs. CO-REFERENCE
46BASIC ARCHITECTURE PROCESSING SYSTEM
- distinct modules
- ? syntax, semantics, discourse
- economy principle cross-modular operations carry
a cost (Reuland 2001) -
- syntax first in time-course-model of processing
phases (Friederici 1995-) -
- ? syntax is cheaper than semantics
- ? semantics is cheaper than discourse
47CROSS-MODULAR OPERATIONS
- Discourse storage (values) a ? a
- C-I objects (variables) x1 x2
- Syntactic objects (chains) C1 C2
- Basic expressions a ß
- Discourse storage (values) a
- C-I objects (variables) x1 ? x1
- Syntactic objects (chains) C1 C2
- Basic expressions a ß
- Discourse storage (values) a
- C-I objects (variables) x1
- Syntactic objects (chains) C1 ? C1
- Basic expressions a ß
48TIME-COURSE SIMPLIFIED
SYNTAX
SEMANTICS
DISCOURSE
49SEMANTICS VS. DISCOURSE
- variable binding (semantic dependency)
- ? Every clown thinks that he is not funny.
- co-reference (discourse dependency)
- ? The clown has a big problem.
- He is not funny.
50PREDICTIONS OF THEORY
- variable binding is cheaper/faster than
co-reference - in an ambiguous situation variable binding has
precedence over co-reference
51END OF INTRO