Experimental Techniques

1
Experimental Techniques Lexical Semantics

• Christer Johansson
• Computational Linguistics
• University of Bergen

2
Acknowledgments

• Laurie Ann Stowe
• Marco Haverkort
• Berry Wijers
• Rienk Withaar
• Frans Zwarts

• PET Center, AZG
• BCN Neuroimaging Center
• NWO

3
Measuring techniques

• Reaction Time Experience
• Event Related Potentials
• (electrical) neuronal activity
• Drawback it is hard to localize the source of
  the signal (as it is distorted by the scull).
• Relatively cheap.
• MEG
• (magnetic) neuronal activity.
• Little distortion
• Very expensive

4
Reaction Time Studies

• The case of the missing adjectives semantics and
  phonotactics?

5
Paradigms

• Interference
• Picture - Word interference
• Priming
• Something related to the target is shown
• Classification
• Difference between pre-classified classes.

6
Classic Stroop

• Which colors are the words written in?
• RED BLUE YELLOW GREEN
• BLUE RED GREEN YELLOW

7
Effects

• Interaction
• Draws simultaneously on the same resources.
• Interferes with more automatic processing
• Facilitation
• Preparation and pre-activation
• Spreading activation model Lowering Thresholds

8
Effects

• Priming
• Priming of many alternatives
• Selection
• Reactivation at relevant positions
• Repetition effects

9
Effects

• Differences in processing profile shows up as
  differences in reaction times.
• All else equal.

10
Semantics of missing adjectives
11
Problematic adjectives

• ... denote properties that are
• Hard to visualize (draw a picture of afraid)
• Hard to verify (How do we know that someone is
  afraid?)
• Internal states not directly observable

12
However

• common neuter
• en trött man, ett trött barn
• A tired man a tired child
• en slug man ett slugt barn
• A sly man a sly child

13
Not only semantics

• en trött man, ett trött barn
• Same form for both genders
• en slug man, ett slugt barn.
• -t easily detectable morpheme boundary.
• (also vowel quality intact).

14
Quick corpus study

• Possible neuters
• Most sött, rött, blått
• cVtobservable neuter, short vowel
• Some trött, mätt
• cVt internal state BUT same form for both
  genders.
• Almost none latt, rätt
• cVt internal state AND different gender forms.

15
Gender semantics nouns

• Tendencies (neuters mostly non-living)
• Neuter mass / one of many, low identifiability
• children, mosquitoes etc.
• Common high individuality.
• a specific nail en spik, any one nail ett spik
• the mosquito that bites you en mygga.
• any one mosquito ett mygg
• a chair seats one individual en stol
• a table is a shared space ett bord.

16
Design

• Subjects 25 students and staff Lund and Oslo.
• Swedish 19 subjects (6 too slow responses)
• Norwegian 21 subjects (4 too slow).
• 240 NPs determiner adjective head noun
• Presented in 4 blocks without repetition.
• Pause between blocks
• Forced choice YES NO
• Decision and reaction time were recorded.

17
10 main classes

• NPROB problematic form NEUTER context
• (et latt barn / a lazy child)
• CPROB problematic form COMMON context
• (en latt unge / a lazy kid)
• CYES concord common gender (en fin unge)
• NYES concord neuter gender (et fint barn)
  
• CNO incongruent common context
  (en
  fint unge)
• NNO incongruent neuter context (et fin barn)
  
• NONS 4 categorier gender x congruent
  (e.g., enet skland sklant ungebarn

18
Exposures in each condition

• NPROB 16 (4 adjective 4 head nouns)
• CPROB 16 (15 for Swedish glad/glatt homonymi).
• CYES 40 (10 4)
• NYES 20 (5 4)
• CNO 20 (5 4)
• NNO 40 (10 4)
• NONS 4 x 4 x 4 64
• Fillers 24 (Training).
• 19 Swedish 21 Norwegian. 4500 to 5000 exposures

19
Balanced Factors

• Phonological and Semantical Complexity
• Head words common neuter
• Kvinna fruntimmer, unge barn, katt lejon,
  varelse djur
• Woman wench kid child cat
  lion, being - animal
• Adjectives
• Lat sömnig, rädd skrämd, rigid styv, gravid
  lycklig
• Lazy sleepy, afraid scared, rigid stiff,
  pregnant happy
• Semantic as well as phonological complexity
  matched.
• Training
• Nouns flicka äpple girl apple
• Adjectives liten söt small sweet

20
Expectation

• Principle Conflicting cues takes time to resolve
• NPROB correct form does the word exist?
  3.
• CPROB wrong form does the word exist? 2.
• CNO/NNO clear incongruency
  1.
• CYES/NYES no conflict
  0.

21
Reaction times (Swedish)

• Normalized (1.3 30 slower than individual
  median).
• Highly significant main differences. NPROB very
  slow.

NPROB
N/CNO
NONS
CPROB
N/CYES
22
Reaction times (Swedish)

• NPROB most problematic. Slowest. Very different.
• Neuter slower than common gender. (Frequency
  effect)
• Novel words are slower for congruency, but faster
  for incongruency.

23
Conclusion (Swedish)

• Reaction times improved by clear decidability and
  high (class) frequency.
• We have detected the problematic forms by means
  of reaction times for lexical decision.

24
Norwegian

• Very similar, but different words
• et krytt/latt/slutt/vantrott/statt
  (1)
• et sultent/snilt/slitent/forkomment/fint(2)
• barn/dyr/esel/kvinnfolk
• en kry/lat/slu/vantro/sta
• unge/person/katt/kvinne

25
Reaction times (Norwegian)
nno
nprob
nyes
cno
cprob
cyes
26
Reaction times Expected answers only
nno kry (yes)
The problematic adjectives were accepted in a
common gender form in a neuter context, with
a lot of hesitation 55 slower, and 3/4
affirmative responses.
nprob (no)
nno (no)
cno (no)
nyes (yes)
cprob (no)
cyes (yes)
27
Reaction time (norwegian)

• NPROB similar to normal incongruency
• Problematic latt, rätt, etc treated as common
  gender???
• Neuter slower than common gender. (Frequency
  effect)
• Just like in swedish!!
• Incongruent slower than congruent.
• CPROB CNO. Problematic forms treated as
  incongruent with common gender!
• Recall Both cases have the same forms
  latt, krytt, slutt, vantrott, statt
• et _ barn (nprob) en _ unge
  (cprob)!!

28
Conclusion 1 (Norwegian)

• Different from Swedish.
• The problematic forms are reacted to as if they
  are incongruent with both genders.
• Non-words?

29
Norwegian novel words
cno
nno
nyes
cyes
30
Conclusion

• Problematic cases
• SWEDISH NORWEGIAN
• N C N
  C
• N 33 23 N 29 11
• C 13 -4 C 49 -1

31
Conclusion

• In swedish, the problematic forms cause
  significantly higher response times, and their
  common gender form is not different from other
  common gender forms.
• In norwegian, the common gender is the preferred
  default (according to dictionaries), but
  accepting the common gender form for the
  problematic forms create significant hesitation.

32
Conclusion

• The observed hesitation to the problematic forms,
  and for Norwegian also for the prescribed default
  forms, indicate that there is a factor involved
  that is not observed directly in the word forms.
• The results suggest that the ability to have
  internal states, specifically to be animate
  and/or individualized, is crucial for the
  formation of Scandinavian gender classes, as
  noted in the semantics of the missing adjectives.

33
Electro-Encephalo-Gram (EEG)
34
EventRelatedPotentials(ERP)
high and low points extent of variability
Averaged responses cancels noise, shows trend
35
Neville et al, 1991

• Semantic contrast
• The scientist criticized Maxs argument
• The scientist criticized Maxs headache

36
(No Transcript)
37
(No Transcript)
38
Semantic Violations

• Negativity, peaking about 400 msec after
  presentation of word
• maximal at parietal electrode sites
• usually larger right than left
• N400

39
Neville et al, 1991

• Syntactic contrast
• The scientist criticized Maxs proof of
• The scientist criticized Maxs of proof

40
(No Transcript)
41
(No Transcript)
42
Phrase Structure Violations

• Negativity
• about 150 msec after presentation of the word
• left frontal electrodes
• Early Left Anterior Negativity ELAN

43
Phrase Structure Violations

• Negativity
• about 400 msec after presentation of the word
• left (anterior) electrodes
• LAN

44
Phrase Structure Violations

• Positivity
• about 600 msec after presentation of the word
• central parietal electrodes
• P600

45
Conclusion from ERP

• semantic and syntactic processing seem to be
  dealt with
• in different stages
• by different brain areas

46
PET ERP

• Measure differences in blood flow to brain areas
  cognitive load?

47
What is measured by PET fMRI?

• Both PET and fMRI measure changes in blood flow.
  
• PET is invasive, i.e., a radioactive substance
  is injected into the blood stream.
• fMRI is non-invasive, but the subject is placed
  in a very strong magnetic field.

48
What does this mean?

• Blocked design.
• Because it takes time for the change to build up.
  We need to jog the brain on similar inputs until
  we have enough change in blood flow.
• Relatively long time between measurements -
  increases risk of movement artifacts.
• Latest developments event related fMRI. J. of
  Neurolinguistics, 2002 15 (Dogil al.)

49
Normalization Talairach deamon

• All pictures need to be aligned -
• Standard set by Talairach Atlas.
• Individual variation
• asymmetry (left/right front/back)
• Effects are manipulated
• using a Gaussian Filter.
• Reason fitting the brain anatomy of a typical
  subject creates uncertainity on the location of
  effects.

50
Blood Flow ChangePET and fMRI

• Measure regional cerebral blood flow
• compare to see where additional processing
  increases blood flow
• Subtract a baseline case that contains all of the
  cognitive operations of the test - except the
  factor we are interested in.

51
Classical Model

• frontal language area Brocas
• posterior language area Wernickes

52
Neuroimaging ConfirmationYES (reading)

• complex sentences vs. letter strings

53
Neuroimaging Confirmation
54
Neuroimaging Confirmation
55
Neuroimaging Confirmation
56
Points

• Confirmation
• of left hemisphere dominance
• involvement of classical language areas
• Modification
• involvement of additional areas

57
Conclusions so far

• Brocas area is involved in the comprehension of
  complex sentences
• not surprising ...

58
Simple Sentences vs. Passive Fixation
59
The role of Brocas area?

• That Brocas area is involved
• does not mean that
• syntactic processing is located in the left
  inferior frontal lobe
• simple sentences do not reliably activate this
  area
• other tasks with similar cognitive components
  also activate this area

60
Wijers et al WM task
61
Wijers et al WM task
62
Wijers et al WM task
63
Function of the frontal lobe in comprehension

• Sentential complexity activates left inferior
  frontal gyrus (Brocas area)
• verbal working memory tasks also activate LIFG

64
Is verbal working memory used to comprehend
complex sentences?
65
Stowe et al (1998)

• Simple sentences
• single clause
• Complex Clauses
• non-final embedded clauses
• Ambiguous Sentences
• resolved to less expected structure
• Word Lists

66
Stowe et al (1998)
One area is activated more for sentences. No
effect of processing load.
W lt A C S Left
Anterior Temporal Lobe
67
Stowe et al (1998)
Processing load

• no
• low
• higher
• highest

• Word Lists
• Simple
• Complex
• Ambiguous

Expectation W lt S lt C lt A
68
Stowe et al (1998)
W lt S lt C lt A Left Posterior
Temporal Gyrus
69
Stowe et al (1998)
Memory Load

• low
• higher
• higher
• Highest

• Simple
• Complex
• Ambiguous
• Word Lists

Expectation S lt C A lt W (by memory
load) Recall W lt S lt C lt A (by processing load)
70
Stowe et al (1998)
S lt C lt A (W doesnt fit!) Almost explained by
memory load.
71
Stowe et al (1998)Combining process and memory
(Process Memory) Load

• low low
• higher higher
• lowest highest
• highest highest

• Simple
• Complex
• Word Lists
• Ambiguous

This predicts S lt C W lt A Is there such an
area?
72
Stowe et al (1998)S lt C W lt A
Left Frontal operculum/insula.
73
Conclusions

• The same area
• left inferior frontal lobe
• is activated by
• syntactic complexity
• verbal working memory load

74
Syntactic Ambiguity

• Syntactically Ambiguous Sentences
• Zij kunnen bakken met zulk deeg
• ...niet verplaatsen.
• They can bake(V)/containers(N) with that-sort-of
  dough
• not move.

75
Syntactically Ambiguous Sentences

• Activate LIFG
• Activate left SFG
• also activated by
• reasoning tasks
• plausibility decision

76
Syntactically Ambiguous Sentences

• right cerebellum and caudate nucleus

77
Syntactically Ambiguous Sentences

• Activate cerebellum
• also activated by verbal working memory tasks
• which are frequently interpreted as covert
  articulation (motor)
• activated in naming tasks where automatic
  response must be inhibited

78
Right Cerebellum Tasks

• Generate verbs
• from visual or auditory word or visual cue
• decreases with practice
• Generate unstudied stem completion
• especially with few good completions
• Generate synonyms
• Generate rhymes
• Generate translations

79
Syntactically Ambiguous Sentences

• Activate caudate nucleus involved in
• sequence learning
• set switching

80
Summary

• Many areas involved
• in language processing.
• Different areas combine
• process and memory load differently.
• The areas seem orchestrated
• the connection in the deep regions of the brain?
• (basal ganglia, caudate nucleus, hippocampus?,
  thalamus??).

81
Summary

• We have also seen that resources are allocated
  dynamically between areas.
• visual areas get to be involved in complex
  language tasks.
• motor areas (cerebellum) get involved in
  complex language tasks.

82
Differences ERP PET

• Economy -
• ERP is cheap PET is expensive.
• Staff intensive.
• Timing ERP has a better temporal resolotion but
  bad spatial resolution (due to reflection of the
  signal).
• ERP might provide us with a temporal signature of
  processing.

83
Thanks

• Laurie Stowe for providing a lot of the material
  for this presentation.
• http//odur.let.rug.nl/stowe/

84
References Dogil. G., Ackermann, H., Grodd, W.,
Haider. H., Kamp H., Mayer J., Riecker A.,
Wildgruber D. 2002. The speaking brain a
tutorial introduction to fMRI experiments in the
production of speech, prosody and syntax. J. of
Neurolinguistics, 15. pp.59-90. Stowe, L.A.,
Withaar, R.G., Wijers, A.A., Broere, C.A.J.,
Paans, A.M.J. (Submitted). Encoding and storage
in working memory during sentence comprehension.
To appear in Merlo, P., Stevenson, S. (1999).
Proceedings of the Eleventh CUNY Conference on
Sentence Processing. Stowe, L.A., Broere,
C.A.J., Paans, A.M.J., Wijers, A.A., Mulder, G.,
Vaalburg,W., Zwarts, F. G., Vaalburg, W.
(1998). Localizing Components of a Complex Task
Sentence Processing and Working Memory.
NeuroReport 92995-2999.. Stowe, L.A., Wijers,
A.A., Paans, A.M.J., Haverkort, M.,Broere,
C.A.J., Mulder, G., Zwarts, F., Vaalburg, W.
(1998). The Role of the Right Hemisphere in
Resolving Lexical Semantic Ambiguity. Submitted
to Brain..
85
ERP - ref. cont. Neville, H., Nicol, J. L.,
Barss, A., Forster, K. I., Garrett, M. F.
(1991). Syntactically based sentence processing
classes Evidence from event-related brain
potentials. Journal of Cognitive Neuroscience,
3(2), 151-165. .. http//www.ling.udel.edu/coli
n/courses/ling890/class12.pdf. Kutas, M.
Kluender, R. What is who violating? A
reconsideration of Linguistic Violations in Light
of Event-Related Brain Potentials. CRL Newsletter
6-1., http//crl.ucsd.edu/newsletter/6-1/Article1.
html Osterhaut, L. and Mobley, L. A. 1995.
Event-Related Potentials Elicited by Failure to
Agree, J.of Memory and Language, 34, 739-773.
86
Object vs. Subject Relatives

• Anna misleidde de spelers
• Anna misled the players
• die de psycholoog ophemelde
• who the psychologist praised-sg (players) ? obj
• die de psycholoog ophemelden
• who (praised) the psychologist praised-pl
  ? subj

87
Object vs. Subject Relatives
88
Left Inferior Frontal Activation for Sentential
Complexity

• Stromswold et al (1996) Just et al (1996)
• Stowe et al (1998) Caplan et al (1998)
• Object vs. Subject Relatives
• Anna misleidde de spelers
• die de psycholoog ophemelde ---gt obj
• die de psycholoog ophemelden ---gt subj

89
Stowe et al (1999)

• Does memory load interact with sentential
  complexity?
• (manipulation of memory load)

90
Stowe et al (1999)

• Memory Load
• 1 tekening

91
Stowe et al (1999)

• Memory Load
• 5 stoep tekening gasten
• herkende schaatste

92
Stowe et al (1999)

• Processing Load
• Simple
• De brand is door de kaars op de kast ontstaan.
• Haar verhaal is belachelijk.
• Complex
• Dat de brand is door de kaars op de kast onstaan
  is
• belachelijk.

93
Stowe et al (1999)
94
Stowe et al (1999)
95
Stowe et al (1999)

• Why do the one word memory load easy sentences
  have higher activation than the five word memory
  load easy sentences?

96
Stowe et al (1999)

• Main effect of Memory load
• in the occipital lobe
• primarily associated with visual processing
• and visual memory processes

97
Stowe et al (1999)

• Memory task may be carried out as a verbal or as
  a visual task
• it is apparently more likely to be carried out as
  a verbal task if there are sufficient resources
  left over from the sentence processing task
• which argues that both make use of the same
  resources

98
Conclusions

• The left inferior frontal gyrus supports a
  verbal memory function
• used in sentence processing as well as in other
  cognitive processes

99
Stowe et al (in progress)

• Lexical Processing Load
• high vs. low frequency words
• Sentence Processing Load
• word lists
• simple sentences
• complex sentences

100
Low vs. High Frequency Words
101
Low vs. High Frequency Words
102
Conclusions

• high frequency words in sentences cause less
  activation than in word lists
• due to predictability?
• Low frequency words in sentences cause more
  activation than in word lists
• due to need to identify words?

103
The Role of the Anterior Temporal Lobe

• Encoding lexical information
• to keep it activated for comprehension
• if so, more work will be necessary for low
  frequency words

104
Anterior Temporal Lobe Interaction Sentence by WF
105
Anterior Temporal Lobe Interaction Sentence by WF
106
Conclusion

• The difference between word list and sentence
  condition is much bigger for the conditions
  containing low frequency words