Task and context effects in bilingual processing

1
Task and context effects in bilingual processing

• Ton Dijkstra
• Bolzano/Bozen, 19-05-2005
• Mitteleuropa Stiftung

2
Overview of lecture

• Part I Context effects on processing
• Language User Framework
• BIA model --gt demo
• Empirical studies (RT)
• Part II Context and brain
• BIA model
• Empirical studies (RT, ERP, and fMRI)

3
Part I Processing and context

• Goals
• Describe a theoretical framework for bilingual
  research
• Examine how different types of context affect
  bilingual word recognition performance

4
Language User Framework

• Psycholinguists consider language processing
  during understanding and production as
  computations that change representations
• A number of components are needed
• Representations and rules (LTM)
• Processing components and processes
• Working memory
• Cognitive control, attention, monitoring

5
Language User Framework

• Globally
• modular and
• locally
• interactive
• components

6
(No Transcript)
7
BIA model

• Computational model
• Localist connectionist
• Representations and associated processing

8
(No Transcript)
9
(No Transcript)
10
BIA model

• Language nonselective lexical access
• Integrated lexicon
• Simulations using as items
• Interlingual homographs
• Neighbors
• Cognates? Not possible (only O)
• Morphological family size? Not possible

11
Computational models

• Advantages of implemented models
• Strong theoretical framework
• Inherent test of consistency and coherence
• Quantitative predictions, even for interactions
• Potential problem Models can be relatively
  inflexible or over-simplified context is often
  underspecified
• There is a tendency to talk about general models,
  e.g., model for word recognition

12
Seidenberg McClelland (1989)
Context
Meaning
Phonology
Orthography
MAKE
/mAK/
13
Types of context

• Stimulus characteristics of word(s) in other
  language -gt interlingual homographs
• Stimulus list composition
• Task demands
• Semantic priming
• Sentence context
• Instruction
• Daily life and immersion context

14
Complexity of context effects in comprehension

• Example target word in sentence / list
• Lexical characteristics of target
• Semantics of preceding words
• Semantics and pragmatics of preceding sentence /
  list
• Language membership of target
• Syntax of preceding words in sentence
• Many others task, instruction, strategies,
  attention, motivation

15
Interlingual homographs (IHs)

• An interlingual homograph is orthographically
  identical but semantically different across
  languages
• The word BRAND means fire in Dutch

16
Examples of Dutch-English interlingual homographs

• ROOM (Dutch cream)
• In a white room, single by Cream
• ANGEL (Dutch sting)
• Angel eyes, single by Sting
• LIST (Dutch trick)
• Schindlers list (E) vs. Schindlers list (D)

17
Interlingual homographs x list composition x task

• Experiments by Dijkstra, Van Jaarsveld, Ten
  Brinke (1998)
• English Lexical Decision with IHs, but without
  exclusively Dutch words
• English Lexical Decision with IHs and Dutch
  words. Dutch words require a no response
• Generalized Lexical Decision with IHs and Dutch
  words. English and Dutch words require a yes
  response
• Late Dutch-English bilinguals, students

18
Dijkstra, Van Jaarsveld, Ten Brinke (1998)

• Predictions
• Experiment 1 replication of Gerard
  Scarborough (1989) --gt null effect for
  homographs
• Experiment 2 distinction between English and
  Dutch words required --gt inhibition for
  homographs
• Experiment 3 both English and the Dutch reading
  of homographs can be used --gt facilitation for
  homographs

19
Stimulus Materials

• Four frequency categories of IHs
• HFE - HFD (BAD)
• HFE - LFD (LIST)
• LFE - HFD (BOOM)
• LFE - LFD (SMART)
• One-language matched English controls
• Nonwords, derived from English words
• Only in Experiments 2 and 3 exclusively Dutch
  words

20
English lexical decision without Dutch words
Stimulus list composition
English lexical decision including Dutch words
Task demands
Generalized lexical decision
21
Interlingual homographs Different proficiency
groups

• Schulpen, Dijkstra, Schriefers (in preparation)
  conducted the same three experiments with four
  different proficiency groups (and three frequency
  categories)
• 15 year old high school students (3VWO)
• 17 year old high school students (5VWO)
• Students of psychology
• Ph.D. students and university researchers
• All result patterns were very similar for
  Experiments 2 and 3 only for Experiment 1 there
  was some change in result patterns across
  proficiency groups

22
15 years old
17 years old
Psychology students
English lexical decision including Dutch words
PhDs
23
Conclusions

• Systematic effects of list composition and task
  demands in lexical decision
• Processing system and task demands functioned
  analogously for bilinguals varying widely in L2
  proficiency
• Automatization already at an early stage of L2
  proficiency
• Degree of cognitive control in lexical decision
  is limited
• Similar word recognition processes and mechanisms
  apply to bilinguals with different proficiencies
  (from 15 years of age onwards) (and also in
  different modalities)

24
(2) Stimulus list and instruction

• Dijkstra, De Bruijn, Schriefers, Ten Brinke
  (2000)
• Dutch-English bilinguals performed an English
  lexical decision task in which they were
  instructed to say no to Dutch words, but Dutch
  words were presented only in the second half of
  the experiment
• If instruction determines response, RTs to IHs
  will be the same in both halves
• If stimulus list composition is important, RTs
  will differ

25
Results in two halves

• Part 1 null-effects
• Experiment 1
• from Dijkstra et al. (1998)
• Part 2 inhibition effects Experiment 2
• from Dijkstra et al. (1998)

26
Conclusion

• Stimulus list composition, not instruction, was a
  determinant of the inhibition effect for IHs
• Different types of context may have different
  types of effect
• Two accounts for these effects
• -- relative activation of English and Dutch
  language selective access in first part?
• -- setting of decision criteria

27
(3) Stimulus list composition

• Effects of cross-linguistic phonological (P)
  similarity in IHs interlingual homophones
• Several studies investigated P-effects in English
  lexical decision by bilinguals
• Dijkstra, Grainger, Van Heuven (1999)
• Lemhöfer Dijkstra (2004)
• Jared (in preparation)
• Example of homophone COW - KOU

28
Dijkstra, Grainger, Van Heuven (1999)
COW - KOU BEE
29
Lemhöfer and Dijkstra (2004)
(15.3)
(15.3)
30
Jared (in prep.)
1000
950
English lexical decision SANK (cinq) vs. SAND
900
Mean Decision Latency (ms)
850
800
750
700
650
Homophone
Control
Participant Group
31
French-English bilinguals Cognates and IHs
present
(12.1)
(20.6)
32
Median split Fast and slow responders
33
Conclusion

• Interlingual homophone effects are facilitatory
• in the absence of cognates
• But they can also be inhibitory, depending on
  stimulus list composition and L2 proficiency
• Potential issue in these studies
• Test and control items are matched, but are the
  different conditions comparable in distribution?

34
(4) Stimulus list composition

• Tuinman Dijkstra (in press in prep.)
• Two English lexical decision studies using
  triplets of matched items
• BRAND (O) - VET (OP) - CRISP (CON)
• AID (P) - BOND (OP) - TOY (CON)
• Original goal to look for P effects

35
Experiment 1
Experiment 2
(23.1)
(26.3)
(24.1)
(20.5)
(18.8)
(10.6)
36
Conclusion

• Stimulus list composition affects processing
  strategies
• As a consequence, performance may not reflect the
  effect of the intended manipulation - in our case
  OP/P and OP/O
• We need to incorporate decision criteria to
  account for strategic effects

37
(5) Task demands

• Cross-linguistic orthographic (O) similarity in
  cognates
• Dijkstra, Brummelhuis, Lemhöfer (in prep.)
• (1) COLOUR-KLEUR
• (2) WHEEL-WIEL
• (3) HOPE-HOOP
• (4) BAKER-BAKKER
• (5)
  ALARM-ALARM
• Two tasks Progressive demasking English
  lexical decision
• Prediction More O-overlap -gt Larger cognate
  facilitation

more O overlap
38
Progressive Demasking (PDM)

• Identify a target word
• Presentation of a target is alternated with that
  of a mask
• Mask presentation time decreases, target
  presentation time increases

39
RT(cog) - RT(con)

• Inhibition
• (ns) of
• cognates
• relative to
• controls!

40
RT(cog) - RT(con)
Facilitation of cognates, increasing with
similarity
41
Conclusion

• PDM and lexical decision usually correlate .90,
  but not in this case!
• Cognate effects are task-dependent
• In PDM, the item must be uniquely identified --gt
  similarity helps only partially
• In English lexical decision, response can be
  based on global activation in lexicon --gt
  similarity helps

42
General conclusion

• To fully understand bilingual processing, we must
  clarify the relationship between the language
  processing system and task / context factors
• In other words, we must develop a systematic and
  testable account of task and context factors
• --gt Part II

43
(No Transcript)
44
Part II Context incorporated

• Goals
• Describe a theoretical framework including
  context factors
• Examine the relationship between word
  identification and task system in more detail
• Consider how these systems are implemented in the
  brain

45
Context

• To what extent is language processing affected by
  different context factors?
• Two general options
• Context and activation of representations are
  completely interwoven and cannot be disentangled
• In this case, we can only make a list of more and
  less important factors
• Some context factors (e.g., task) can be
  separated from representational processes

46

• Language makes use of perceptual, motor, and
  association functions and is widely represented
  over the brain

47
Non-linguistic context -----gt
BIA model
Linguistic context -----gt
48
BIA model

• Extends the BIA model with respect to
• Representations O, P, S
• Task / decision system
• Interaction within the language processing
  system O/P/S codes sentence context (linguistic
  context) can affect word recognition
• Task system (non-linguistic context) does not
  directly modulate activation of L1 and L2 words
• Decision / response criteria can be changed

49
A. Implementation (SOPHIA)

• Van Heuven Dijkstra (in prep.)
• Localist connectionist model
• Sublexical representations
• Problem of position-specific letter coding in
  other models
• Solution onset, nucleus, and coda (ONC)
  representations
• Example BOOK
• Orthography B, OO, K
• Phonology /b/, /u/, /k/
• Frequency-sensitive ONC mapping orthography to
  phonology
• Based on O-P mappings in CELEX database (gt 8000
  monosyllabic English words)

50
Orthography to phonology
Letter clusters can consist of single letters
51
(6) Interlingual phonological priming
Brysbaert, Van Dijck, Van de Poel (1999) Dutch
homophones prime French words WIE (DW)- OUI SOER
(DNW) - SOURD
.41 .34 .16
.30 .23 .17
Proportion perceptually identified
52
B. No top-down effect of task on lexical
activation

• Suppose that suppression of non-target language
  activation occurs when it is beneficial in a task
  situation
• Suppression should then be visible in RTs
  relative to a similar task situation where no
  suppression is needed
• However, so far there is little evidence in
  support of this view

53
Empirical evidence

• Homograph studies no suppression of non-target
  reading is possible
• Phonological effects Dutch P-effects still
  present in purely English lexical decision
• Experiment with two halves differing in list
  composition English control items had almost the
  same RTs (581 ms in part 1 vs. 592 ms in part 2)

54

• Control conditions in three experiments by
  Dijkstra et al. (1998) and had nearly the same
  RTs across very different task situations (max.
  15 ms difference)
• Control conditions in De Groot, Delmaar, Lupker
  (2000) also had similar RTs
• Mixed and pure PDM correlated .98 for English and
  Dutch targets (Van Heuven et al., 1998)

55
(7) Schulpen, Dijkstra, Schriefers (in prep.)
7. English RTs in three lexical decision
experiments for Dutch-English students of
psychology
No interaction between task and frequency
effect
56
Conclusion

• No evidence that the relative activation of words
  from two languages (relative language
  activation) is modulated by task demands or
  participant strategies
• No strong support for top-down feedback as in the
  language mode hypothesis (Grosjean, 1997)

57
(8) Bilingual semantic priming - test of
(sub)model
58
Kerkhofs, Dijkstra, Chwilla, De Bruijn
(submitted)

• ERP-study involving semantic priming with
  Dutch-English bilinguals
• English lexical decision on targets preceded by
  primes
• HEAVEN - ANGEL
• English prime - Interlingual homograph
• Stimuli
• 70 pairs related prime - homograph
• 70 pairs unrelated prime - homograph
• 70 pairs unrelated prime - filler word
• 2 105 pairs unrelated prime - nonword

59
Frequency manipulation

• English frequency IH high / low
• Dutch frequency IH high / low
• Participants
• - 10 bilingual participants - RT
• - 16 bilingual participants - RT ERP
• RTs unrelated homographs (1st
  presentation)

Inhibition
Inhibition
60
ERP measurements

• Recording parameters
• of the ERP study
• EEG was recorded from
• 27 electrodes,
• referenced to the left
• mastoid and digitized
• on-line with a sampling
• frequency of 200 Hz
• using a 12 bit A/D
• converter

61
Unrelated homographs Effect of Dutch frequency

• BUSH -ANGEL
• N400 effect

62
Related vs. unrelated homographs
Facilitation
Facilitation

• Inhibition by the Dutch reading of the IH in the
  HFD conditions is reduced by semantic relatedness
  
• --gt HEAVEN helps the English reading of ANGEL

63
Relatedness effect in N400

• English frequency of IH modulates size of N400
  effect interaction between Relatedness, English
  frequency, and midline electrodes

64
Dutch frequency effect on N400

• Dutch frequency of IH shifts ERP pattern in more
  negative direction main effect of Dutch frequency

65
Conclusions

• Semantic priming effect and N400 effect for
  interlingual homographs in a second language (L2)
• Effects of both L1 (Dutch) and L2 (English)
  frequency on the size and nature of the N400
  effect
• Cross-linguistic interference effects in both RTs
  and ERPs
• These ERP data are compatible with available RT
  data, which support the language nonselective
  access model
• But they reject a study by Rodrigues-Fornells,
  Rotte, Heinze, Nösselt, Münte (2002)

66
(9) Sentence context

• Dijkstra, Van Hell, Brenders (in prep.)
• Stimulus materials from Van Hell (in preparation)
  that were rated with respect to several important
  factors
• Presentation of sentences and target words using
  the RSVP (Rapid Serial Visual Presentation)
  technique
• Item presented for 345 ms, blank for 300 ms
• / The / man / brought / his / sick / son / to /
  the / doctor .

67
Task and design

• Lexical decision on target word (followed by dot)
• Measurement of ERPs for the same materials
• Manipulated dimensions
• High and low semantic constraint sentence context
  (CLOZE)
• Dutch and English sentence context
• Non-identical cognates and one-language target
  items

68
Reaction-time results

• Example sentences (high-constraint)
• The man brought his sick son to the doctor .
• De man bracht zijn zieke zoon naar de doctor .
• dokter (Dutch)

cognate facilitation
69
ERP results Constraint

• N400

High constraint black
70
ERP results Language

• N400

English sentence black
71
What is going on?

• The over-all effect of language is opposite to
  what one would expect in the N400 time window!
• English is more negative going than Dutch rather
  than the opposite
• However, we should look at the interactions of
  Language with Semantic Constraint and Cognate
  Status
• Further analyses show that the effect must be
  attributed to the language switch

72
English high constraint sentence cognates
easier to integrate than pure English words
N400
Dutch high constraint sentence cognates more
difficult to integrate than pure English words
73
Conclusions

• Cognate effects in RTs and ERPs can also be found
  for words in code-switched sentences
• At least three factors are affecting word
  recognition in sentences (cf. Altarriba et al.)
• Language of the preceding sentence
  (English/Dutch)
• Lexically expected item
• Semantic constraint (High/Low)

74
(10) Task, control, and brain

• Van Heuven, Schriefers, Dijkstra, Hagoort
  (submitted) fMRI study involving monolinguals
  and bilinguals
• Three lexical decision experiments
• English visual lexical decision (EVLD) by
  monolinguals
• English visual lexical decision by bilinguals
• Generalized visual lexical decision (GVLD) by
  bilinguals
• Participants
• 12 English monolinguals
• 2 12 Dutch-English bilinguals

75
Stimuli

• 36 interlingual homographs (e.g., BREED)
• English frequency 28 opm
• Dutch frequency 55 opm
• 36 English control words (e.g., BOUND)
• English frequency 28 opm
• 72 English filler words (e.g., BRUSH)
• 144 pseudowords (e.g., BINCH)
• Homographs were matched to English control words
  with respect to word frequency, word length,
  number of phonemes, and first phoneme

76
fMRI data Method and analysis

• Participants performed the tasks in a 1.5T
  scanner (TR 2000 ms, TE 40 ms, flip-angle 90
  degree)
• Letter strings were presented for 500 ms
• Data were preprocessed and analyzed with SPM99
• Goal of the study
• to test the effects of task differences
• to test the BIA models account of response
  conflict for IHs

77
Conflict in BIA
Response conflict in task/decision system -- in
EVLD only
Stimulus-based conflict in identification system
-- in EVLD and GVLD
--gt Can we demonstrate response conflict?
78
Lexical decision RTs(in scanner)

• English lexical decision
• Monolinguals IHs controls
• Bilinguals IHs gt controls
• Generalized lexical decision
• Bilinguals IH controls

79
Dutch-English bilinguals comparison of tasks
Activation for interlingual homographs greater
than for English control words
Left inferior / middle frontal cortex
Dorsal Anterior Cingulate Cortex
Generalized lexical decision
English lexical decision
Analysis of correct responses
80
Bilingual data

• Bilingual EVLD
• - Activation IH gt Con in left inferior /
  middle frontal gyrus (BA 44/45/46)
• - Activation IH gt Con in medial part of
  superior frontal cortex (BA 32/6/8) part of the
  dorsal anterior cingulate cortex (dACC)
• - Behavioral data slower RTs for IHs than for
  Controls
• Bilingual GVLD
• - Activation IH gt Con in left inferior /
  middle frontal gyrus
• - Activation IH Con in medial / superior
  frontal cortex
• - No behavioral differences between IHs and
  controls
• Monolingual EVLD
• - No activation differences for IH and Con
• - Behavioral data no RT differences for IH
  and Controls

81
Task differences

• Strong dACC activity is present in the EVLD but
  not in the GVLD
• There are also inhibitory effects in the RTs for
  the IHs in the EVLD
• The dACC has been associated with conflict
  detection and monitoring incompatible responses
  (Barch et al., 2000 Botvinick et al., 2001
  Gehring et al., 2001)
• The findings can be interpreted as the
  consequence of response-based conflict
• This view is supported when we examine the BOLD
  responses in the dACC and left middle frontal
  cortex

82
Dutch-English bilinguals in EVLD
Activation IH gt Con
z 45
z 22
83
Basic findings

• BOLD for IH higher than for Con and PsW
• BOLD for Con not different from that for PsW
• No activation difference between control words
  and pseudowords
• Thus, the activation difference for IH is not
  caused by the difference in lexical status
• One explanation is that the IH effect arises
  outside the lexicon -gt Support for response
  conflict account

84
Direct comparison of tasks
Activation IH gt Con for EVLD than for GVLD in
dACC / pre-SMA, basal ganglia (caudate nucleus),
and cerebellum
85
Network account

• A network for action is recruited to resolve the
  response conflict for IHs
• Left Prefrontal multiple representations Cortex
  (BA46, 9) in working memory IH gt
  Con (retrieved from posterior
  language areas)
• dACC IH gt Con IH linked to two different
  responses
  (actions)
• Basal ganglia resolving the conflict
• Cerebellum

86
Conclusions

• Behavioral results for Dutch-English bilinguals
  indicate that both readings of an IH become
  active and compete
• Imaging data show that the most involved brain
  areas are located in the left superior frontal
  regions (medial part) and bilaterally in the
  inferior and middle frontal regions
• The dorsal anterior cingulate cortex is more
  active when ambiguous language information leads
  to competing task-relevant responses
• Language conflicts are resolved in brain areas
  associated with cognitive control

87
General conclusions

• ERP and fMRI data for interlingual homographs are
  compatible with language non-selective RT data
  (rejecting Rodriguez-Fornells et al.s, 2002,
  view of selective access)
• They provide non-trivial additional information
  about the division between a representational
  system and cognitive control / task aspects
• So far, both RT and neurophysiological data are
  quite compatible with the BIA framework

88
Questions under investigation

• Do some context factors change the activation of
  the lexical items?
• What (on-line) role can language membership
  information play?
• What is the nature of the decision criteria and
  strategies that participants use?

89
(No Transcript)