Brain, Psycholinguistics,

1
Brain, Psycholinguistics, Cognitive Science
2
Outline

• How does psycholinguistics fit within the
  umbrella of cognitive science?
• What do we know about language and the brain?

3
Inter-relationships
4
What do cognitive psychologists do?

• Construct theories that describe how mental
  processing works in real time
• Within some domain, what are the mental
  representations and how are they manipulated?
• How does information flow through the mind?
• How is processing impacted by memory constraints,
  stimulus quality, mode of input, tasks, etc?
• Levels of processing
• Serial vs. parallel processing
• Top-down/bottom-up
• Automatic vs. strategic processing

5
What do (formal theoretical) linguists do?

• Construct formal theories of our linguistic
  knowledge
• Sets of rules/principles/assumptions for
  generating utterances
• Criteria for a good theory
• The rules generate all and only grammatical
  outputs (intuitions)
• In the simplest way

6
Derivational Theory of Complexity

• Millers clause-processing model of syntactic
  processing, motivated by Chomskys
  Transformational Grammar.
• Example of taking a linguistic theory and trying
  to implement it directly as a psychological
  theory of processing.

7
Transformational Grammar

• Two levels of syntactic structure, related by
  transformational rules accounts for similarity
  in meaning across different strings.
• Surface structure ((John) (picked __ (the box)
  up)).
• Transformations Particle Movement
• Deep Structure ((John) (picked up (the box))).
• Phrase structure rules and lexical insertion
  rules used to construct DS

8
Diff SS, Same DS

• Transformations can move, delete, or add
  words. Complex SSs require a sequence of
  transformations
• Wh-movement Subj/Aux Inversion
• SS Which book did Jim buy __?
• SS Jim did buy a book.
• DS Jim did buy a/which book
• Passivization
• SS The beer was drunk by Jim.
• SS Jim drank the beer.
• DS Jim drank the beer.

9
DTC

• Determine the surface structure of the sentence
• Reverse the transformations one by one
• Recover the deep structure
• Map DS to meaning

10
Processing Unit is entire Sentence

• Wh-movement Subj/Aux Inversion
• SS Which book did Jim buy __?
• SS Jim did buy a book.
• DS Jim did buy a/which book
• Passivization
• SS The beer was drunk by Jim.
• SS Jim drunk the beer.
• DS Jim drunk the beer.

11
How would you test the DTC?

• Early evidence seemed to support it.
• Later evidence was problematic
• No longer a viable theory of sentence
  comprehension.

12
Memory Language

• Semantic Memory LTM storehouse of conceptual
  knowledge
• What is a cow? What is truth?
• Lexicon LTM storehouse of knowledge about words
• Lexical Entry For each word/morpheme,
• Spelling, pronunciation, syntactic category,
  pointer to semantic memory
• Episodic Memory LTM storehouse of our experiences

13
Spreading Activation Model (Collins Loftus,
1975)

• Length codes typicality
• Link codes type of relation
• Some redundancy

14
Commmon Current Assumptions about Semantic Memory
the Lexicon

• Semantic Memory is a network of concepts,
  organized by semantic similarity
• Lexicon is a network of words, organized by
  phonological similarity
• Interconnections link meanings to words

15
Cognitive Neuroscience

• Cognitive psychologists tend to talk about the
  architecture of the mind in terms of
  functionality
• E.g., what is the input to word recognition? What
  is the output?
• Ultimately, the mental operations described by
  cognitive psychology occur in the brain
• In some cases, neuroscience can inform cognitive
  psychology
• E.g., we may be able to learn about how
  words/concepts are represented by investigating
  activation patterns in response to different
  classes of words action verbs activate motor
  cortex perception verbs activate visual cortex

16
Brain Anatomy ( Language)

• Is language localized in the brain?
• Is language lateralized?

17
LH lobes
Parietal lobe
Frontal lobe
Occipital lobe
Temporal lobe
Cerebellum
18
Some Lg-relevant areas
19
Video Clip (15 min)

• Brain Story First among Equals
• The first segment has been comparing human and
  chimp abilities to plan. They conclude that
  chimps can make and execute plans, but not as far
  into the future as humans.
• Well watch a segment on LG
• Aphasia patient
• Mapping the brain prior to surgery

20
What did you learn from the clip?

• What does it mean to have aphasia?
• Do aphasics recover language function? How?
• What is the current view on the role of Brocas
  and Wernickes areas?

21
Schiff et al. (2005)

• Do minimally conscious patients process speech?

22
Averaged healthy data
Figure 1. Functional maps obtained during
listening to narratives
Yellow forward Blue backward Red both
Schiff, N. D. et al. Neurology 200564514-523
23
Figure 2. Volumes of activation during the
passive listening tasks. The 2 patients are in
blue and red Averaged healthy activation in
black.
Schiff, N. D. et al. Neurology 200564514-523
24
Schiff et al.

• paper was published shortly after the Terry
  Schiavo media/political frenzy.
• How do we decide whether a minimally conscious
  patient is experiencing a life worth living?
• Is language comprehension relevant?
• Why dont we care as much about how their brain
  responds to smells?

25
Speech Spoken Word Recognition
26
Outline

• Why is speech perception difficult for computers
  to do?
• Problem of Invariance
• How do humans do it so easily?
• Bottom-up information (acoustic signal)
• Top-down information (higher level context)

27
Automatic Speech Recognition Follies (David
Pogue, NYTimes, 8/15/02)

• What I said
• bookmark it
• Motorolla
• modem port
• a procedure
• and then stick it in the mail
• movie clips
• I might add
• Inscrutable
• the right or left

• What was transcribed
• book market
• motor roll a
• mode import
• upper seizure
• and dense thicket in the mail
• move eclipse
• I my dad
• in screw double
• the writer left

28
The Problem of Invariance

• Individual phonemes do not have invariant
  acoustic cues.
• Theres a lot of variability in the acoustic
  signal!
• Variance in the acoustic signal has many sources
• Coarticulation
• Differences among speakers
• Differences within speakers yelling/whispering,
  phone/in-person, etc.
• If the bottom-up acoustic signal doesnt provide
  consistent cues, how do we recognize
  phonemes/words?

29
Reading a spectrogram

• formants

Can you see invariant cues associated with /i/ ?
30
If we cant/dont rely solely on bottom-up input,
how do we recognize speech?

• Perceive speech as (intended?) articulatory
  gestures, not as acoustic signal?
• McGurk Effect
• Sine Wave Speech There are no essential acoustic
  properties that enable speech perception. Rather
  second-order changes in frequency and amplitude
  over time are responsible. sine wave speech demos
• Use top-down information (word and sentence
  context) to complement bottom-up information
• If so, when and how?

31
Is Speech Special?

• Modularity Thesis (Fodor, 1983)
• The mind is not a unified whole. In addition to
  central processes, there are specialized
  input-output modules
• Central decision making
• Input color perception, voice recognition
• Output throwing, touch-typing, articulation
• Modules are fast, informationally-encapsulated,
  mandatory, exhibit characteristic breakdowns, and
  have shallow outputs.
• Speech perception may be handled by a specialized
  input module

32
The Essence of Motor Theory

• Speech perception is grounded in our knowledge of
  speech production. We recognize phonemes by
  covertly re-creating the articulatory gestures.
  (Lieberman et al., 1967)
• Consistent with philosophy that performance
  perception are inextricably linked.
• Assume innate, encapsulated phonetic module

33
Spoken Word Recognition

• Overcoming the problem of invariance in speech
  perception
• Motor Theory
• Top-Down Feedback Word to Phoneme
• TRACE
• Cohort theory of Spoken Word Recognition

34
Evidence for Top-Down influence on speech
perception

• Phoneme Restoration Effect (Warren, 1970)
• Lexical bias in categorical perception task, e.g.
  dype vs. type (Clifton Connine, 1987)

35
TRACE (McClelland Elman, 1986)

• Interactive connectionist model
• Nodes in network represent phonetic features,
  segments, words
• Feature nodes activated by consistent input
• Activation spreads up through network back down
  again
• Predicts top-down effects

36
Example initial phoneme in pick is ambiguous
betwn /b/ /p/.(Lexical Bias)
37
Is word recognition Automatic Modular?

• Automatic Processes
• Fast
• Do not require attention
• Feed-forward (cant be guided, controlled, or
  stopped midstream)
• Not subject to top-down feedback (informational
  encapsulation)

38
Stroop Effect

• Name font color

RED GREEN BLUE YELLOW GREEN
What happens if you have to name word?
39
Differences between spoken and written word
recognition

• For relatively short words, letters in a written
  word are processed in parallel
• Eye movement data
• Word superiority effect
• Letter-Search Task
• Spoken word unfolds across time
• Can recognize some words before they are
  completely pronounced.

40
Research on the Lexicon (Outline)

• How are lexical entries accessed? (Word
  Recognition) What is the input?
• Speaking (Ashcraft)
• Reading
• Listening
• How is lexical ambiguity resolved?