Week 4. Sentence processing

1
GRS LX 865Topics in Linguistics

• Week 4. Sentence processing

2
Complexity

• The sandwich arrived.
• The sandwich the judge ordered arrived.
• The sandwich the judge the president appointed
  ordered arrived.
• The president appointed the judge who ordered the
  sandwich that arrived.

3
Complexity

• The nanny was adored by all the children.
• The nanny who the agency sent was adored by all
  the children.
• The nanny who the agency that the neighbors
  recommended sent was adored by all the children.
• The neighbors recommended the agency that sent
  the nanny who was adored by all the children.

4
What makes those first sentences so difficult?

• Some kind of processing difficulty.
• Obvious candidate (Chomsky Miller 1963, Kimball
  1973) You cant keep track of more than two
  sentences at a time.
• The sandwich the judge the president
  appointed ordered arrived .
• If at any point you need more than two verbs to
  finish, its hard.

5
Processing load

• The idea behind this is that the human sentence
  processing mechanism has some limited amount of
  storage capacity. Its memory-related, in some
  sense.
• (Cf. the 7 2 digit spanshort term memory has
  limits, the parser is sensitive to those/similar
  limits)

6
Thats easy enough

• The celebrity that attacked the photographer
  apologized on national TV.
• The celebrity that the photographer attacked
  applied for a restraining order.
• The first one is slightly easier, but we have no
  explanation for it under the two sentences
  view.
• Whats different?

7
Perhaps its floating q-roles

• The celebrity that _ attacked the photographer
  apologized.
• Never more than one floating q-role.
• The celebrity that the photographer attacked _
  applied
• At one point, two floating q-roles.
• There seems to be something about hanging onto
  these nouns without having something to hook them
  onto. (Also sounds digit-span-like Theres a
  reason phone numbers are divided).

8
Complexity

• The nanny who the agency that John recommended
  sent was adored by all the children.
• (Thanks!) The nanny who the agency that you
  recommended sent was adored by all the
  children.
• Well, thats funnynow whats different?

9
Reference

• The nanny who the agency that you recommended
  sent was adored
• The nanny who the agency that John recommended
  sent was adored
• The nanny who the agency that the neighbor
  recommended sent was adored
• The nanny who the agency that they recommended
  sent was adored

10
It seems like theres a real differenceis there?

• Here is where the psycholinguistic experiment
  comes in.
• Suppose we want to testwhats the real
  difference in processing difficulty between
  these
• pronouns with a referent (you)
• proper names (John)
• definite descriptions (the student)
• pronouns without a referent (they)

11
Designing an experiment

• A couple of ways to go about this
• Questionnaire
• The rat the cat the dog chased caught died.
• (bad) 1 2 3 4 5 (good)
• On-line reaction time processing
• The rat --- --- --- --- ----- ------ ----.
• --- --- the cat --- --- ----- ------ ----.
• --- --- --- --- the dog ------ ------ ----.
• --- --- --- --- --- --- chased ------- ----.

12
Designing an experiment

• Questionnaires are easy, quick, easy to
  administer.
• They give you only course-grained judgments about
  the whole sentence (probably about the point of
  maximum complexity)
• On-line experiments are more difficult, but we
  can see where people get bogged down.

13
Conditions

• At the outset, we need to define what were going
  to test for.
• Suppose were going to do a simple test of the
  that-trace effect.
• The question is are sentences that violate the
  that-trace filter worse than those that dont?
• Who did John say that left?
• Which capybara did Madonna meet on Mars?

14
Confounds

• Controlling for confounds is one of the most
  important things you have to do.
• That-trace filter violations are not the only
  things that differentiate these sentences.
• Who did John say that left?
• Which capybara did Madonna meet on Mars?

15
Confounds

• Who did John say that left?
• Which capybara did Madonna meet on Mars?
• Differences in lexical frequency can have a big
  effect on processing difficulty/time.
• Differences in plausibility can have a big effect
  on ratings from subjects.
• Differences in length can conceivably play a
  role.
• Differences in structure can have an effect.

16
Confounds

• Who did John say that left?
• Which capybara did Madonna meet on Mars?
• The point is If you find that one sentence is
  judged worse than the other, weve learned
  nothing. We have no idea to what extent the
  that-trace violation played a role in the
  difference.

17
Confounds

• You want to do everything you can to be testing
  exactly what you mean to be testing for.
• We cant control frequency, familiarity,
  plausibility very reliablybut we can control for
  them to some extent.
• Who did John say that left?
• Who did John say left?
• Keep everything the same and at least they dont
  differ in structure, frequency, plausibilityonly
  in that-trace. (Well, and here, length).
• Howevernote that length now works against
  that-trace, unless shorter sentences are harder.

18
Conditions

• To start, we might say we want to test two
  conditions
• Sentences with a that-trace violation
• Sentences with no that-trace violation
• But we cant build these without a length
  confoundholding everything else constant, we
  still have one fewer words in the that-trace
  case. How do we solve this?
• How can we show that the effect of the extra word
  that isnt responsible for the overall effect?

19
Conditions

• The trick well use is to have a second set of
  conditions, testing only the exact length issue.
  Theres no that-trace problem in object
  questions, so we can compare
• Who did John say Mary met?
• Who did John say that Mary met?
• to see how the difference compares to
• Who did John say met Mary?
• Who did John say that met Mary?

20
Factors

• We now have two factorsour sentences differ in
  terms of
• subject vs. object question
• presence vs. absence of that
• When we analyze the result, we can determine the
  extent of the influence of the second factor by
  looking at the object condition and comparing it
  to the (disproportionately larger) effect of the
  presence of that in the subject condition.

21
Context

• It turns out that the context also seems to have
  an effect on peoples ratings of sentences.
• What comes before can color your subjects
  opinions. This too needs to be controlled for.
• One aspect of this is that we generally avoid
  showing a single subject two versions of the same
  sentence (more relevant when theyre more unique
  than the John and Mary sentences)the reaction to
  the second viewing may be based a lot on the
  first one.
• Another is that you want to give the sentences in
  a different order to different subjects.

22
Strategy

• You also dont want your subjects to catch on
  to what youre testing forthey will often see
  that theyre getting a lot of sentences with a
  particular structure and start responding to them
  based on their own theory of whether the sentence
  should be good or not, no longer performing the
  task.
• Nor do you want to include people who seem to
  simply have a crazy grammar (or more likely just
  arent understanding or doing the task).

23
Fillers

• The solution to both problems is traditionally to
  use fillers, sentences which are not really
  part of the experiment.
• These can provide a baseline to show that a given
  subject is behaving normally and can serve to
  obscure the real test items.
• Theres no answer to how many fillers should
  there be? but it shouldnt be fewer than the
  test items, and probably a 21 (fillertest item)
  ration is a good idea.
• Fillers cant be all good! About half should be
  bad.

24
Instructions and practice

• Another vital aspect of this procedure is to be
  sure that the subjects understand the task that
  they are supposed to be performing (and all in
  the same way).
• The wordings of the instructions and the rating
  scales are very important, and its a good idea
  to give subjects a few practice items before
  the test begins (clear cases for which the
  answers are provided).

25
Instructions

• Is the sentence grammatical? is not a good
  instruction.
• The closest the naïve subject can come to
  grammatical will probably be to evaluate based
  on prescriptive rules learned in grammar
  classesthe term does not have the same meaning
  in common usage.
• Is this a good sentence? also has problems.
• Id never say that, Id say it another way.
• That could never happen.

26
Numerical/category ratings

• How do you ask people to judge?
• Good/bad
• Forces a choice, for anything other than
  certainly good and certainly bad theres a
  chance that it doesnt reflect the subjects
  actual opinionno differentiation between
  great! and well, kind of ok
• Good/neutral/bad
• Neutral also tends to get used for I cant
  decide which is different from Im confident it
  has an in-between status (doesnt change much if
  you call it in-between)

27
Numerical/category ratings

• Rate the sentence (good) 1 2 3 4 5 (bad)
• Some people will never use the ends of the scale,
  likely to confound certainty with acceptability.
  Also, for certain applications, 3 is unusable.
• Rate the sentence (good) 1 2 3 4 (bad)
• Can be treated as a categorial judgment, may be
  able to factor out some personality aspects. This
  is the one I tend to like best.

28
Online tasks

• The nice thing about an online experiment is it
  to some extent takes it out of their hands. The
  subject simply reacts, and we time it.
• Nevertheless, it is still important to ensure
  that the subject is performing the task, paying
  attention.
• Often can be addressed by questions about the
  sentence afterwards they must answer.
• Feedback can strengthen the motivation.

29
The plan

• We are going to test for the effect of NP type in
  doubly-center-embedded contexts (cf. experiment 1
  from Warren Gibson 2002).
• Well use PsyScript on the Mac to do the actual
  experiment and get some data
• Then well go back to Excel and/or SPSS to
  analyze it and see what we got and to what extent
  our results are statistically valid.

30
The first part

• The first thing we need to do is come up with the
  items and fillers that we will use in the
  experiment. Agonizing yet character-building.
• What we need is a block of four sentences
  differing only in NP type, and then a comparable
  filler for each.

31
Homework

• Before next week, come up with four such sentence
  blocks and four filler items (two good, two bad),
  then well pool the results next week
• The student who the professor who I collaborated
  with had advised copied the article.
• The student who the professor who they
  collaborated with had advised copied the article.
• The student who the professor who Jen
  collaborated with had advised copied the article.
• The student who the professor who the scientist
  collaborated with had advised copied the article.
• The stone Toby threw missed the bucket and cost
  him the game.
• The secretary the president asked questions of
  was crazy left early.

32
?

• ? ?
• ?
• ? ?
• ? ?
• ?
• ?