Evaluating SchoolAge Childrens Fictional Narratives

1
Evaluating School-Age Childrens Fictional
Narratives

• Marc E. Fey, Ph.D.
• University of Kansas Intercampus
• Program in Communicative Disorders
• University of Kansas Medical Center
• Email mfey_at_kumc.edu
• Presented at the 2001 IALP World Congress/CASLPA
  Convention
  
• Montreal, Quebec, Canada
• August 07

2
Child Language Research CenterNIDCD
PI J. Bruce Tomblin University of Iowa Relati
onship Between Spoken and Written Language Hu
gh W. Catts Marc E. Fey
KU Intercampus Program in Communicative Disorders
3
Participants
Iowa Epidemiologic Study N 7218
Child Language Research Center N 604 (570)
Kindergarten Typical N 276
Kindergarten Problems N 328
Low Cognition N 80
Typical Language N 350
NonSpecific LI N 79
Specific LI N 47
4
Why Assess Narratives? (After Hughes, 1997)

• They are associated with other academic and
  social abilities.
  
• Both comprehension and production can be assessed.

5
Why Assess Narratives? (After Hughes, 1997)

• They allow for ecologically valid evaluation of
  many language and cognitive skills.
  
• connected speech and fluency
• language and story form and content in two
  modalities
  
• coherency and cohesion analyses
• handwriting, spelling, and punctuation

6
Why Assess Narratives?

• They are an important part of many cultures and
  have value in their own right.
  
• If story-telling is a part of the curriculum,
  abilities should be assessed in ways that lead to
  teaching strategies.
  
• Results lead to straightforward suggestions for
  treatment in an ecologically valid context.
  

7
Selecting the Story Stimuli

• Explain to the child that you want her to make up
  two stories, one to tell and one to write.
  
• Show the child 4 pictures, one representing each
  of 4 stories.
  
• The child picks one story to tell and one to
  write.
  

8
Demonstrating a Story

• Demonstrate what you mean, by telling a
  pre-written story
  
• Show the child a set of 3 pictures from a story
  she did not select.
  
• Identify the key elements of the story as shown
  in the pictures
  
• Tell a pre-written story about the pictures

9
Collecting the Stories

• Tell the child you would like her to tell a story
  for her mom or a friend and to be sure to tell a
  story dont just talk about the pictures.
  
• Show the child a set of 3 pictures from a story
  she selected earlier.
  
• Have the child identify the key elements of the
  story as shown in the pictures.
  
• Remind the child not just to talk about the
  pictures.
  
• Ask the child, Is that all? before stopping.

10
Transcribing the Stories

• Segment the utterances into C-units.
• includes one independent clause and all
  associated dependent clauses.
  
• Sentence fragments that are prosodically complete
  and/or grammatically distinct are transcribed as
  distinct C-units.

11
Inter-Judge Reliability (r) for Measures
Sensitive to Transcription Errors

• 
  Written Oral
  
• Total number of c-units - .99
  .99
  
• Total number of words - .99
  .99
  
• MLU
  .98 .97
  
• Number of different words - .99
  .99
  
• Clausal Density- .97
  .97
  
• Total number of coordinators- .99
  .99
  
• Percentage of c-units with errors- .89
  .83
  

12
Quantifying Story Quality

• Setting 0 - 3 points
• Characters 0 - 3 points
• Plot 0 - 6
• Ending 0 - 3
• Language Sophistication 0 - 3 points
• Maximum total 18 points

13
Scoring the Story Setting

• The setting includes explicit reference to the
  physical and temporal context prior to the
  establishment of a problem.
  
• 0 points No mention of the setting
• 1 point reference to parts of the setting
  observable in the story pictures

14
Scoring the Story Setting

• The setting includes explicit reference to the
  physical and temporal context prior to the
  establishment of a problem.
  
• 2 points reference to or description of
  non-pictured parts of the setting
  
• 3 points reference to non-pictured elements that
  play a key role in developing the problem or
  resolution

15
Scoring the Story Characters

• Includes explicit reference to all characters
  prior to the establishment of a problem.
  
• 0 points characters are not mentioned
• 1 point characters are labeled, including family
  relationships (e.g., mom, brother, sister)
  
• 2 points characters are given names or
  characteristics not observable in the pictures
  
• 3 points characteristics that play a key role in
  developing the plot are identified

16
Scoring Plot Development
Is there a nuclear dyad?
yes
no
Are actions included?
Is there more than 1 dyad?
no
Score 0
yes
no
yes
Are actions sequenced?
Are there complications?
Is 1 dyad embedded?
yes
no
no
yes
Score 2
Score 1
Score 4
Score 3
no
yes
Are there complications?
yes
no
Score 6
Score 4
Score 5
17
What is a Nuclear Dyad?

• Must contain an overtly identified Problem or
  Conflict
  
• The Problem/Conflict must be overtly Resolved
• Most dyads contain actions that move the
  characters toward the Resolution.

18
What is a Problem?

• an overtly identified need, desire, conflict,
  danger, or goal of the character(s)
  
• may be explicitly identified, often as an
  internal response
  
• actions of the characters may be elaborated in a
  way that makes a problem overt
  
• the resolution or ending may contain language,
  making the presence of a problem explicit

19
What is a Resolution?

• an overt indication that the problem has or has
  not been resolved, giving closure to an episode
  
• may be an explicit statement
• may be entailed in an ending statement indicating
  the characters future intentions or patterns of
  behavior
  

20
What is a Complication?

• any obstacle or interruption in the actions
  toward a resolution that adds tension, conflict,
  or drama
  
• must not result in the creation of an additional
  nuclear dyad

21
Scoring the Story Ending

• Includes relevant information, character
  responses to the resolution, and statements
  concerning future behaviors following the
  resolution of the final episode.
• 0 points no ending of any sort
• 1 point stereotypic endings (e.g., The end, They
  lived happily ever after)

22
Scoring the Story Ending

• Includes relevant information, character
  responses to the resolution, and statements
  concerning future behaviors following the
  resolution of the final episode.
• 2 points internal or external responses to the
  storys problem and/or resolution are provided.
  
• 3 points some statement indicating a moral or
  that future behavior will change as a result the
  characters experiences is provided.

23
Scoring Language Sophistication

• Designed to capture the childs use of the
  language of literacy or of narration.
  
• 0 points No use of target features
• 1 point Use of one or more of the target
  features
  
• 2 points Moderate use of one or more of the
  target features
  
• 3 points Established use of more than one of the
  target features to story-telling effect.

24
Target Features of Language of Literacy or
Narration

• Fronted adverbial clauses and phrases
• Relative clauses and post-modifying phrases
• Story vocabulary, including -ly adverbs
• Direct quotations
• Sentences with multiple auxiliaries
• Passive sentences

25
Psychometric Information

• Inter-judge reliability
• Predictive validity
• Correlations between 2nd and 4th grade measures
• Construct validity
• Effects of age and modality
• Effects distinguishing groups of children with
  different linguistic and cognitive profiles
  

26
Inter-judge Reliabilityfor the Narrative Quality
Score

• A primary judge scored all stories 2300
• A second judge scored 10 of the stories
  independently
  
• Correlation for Total Narrative Score - .83

27
Predictive Validity 2nd 4th Grade
Correlations

• 
  Written Oral
  
• Total number of c-units - .28
  .24
  
• Total number of words - .43
  .33
  
• MLU
  .25 .21
  
• Number of different words - .49
  .37
  
• Clausal Density- .21
  .14
  
• Total number of coordinators- .23
  .44
  
• Percentage of c-units with errors- .23
  .19
  
• Narrative quality score- .47
  .29
  

28
Construct Validity Differences Across Grade,
Modality, and Dx Group

• If the procedure is a valid indicator of
  narrative development, it should yield scorable
  stories that yield differences across grades,
  modalities, and diagnostic groups.

29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
Some Tentative Conclusions About the Procedure

• The procedure yields stories of sufficient length
  to allow for many different types of analysis.
  
• Stories can be reliably transcribed to yield
  quantitative measures of story length, content,
  and complexity.
  
• It has weak predictive validity, limiting its
  usefulness as a pre-/post-intervention measure.
  
• Clinicians might consider collecting 2 stories in
  each modality at each time to enhance stability
  of the scores.
  

37
Some Tentative Conclusions About the Procedure

• It has acceptable construct validity as a measure
  of narrative development.
  
• It should not be used as the sole basis for
  determining that a child has a language disorder,
  but may be useful as a screening instrument for
  children whose narrative skills are in question.
• It should be useful as a means of demonstrating a
  childs progress in story-telling, especially
  when examined qualitatively with other data,
  e.g., in portfolio development and assessment.

38
Some Tentative Conclusions About the Procedure

• There is no reason to believe that other measures
  developed by clinicians to assess narrative
  generation abilities across 2nd to 4th grade
  would be less subject to the same limitations
  observed in this procedure.
• Clinicians must be cautious when they apply their
  own methods of evaluation to childrens stories
  

39
Oral Story 2nd Grade Narrative Quality Score -
13

• e once upon a time, there was
• c a colt.
• c and he want/ed to make his mother proud of him
  so he told her that he could go into
  
• the place and get two big apples for them if
  she want/ed him to.
  
• c and he thought that she would not (let) want
  him to go over there.
  
• c but she said, "fine, then let me see you jump
  over the fence and get us some apples".
  
• c then, he jump
• c then, he was kind of scared.
• c and (he had) he took a jump over the fence
  grab/ed two apples in his mouth.
  
• c the bull with the sharp horns was come/ing
  right towards him.
  
• c but he ran as fast as he could and leap/ed over
  the fence with the two delicious apples.
  
• c and he escape/ed just in time.
• c she said, well then, that prove/3s that you
  can.
  
• c but you should not brag so much.
• f the end.