Thinkaloud as a research tool: Elicitation, coding and analysis

1
Think-aloud as a research tool Elicitation,
coding and analysis

• GU Yongqi
• Victoria University of Wellington
• New Zealand
• peter.gu_at_vuw.ac.nz

2
This session

• What is think-aloud?
• How does think-aloud reveal strategies?
• Eliciting think-aloud data among young learners
• Coding and analysing think-aloud protocols

3
What is think-aloud?
4
How can think-aloud reveal strategies?
5
What strategies do you see?

• (dictionary work)
• S SMART is usually an adj, it doesnt seem an
  adj here. SMARTS, (reads in dictionary). SMART,
  SMART is definitely not an adj. here, so Ill go
  for the verb. Oh, theres such a meaning for
  SMART, it means to sting. No? SMARTS the eyes,
  oh, yes, it is to sting, its this meaning then.
  SMART, let me see if therere any set expressions
  that go with it. Ah, I see an adj with a similar
  meaning here. a SMART blow, a SMART blow, a
  good beating (wrong in dictionary), this, need to
  remember this.

6
Coding of previous data
7
Issues in think-aloud elicitation
8
Common problems and solutions-1
9
Common problems and solutions-2
10
Can primary school children think aloud?

• Children can do think-alouds, upper primary
  better than lower primary
• Chamot El-Dinary (1999, p.331) students as
  young as grade 1 were often able to describe
  their thinking in rich detail
• Gu, Hu, Zhang (2005) yes, but need intensive
  probing and immediate clarifications

11
Our elicitation problems
12
Our solutions-1
13
Our solutions-2
14
Coding and analysing think-aloud data
15
Coding and analysing think-aloud protocols

• Coding The quantitative approach to qualitative
  data
• Where do codes come from?
• Strategy coding What to code and how
• Quality of strategy use
• Coder subjectivity
• Specificity of strategies

16
Coding The quantitative approach to qualitative
data Adapted from Bernard, 1996
17
Where do codes come from?
18
Our Coding Scheme

• Top-down for framework
• Initial scheme based on
• Theoretical analysis
• 3 empirical studies
• O'Malley, Chamot, and Kupper (1989),
• Goh (2002), and Vandergrift (2003)
• Bottom up for fine-tuning
• Adding, deleting, and adjusting till saturation
  point

19
Theoretical analysis The listening task

• Simultaneous integration of information from
  multiple sources in WM
• Massively parallel, interactive processing
  (Lynch, 2002, p. 39) of phonetic, phonological,
  prosodic, lexical, syntactic, semantic,
  pragmatic, and background information..

20
The listening task

• Chaotic at first, both bottom-up and top-down
  processing to form a coherent mental
  representational framework (Kintsch, 1998) in WM.
  This framework then
• serves as a perceptual filter to suppress
  irrelevant upcoming information and
• fills meaning gaps through its influence on the
  listeners selection and use of comprehension
  strategies such as inferencing

21
The listening taskThree levels of cognitive
processing

• Perceptual processing selectively attending and
  screening of incoming sounds
• Parsing encoding for meaningful representation,
  i.e., making sense
• Utilization elaborating to enhance meaning and
  to store it in LTM
• Anderson (2000)

22
Listening strategies

• Good listeners
• Metacognitively
• in control of the listening process,
• actively engage in planning
• monitor incoming input for congruence with
  expectations to construct a mental representation
  of the text in memory
• Cognitively
• monitor and direct their attention on task,
• listen to larger chunks and use both top-down and
  bottom-up processing.
• relate new information to existing world
  knowledge and/or personal knowledge, and
  critically evaluate the usefulness of the
  information
• O'Malley, Chamot, and Küpper (1989) Vandergrift
  (2003)

23
Listening strategies

• Poor listeners
• Metacognitively
• rarely engage in comprehension monitoring
• Cognitively
• easily distracted by unknown words or phrases and
  were unaware of their inattention
• Use word-by-word, bottom-up decoding exclusively
• have fewer elaborations, and do not make any
  connections between the new information and their
  own lives
• O'Malley, Chamot, and Küpper (1989) Vandergrift
  (2003)

24
Listening strategiesbottom-up vs top-down
processing

• Established way of thinking
• higher proficiency - top-down processing
• lower proficiency - bottom-up processing
• (Chien Li, 1998 O'Malley, Chamot, and
  Küpper, 1989 Vandergrift, 2003)
• Some recent findings
• Wu (1998) two types of top-down processing
• Facilitating successful comprehension, skilled
  listener
• Compensatory less successful comprehension,
  unskilled listener
• Tsui Fullilove (1998) less skilled listeners
  weak in automatic decoding, and have to use
  top-down support for compensation, i.e., need
  better bottom-up skills, less wild guessing
• Field (2004) neither bottom-up nor top-down but
  lexical a rough attempt at a one-to-one match
  with a known item which potentially overrules
  contextual information and modifies perceptual
  (p. 373)

25
Bottom-up coding using NVivo

• Apply initial coding scheme on think-aloud data
• Add, delete, adjust categories based on data
• Example

26
Our final coding scheme Levels 1-3
27
Our coding scheme Levels 1-3
28
Our final coding scheme Levels 1-3
29
Our coding scheme Levels 3 4
30
Coding procedures

• Three independent coders
• Use NVivo
• Regular meetings for calibration
• Agree to a final coding scheme
• Cross coding to ensure consistency
• Coder A Coder B
• Coder B Coder C
• Coder C Coder A

31
Patterns of listening strategy useMean
Frequency of Strategy Use by Proficiency Level
32
Patterns of listening strategy use Mean
Frequency of Strategy Use by Grade
33
Coding and analysis Unresolved problems
34
Strategy codingWhat is coding for?

• pattern finding tallies, e.g., repertoire,
  frequencies
• hypothesis generation
• insights exploration

35
Strategy coding

• What do we code and how do we code
• highlighting strategy presence, but not absence
• mostly good strategies
• How do we code the following?
• motivation for strategy choice
• quality of strategy use
• flexibility of strategy use
• efficiency of a strategy
• orchestration of strategy use

36
Strategy codingQuality of strategy use From
whether to how

• The same strategy often used by both a less
  successful pupil and a more successful
  counterpart (e.g., inferencing and prediction)
• Good and poor listeners differed in other
  aspects, e.g.,
• varying accuracy of bottom-up decoding,
• retrieval of schemata of varying relevance, or
• ability to form a relevant conceptual framework,
• ability to adjust the framework according to new
  input, etc.
• Do we need to code these qualitative differences?
  How?

37
Strategy codingCoder subjectivity

• Quality of strategy use graded or binary?
• effective vs ineffective
• not at all effective, somewhat effective, very
  effective?
• Whose perspective researchers (Etic) or
  learners (Emic)? e.g.,
• maintaining on-task attention
• continuing to listen despite difficulty

38
Strategy codingSpecificity of strategies

• The greater specificity of the strategies, the
  more likely it is to find interpersonal
  differences.
• However, increasing specificity of strategies can
  make a taxonomy very unwieldy, and findings less
  generalizable.
• What is the optimal level of specificity? How is
  this to be determined?

39
Overall Summary

• A Chinese fable (in Lushi Chunqiu Chajin)
• A man from the state of Chu was traveling in a
  boat when his sword fell into the river. He
  instantly drew out his dagger and cut a mark on
  one side of the boat and said to himself this is
  where my sword fell. After the boat finally
  pulled ashore, he jumped into the water from the
  marked point of the boat and searched in vain for
  his sword.
• Moral
• when set out to get something, we need to get
  it the right way, or we risk not getting it at
  all.

40
Reflection

• Think of one point about the think-aloud
  technique that you have learned from this
  session.
• Share with someone next to you.

41
References

• Bernard, H. R. (1996). Qualitative data,
  quantitative analysis. The Cultural Anthropology
  Methods Journal, 8(1), Retrieved 2 September 2004
  from http//www.analytictech.com/borgatti/qualqua.
  htm.
• Chamot, A. U., El-Dinary, P. B. (1999).
  Children's learning strategies in language
  immersion classrooms. The Modern Language
  Journal, 83(3), 319-338.
• Chesterfield, R., Chesterfield, K. B. (1985).
  Natural order in children's use of second
  language learning strategies. Applied
  Linguistics, 6(1), 45-59.
• Cohen, A. D. (1998). Strategies in learning and
  using a second language. London Longman.
• Ericsson, K. A., Simon, H. A. (1993). Protocol
  analysis Verbal reports as data (Revised ed.).
  Cambridge, MA MIT Press.
• Flavell, J. H., Green, F. L., Flavell, E. R.
  (1995). Young children's knowledge about
  thinking. With commentary by Paul L. Harris and
  Janet Wilde Astington. Chicago Monographs of the
  Society for Research in Child Development, 60(1,
  Serial No.243).
• Goh, C. C. M. (2002). Exploring listening
  comprehension tactics and their interaction
  patterns. System, 30(2), 185-206.
• Gu, Y., Hu, G., Zhang, L. J. (2005).
  Investigating language learner strategies among
  lower primary school pupils in Singapore.
  Language and Education, 19(4), 281-303.

42
References

• Lan, R., Oxford, R. L. (2003). Language
  learning strategy profiles of elementary school
  students in Taiwan. IRAL, 41(4), 339-379.
• Meadows, S. (1993). The child as thinker The
  development and acquisition of cognition in
  childhood. London Routledge.
• O'Malley, J. M., Chamot, A. U., Kupper, L.
  (1989). Listening comprehension strategies in
  second language acquisition. Applied Linguistics,
  10(4), 418-435.
• Pressley, M., Borkowski, J. G., Schneider, W.
  (1987). Cognitive strategies Good strategy users
  coordinate metacognition and knowledge. In R.
  Vasta G. Whitehurst (Eds.), Annals of child
  development Vol. 5 (pp. 89-129). Greenwich, CT
  JAI Press.
• Purdie, N., Oliver, R. (1999). Language
  learning strategies used by bilingual school-aged
  children. System, 27, 375-388.
• Schouten-van Parreren, C. (1989). Vocabulary
  learning through reading Which conditions should
  be met when presenting words in texts? AILA
  Review 6 Vocabulary Acquisition, 75-85.
• Scott, J. (2000). Children as respondents The
  challenge for quantitative methods. In P.
  Christensen A. James (Eds.), Research with
  children Perspectives and practices (pp.
  98-119). London Falmer Press.
• Sugeng, B. (1997). A learning strategy profile of
  Indonesian elementary school students. RELC
  Journal, 28(2), 82-106.
• Vandergrift, L. (2003). Orchestrating strategy
  use Toward a model of the skilled second
  language listener. Language Learning, 53(3),
  463-496.

43
Thank you!