Using interviews to investigate implicit knowledge in computer programming

1
Using interviews to investigate implicit
knowledge in computer programming

• Amber Settle
• Educational research seminar
• September 29, 2006

2
Topic of talk

• Rebecca Mancy and Norman Reid
• Using interviews to investigate implicit
  knowledge in computer programming
• Appeared in the International Conference of the
  Learning Sciences in July 2006
• http//portal.acm.org/citation.cfm?id1150034.1150
  101colldlACMCFID15151515CFTOKEN6184618

3
ICLS

• International Conference of the Learning Sciences
• http//www.isls.org/
• http//www.isls.org/icls2006/
• Focus of the conference is both K-12 and
  college-level education

4
Brief history of implicit learning

• Term first surfaced in 1967
• A.S. Reber, Implicit learning of artificial
  grammars, Journal of Verbal Learning and Verbal
  Behavior
• M. Polanyi, The Tacit Dimension
• Investigated in both philosophy and psychology
• Mancy and Reid focus on psychological literature
• Interested in the processes by which it is gained

5
Reber

• Artificial Grammar Learning experiments
• Learning phase
• Subjects see strings of letters generated by
  finite-state machines
• They then carry out a task on them such as
  repeating them
• Testing phase
• Informed of existence of an underlying grammar
• See letter strings and classify them as belonging
  or not OR
• See pairs of letter strings, are told that one
  belongs and one does not, and must identify the
  one that does belong

6
Reber

• Results
• Participants are able to correctly classify
  strings at above chance level
• Participants cannot articulate the rules
• In short
• Act as though they possess abstract rules
• Rules are not declarative
• Participants claim no awareness of the rules
• Numerous subsequent studies have unambiguously
  demonstrated the same phenomenon
• Most studies concentrated on somewhat superficial
  forms of implicit learning

7
Implicit learning education

• Existence of two modes of learning
• With conscious awareness (explicit)
• Without conscious awareness (implicit)
• Important considerations
• Efficiency Finding the best mode for a given
  topic
• Strategies Encouraging both modes of learning
• Assessment Testing implicit knowledge is
  difficult

8
Implicit learning education

• Little research on implicit learning and
  education
• Possible reasons
• Assumption that explicit, articulate thinking is
  the most powerful form of cognition
• Narrow focus of research on implicit learning
• Rooted in experimental psychology
• Control as many parameters as possible
• Tasks that are far removed from complex,
  real-life learning situations
• Mancy and Reid Develop new research methods that
  can be applied to real-life situations
• Interviews

9
Problems with interviews

• Interview Verbal report of some knowledge
• Issues
• Verbal reports may not correspond to actual
  knowledge if not all memory is reported
• Ericsson Simon, 1980
• What is reported depends on a subjects
  perception of their understanding
• Nisbett Wilson, 1977
• Offer explanations plausible to them
• Report nothing or an alternative explanation

10
Problems with interviews

• Issues (continued)
• Conscious and unconscious knowledge may co-occur
• Subject may offer conscious knowledge
• Does not imply unconscious knowledge is not
  present
• Explicit knowledge may be offered but not govern
  behavior when faced with a task using the skills
• Potential
• Verbal reports have been used in areas such as
  sensory processes, imagery and immediate memory

11
Mancy and Reid

• Context
• First-year students
• Introductory Ada course
• University of Glasglow
• 2003-2004
• 12 (male) students recruited through requests
  during lecture and e-mails
• Conducted near the end of the course
• Paid for their assistance

12
Mancy and Reid

• Interview questions
• Decision points in programming
• Questions
• Asked to choose between possible language
  structures
• As a follow up, had they thought about this issue
  previously
• Examples
• How do you decide when to use a semicolon?
• Can you explain how you use indentation, how you
  indent your code, when you write it?
• How do you decide when to use an array?
• Topics
• Explicit rule When to choose a for versus a
  while loop
• No explicit rule When to choose an array

13
Mancy and Reid

• Process
• Semi-structured interviews
• Hour in length
• Tape recorded
• Transcribed in full by investigators
• Attention paid to aspects such as hesitation
• Grounded-theory approach
• Identify statements that seem to indicate
  implicit knowledge
• Later categorize them according to indicators of
  the implicit status of that knowledge

14
Mancy and Reid

• Indicators of implicit knowledge
• Little or no knowledge about how a skill is used
• Claims of difficulty in articulating knowledge
• Reliance on concrete examples
• Use of language
• Surprise

15
Mancy and Reid

• Little or no knowledge about how a skill is used
• Interaction
• Interviewer How did you decide when to use a
  semicolon?
• Victor I dont, I just do it. Occasionally
  like erm, I just its when, it comes
  naturally when Im typing something.
• No conscious consideration or deliberation

16
Mancy and Reid

• Reliance on concrete examples
• Interaction
• Halsey (in response to a question about using
  arrays) actually when you use an array its
  .. useful for .. erm lots .. of the same thing
  held like, if youre wanting to hold the number
  of cups of tea you drink in a day, you would have
  like you know the numbers would be day one to
  seven would be your day .. and inside each would
  be your number of cups of tea.
• Features
• Much more confident when describing the example
• Example not considered in lecture
• Mathematics education Concrete examples as
  prototypes (Tall Pinto, 2002)

17
Mancy and Reid

• Surprise
• Interaction
• Elwin (Asked to describe what might consitute a
  condition in a while loop) I suppose it could
  be any kind of expression sort of to do with
  while E is equal to five, or take it theres
  gonna be a value to come back? aha! Perhaps,
  does it always have to come back with a Boolean
  value, then?
• Interviewer It does.
• Elwin Yeah, it does, doesnt it! Sounds
  surprised
• Interviewer Youd not thought about that
  before?
• Elwin No no! laughs
• Elwin demonstrates noticing, where implicit
  knowledge becomes explicit (Schmidt 1990
  Truscott, 1998)

18
Mancy and Reid

• Implicit knowledge
• lends itself to explanation using examples
• is very loosely structured, with ideas linked
  associatively rather than logically
• Conclusions about process
• Subtle clues can be used to show implicit
  learning
• Interviews are a richer source of information

19
Mancy and Reid

• Suggest a graduated scale of conscious status of
  knowledge
• No awareness of any knowledge
• Awareness of knowledge, but no awareness of
  content
• Some awareness of knowledge, but it is structured
  in a way that makes it difficult to be
  communicated
• Structured declarative or explicit knowledge

20
Mancy and Reid

• Shortcoming
• Rate of occurrence of unambiguous evidence for
  implicit knowledge is low
• Method is inefficient
• Future work
• Change type and wording of questions
• Enable respondents to give pertinent answers
• Note Interview questions not included in paper

21
Upcoming seminar events

• Friday, October 27th, time and location TBA
• Workshop on teaching programming
• Presenter Keith Whittington of RIT
  (http//www.rit.edu/)
• Dr. Whittington has an NSF grant to study the
  teaching of programming and will also be here on
  Thursday, October 26th to meet with faculty on an
  individual basis
• Tell me TODAY if you want to meet with him on
  Oct. 26th or 27th
• Early November Distance learning panel
• Organized by Evelyn Lulis in conjunction with the
  Distance Learning Committee
• Topic Excellence in Teaching Distance Learning
• Panelists TBA
• Watch the Research Seminar page for details