Cognitive Principles in Tutor

1
Cognitive Principles in Tutor e-Learning Design

• Ken Koedinger
• Human-Computer Interaction Psychology
• Carnegie Mellon University
• CMU Director of the Pittsburgh Science of
  Learning Center

2
Lots of Lists of Principles 1

• Cognitive Tutor Principles
• Koedinger, K. R. Corbett, A. T. (2006).
  Cognitive Tutors Technology bringing learning
  science to the classroom. Handbook of the
  Learning Sciences.
• Anderson, J. R., Corbett, A. T., Koedinger, K.
  R., Pelletier, R. (1995). Cognitive tutors
  Lessons learned. The Journal of the Learning
  Sciences, 4 (2), 167-207.
• Multimedia eLearning Principles
• Mayer, R. E. (2001). Multimedia Learning.
  Cambridge University Press.
• Clark, R. C., Mayer, R. E. (2003). e-Learning
  and the Science of Instruction Proven Guidelines
  for Consumers and Designers of Multimedia
  Learning. San Francisco Jossey-Bass.
• How People Learn Principles
• Donovan, M. S., Bransford, J. D., Pellegrino,
  J.W. (1999). How people learn Bridging
  research and practice. Washington, D.C.
  National Academy Press.
• Progressive Abstraction or Bridging Principles
• Koedinger, K. R. (2002). Toward evidence for
  instructional design principles Examples from
  Cognitive Tutor Math 6. Invited paper in
  Proceedings of PME-NA.
• Other lists on the web
• See learnlab.org/research/wiki

3
Principles on web See learnlab.org/research/wiki
4
Overview

• Cognitive Tutor Principles
• Multimedia Principles
• Theoretical Experimental evidence
• Instructional Bridging Principles
• Need empirical methods to apply
• PSLC Principles

5
Cognitive Tutor Principles

• Represent student competence as a production set
• Communicate the goal structure underlying the
  problem solving
• Provide instruction in the problem-solving
  context
• Promote an abstract understanding of the
  problem-solving knowledge
• Minimize working memory load
• Provide immediate feedback on errors

6
1. Represent student competence as a production
set

• Accurate model of target skill to
• Inform design of
• Curriculum scope sequence, interface, error
  feedback hints, problem selection promotion
• Interpret student actions in tutor
• Knowledge decomposition!
• Identify the components of learning

7
6. Provide immediate feedback on errors

• Productions are learned from the examples that
  are the product of problem solving
• Benefits
• Cuts down time students spend in error states
• Eases interpretation of student problem solving
  steps
• Evidence LISP Tutor
• Smart delayed feedback can be helpful
• Excel Tutor

8
Feedback Studies in LISP Tutor (Corbett
Anderson, 1991)
Time to Complete Programming Problems in LISP
Tutor Immediate Feedback Vs
Student-Controlled Feedback
9
Tutoring Self-Correction of Errors

• Recast delayed vs. immediate feedback debate as
  contrasting model of desired performance
• Expert Model
• Goal students should not make errors
• Intelligent Novice Model
• Goal students can make some errors, but
  recognize them take action to self-correct
• Both provide immediate feedback
• Relative to different models of desired
  performance

Mathan, S. Koedinger, K. R. (2003). Recasting
the feedback debate Benefits of tutoring error
detection and correction skills. In Hoppe,
Verdejo, Kay (Eds.), Proceedings of Artificial
Intelligence in Education (pp. 13-18). Amsterdam,
IOS Press. Best Student Paper.
10
Intelligent Novice Condition Learns More
F 4.23, p lt .05
11
Learning Curves Difference Between Conditions
Emerges Early

• Number of attempts at a step by opportunities to
  apply a production rule

12
Overview

• Cognitive Tutor Principles
• Multimedia Principles
• Theoretical Experimental evidence
• Instructional Bridging Principles
• Need empirical methods to apply
• PSLC Principles

13
Media Element Principles of E-Learning

• 1. Multimedia
• 2. Contiguity
• 3. Coherence
• 4. Modality
• 5. Redundancy
• 6. Personalization

14
Cognitive Processing of Instructional Materials

• Instructional material is
• Processed by our eyes or ears
• Stored in corresponding working memory (WM)
• Must be integrated to develop an understanding
• Stored in long term memory

Narration
Auditory WM
Build Referential Connections
Long Term Memory
OnScreen Text
Animation
Visual WM
15
Multimedia Principle

• Which is better for student learning?
• A. Learning from words and pictures
• B. Learning from words alone
• Example Description of how lightning works with
  or without a graphic
• A. Words pictures
• Why?
• Students can mentally build both a verbal
  pictorial model then make connections between
  them

16
Coherence Principle

• Which is better for student learning?
• A. When extraneous, entertaining material is
  included
• B. When extraneous, entertaining material is
  excluded
• Example Including a picture of an airplane being
  struck by lightning
• B. Excluded
• Why?
• Extraneous material competes for cognitive
  resources in working memory and diverts attention
  from the important material

17
Modality Principle

• Which is better for student learning?
• A. Spoken narration animation
• B. On-screen text animation
• Example Verbal description of lightning process
  is presented either in audio or text
• A. Spoken narration animation
• Why?
• Presenting text animation at the same time can
  overload visual working memory leaves auditory
  working memory unused.

18
Working Memory Explanation of Modality

• When visual information is being explained,
  better to present words as audio narration than
  onscreen text

19
Scientific Evidence That Principles Really Work
Summary of Research Results from the Six Media
Elements Principles. (From Mayer, 2001)
Used similar instructional materials in the
same lab.
20
Summary of Media Element Principles of E-Learning

• Multimedia Present both words pictures
• Contiguity Present words within picture near
  relevant objects
• Coherence Exclude extraneous material
• Modality Use spoken narration rather than
  written text along with pictures
• Redundancy Do not include text spoken
  narration along with pictures
• Personalization Use a conversational rather than
  a formal style of instruction

21
Overview

• Cognitive Tutor Principles
• Multimedia Principles
• Theoretical Experimental evidence
• Instructional Bridging Principles
• Need empirical methods to apply
• PSLC Principles

22
How People Learn Principles

• How People Learn book
• Build on prior knowledge
• Connect facts procedures with concepts
• Support meta-cognition

Bransford, Brown, Cocking (1999). How people
learn Brain, mind, experience and school. D.C.
National Academy Press.
23
But What prior knowledge do students have?How
can instruction best build on this knowledge?
24
Instructional Bridging Principles

• 1. Situation-Abstraction Concrete situational
  lt-gt abstract symbolic reps
• 2. Action-GeneralizationDoing with instances lt-gt
  explaining with generalizations
• 3. Visual-VerbalVisual/pictorial lt-gt
  verbal/symbolic reps
• 4. Conceptual-ProceduralConceptual lt-gt procedural

Koedinger, K. R. (2002). Toward evidence for
instructional design principles Examples from
Cognitive Tutor Math 6. Invited paper in
Proceedings of PME-NA.
25
Dimensions of Building on Prior Knowledge

• Situation (candy bar) vs. Abstraction
  (content-free)
• Visual (pictures) vs. Verbal (words, numbers)
• Conceptual (fraction equiv) vs. Procedural
  (fraction add)

Situation
Abstraction
26
Which is easier, situation or analogous abstract
problem?
27
Which is easier, situation or analogous abstract
problem?
28
Key PointDesign principles require empirical
methods to successfully implement
29
Overview

• Cognitive Tutor Principles
• Multimedia Principles
• Theoretical Experimental evidence
• Instructional Bridging Principles
• Need empirical methods to apply
• PSLC Principles

30
PSLC Theory Wiki Instructional principle
study pages demo
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
more in demo
38

• External validity of example-rule coordination
• Same principle tested across
• different background contexts (bs)
• different courses

39
Cross-Cluster Theoretical Integration Assistance
Dilemma

• How should learning environments balance
  information or assistance giving and withholding
  to achieve optimal student learning?
• Koedinger Aleven, 2007

• Row 1 illustrates how higher levels of
  instructional assistance can sometimes be a
  crutch that harms learning, but other times be
  a scaffold that bootstraps learning. Row 2
  illustrates how lower levels of assistance (or
  inversely greater imposed demands on students)
  can sometimes lead to poorer learning and other
  times lead to better learning. A long line of
  research on cognitive load theory (e.g.,
  Sweller, Van Merriënboer, Paas, 1998) suggests
  how some typical forms of instruction, like
  homework practice problems, put extraneous
  processing demands (or extraneous load) on
  students that may detract from learning. Another
  line of research on desirable difficulties
  suggests ways in which making task performance
  harder during instruction (reducing assistance),
  for instance, by delaying feedback, enhances
  learning (Schmidt and Bjork, 1992). And even
  within cognitive load theory, some task demands
  (e.g., increased problem variability) elicit
  germane rather than extraneous cognitive load
  and lead to better learning.
• Long-standing notions like zone of proximal
  development (Vygotsky, 1978), aptitude-treatment
  interactions (Cronbach Snow, 1977), or
  model-scaffold-fade (Collins, Brown, Newman,
  1990) suggest that instructional assistance
  should be greater for beginning learners and be
  reduced as student competence increases. So,
  whats the dilemma? Why not just give novices
  high assistance and fade it away as they become
  more expert. The theoretical problem, the
  dilemma, is that current theory does not predict
  how much assistance to initially provide nor when
  and how fast to fade it. It does not provide
  predictive guidance as to when an instructional
  demand is germane or extraneous, desirable
  or undesirable. The Assistance Dilemma remains
  unresolved because we do not have adequate
  cognitive theory to make a priori predictions
  about what forms and levels of assistance yield
  robust learning under what conditions.
• In Koedinger et al. (2008), we outlined the
  following steps toward resolving the dilemma
• 1. Decompose Identify and distinguish relevant
  dimensions of assistance, like giving lots of
  example solutions vs. withholding them
  (problems), giving vs. withholding immediate
  feedback, giving low vs. high variability
  examples.
• 2. Integrate For each dimension Collect,
  summarize, and integrate the relevant empirical
  and theoretical results from the research
  literature.
• 3. Mathematize For each dimension Characterize
  a set of conditions and parameters that can be
  used as part of a precise theoretical model that
  makes computable predictions about robust
  learning efficiency.
• 4. Test Use the model to make a priori
  predictions about what level(s) of assistance
  under what conditions yield the greatest robust
  learning efficiency. Test those predictions in
  laboratory and in vivo experiments.
• A key goal of PSLCs theory wiki is to get
  researchers working together, both within PSLC
  and within the broader learning science and
  education research community, to perform the
  gargantuan effort implied by steps 1 and 2.
  illustrates how we have carried out steps 3 and 4
  with respect to the practice spacing dimension
  of assistance (Koedinger et al., 2008 Pavlik
  Anderson, 2005).
• Figure . In the lower left is the assistance
  curve for the practice-spacing dimension. The
  top-level equation that generates the curve is
  shown above where effm is the y-axis and m is the
  y-axis. Other equations, not shown, map from m
  to the variables that have m as subscript pm,
  gm, and tm.
• An output of step 3 is a mathematical function
  (or set of functions) that can produce an
  assistance curve. As shown on the left in ,
  this curve has an inverted-U shape for most
  reasonable values of the parameters in the
  equation (shown on the right). Consistent with
  notions like zone of proximal development
  described above, we suspect this inverted-U form
  will characterize most assistance dimensions.
  But, the key to resolving the assistance dilemma
  is creating the mathematical equations and
  parameters that generate the inverted-U in way
  that is consistent with cognitive theory and with
  available empirical data.
• Drawing on a number of PSLC projects and data
  from many domains, we have made considerable
  progress on a second dimension of assistance, the
  example-problem dimension -- see the Coordinative
  Learning section. The generation of the
  mathematical equations for this dimension (step
  3) are being driven in part by our SimStudent
  model (Matsuda et al., 2008) -- see the Data
  Mining, Knowledge Representation, and Learning
  section. More generally, use of the Assistance
  Dilemma has driven analysis and interpretation of
  many other PSLC projects, some of which are
  described below (e.g., in the Interactive
  Communication section).

• Need more in vivo experiments to produce better
  theory resolve dilemma

Need predictive theory when does assisting
performance during instruction aid vs. harm
learning
40
Exploring dimensions of instructional assistance
Examples Added
Example assistance
Home-work
Problem Solving
Tutored
Untutored
Low Feedback assistance Higher
41
A step toward resolving assistance dilemma with
very broad impact implications!
Instructor options
42
Summary of Learning Principles

• Lots of lists of principles
• 6 Cognitive Tutor Principles
• 6 Multimedia Principles
• See PSLC wiki for others
• Should be Based on both
• Cognitive theory
• Experimental studies
• Need Cognitive Task Analysis to apply
• Domain general principles are not enough
• Need to study details of how students think
  learn in the domain you are teaching

43
EXTRAS
44
2. Communicate the goal structure underlying the
problem solving

• Successful problem solving involves breaking a
  problem down into subgoals
• Reification making thinking visible
• Make goals explicit in interface

45
ANGLE Tutor for geometry proof
Working backward
Scaffolding diagrammatic reasoning
Extending physical metaphor search space
paths
Working forward
46
Common Error in ITS Design

• New notation new learning burden
• Examples Goal trees, visual programming
  languages
• Alternative Use existing interfaces as scaffolds
  
• Adding columns in a spreadsheet
• Used in Geometry Cognitive Tutor
• Writing an outline for a final report
• Used in Statistics OLI course

47
Principle 2a Create transfer appropriate goal
scaffolding interfaces

• Avoid creating new interfaces to scaffold goals
• Worth it when
• Alternative Find an existing interface to use
  for goal scaffolding
• Transfer appropriate because interface is
  useful outside of tutor
• Cost of learning interface is low or pays off

Savings in domain learning due to goal scaffolding
lt
Cost of learning new interface
48
3. Provide instruction in the problem-solving
context

• Research context-specificity of learning
• This is how students learn the critical if-part
  of the production rule!
• Does not address exactly when to provide
  instruction
• Before class, b/f each problem, during?
• LISP tutor
• Before each tutor section when a new production
  is introduced
• And as-needed during problem solving
• Cognitive Tutor Algebra course
• Instruction via guided discovery as demanded by
  students needs or when they do not discover on
  their own

49
3a. Use transfer appropriate problem solving
contexts

• Use authentic problems that make sense to kids
• Intrinsically interesting, like puzzles
• Relevant to employment, probs about
• Relevant to citizens, rate of deforestation
• Why?
• Motivation Inspire interest
• Cognitive So students learn the goal structures
  planning that will transfer outside of the
  classroom

50
4. Promote an abstract understanding of the
problem-solving knowledge

• To maximize transfer, want those variables in
  if-parts of productions to be as general as
  possible!
• Reinforce generalization through the language of
  hint feedback messages
• Cannot be simply directly applied
• Trade-off between concrete specifics vs. abstract
  terms students may not understand

51
5. Minimize working memory load

• ACT-R analogy requires info to be active in
  working memory to learn new production rules
• Minimize info presentation to only what is
  relevant to current problem-solving step
• Impacts curriculum design as well as declarative
  instruction
• Sequence problems so prior productions are
  mastered before introducing new productions
• Supported by research on Cognitive Load Theory
• Eliminate extraneous load, leave intrinsic load,
  optimize germane load

52
Contiguity Principle

• Which is better for student learning?
• A. When corresponding words pictures are
  presented far from each other on the page or
  screen
• B. When corresponding words pictures are
  presented near each other on the page or screen
• Example Ice crystals label in text off to the
  side of the picture or next to cloud image in the
  picture
• B. Near
• Why?
• Students do not have to use limited mental
  resources to visually search the page. They are
  more likely to hold both corresponding words
  pictures in working memory process them at the
  same time to make connections.

53
Redundancy Principle

• Which is better for student learning?
• A. Animation narration
• B. Animation, narration, text
• Example The description of lightning occurs in
  pictures, is spoken. Text with the same words is
  present or not.
• A. Animation narration
• Why?
• Text animation are both processed in visual
  working memory and may overload it. Further,
  students eyes may be looking at the text when
  they should be looking at the animation. So,
  leave out the text which is redundant.

54
Personalization Principle

• Which is better for student learning?
• A. Formal style of instruction.
• B. Conversational style of instruction
• Example Exercise caution when opening
  containers that contain pyrotechnics vs. You
  should be very careful if you open any containers
  with pyrotechnics
• B. Conversational
• Why?
• Humans strive to make sense of presented
  material by applying appropriate processes.
  Conversational instruction better primes
  appropriate processes because when people feel
  they are in a conversation they work harder to
  understand material.
• NoteRecent in vivo learning experiment in
  Chemistry LearnLab did not replicate this
  principle.