Kognitive Architekturen

1
Cognitive Modeling The Good, the Bad, and the
Ugly John R. Anderson, Carnegie
Mellon http//act.psy.cmu.edu
The Good There are now many models that fit the
data exquisitely. The Bad There are only a few
models that are not hand crafted to do the task
rather than learn to do the task like people. The
Ugly There are no models (to my knowledge) that
creatively come up with solutions in situations
for which they have prior preparation.
2
Outline of Talk

• Briefly describe a model that is both good and
  not bad That is, it both fits the data
  exquisitely and comes into being by learning from
  experimental instructions just like the
  participants.
• Show that this model is still ugly -- put it in
  an unanticipated situation and show it fall
  apart.
• But more important show that regular people (not
  just the Einsteins) are quite capable of
  inventing solutions in such unanticipated
  situations.
• Acknowledge the reasoning and discuss the
  metacognitive abilities that such a model would
  need not to be ugly -- i.e. to adapt to the
  unexpected.
• Warning This is a report of research in progress
  and I will end in an imcomplete state asking you
  for ideas.

3
The Good Experiment Qin, Anderson, Silk,
Stenger, Carter (2004)

• 11-14 year-olds just about to start Algebra 1
• Day 0 Instruction, paper pencil practice,
  coaching
• Days 1 - 5 Computer-based practice

4.Student types answer by pressing finger in data
glove. 5. Imaged in fMRI scanner on Day 1 and 5.
4
ACT-R 5.0 Modules and Buffers
ACT-R
Parse 3x-57
Visual Perception
Manual Control
Type x4
Production System
Declarative Memory
Retrieve 7512
ProblemState
Hold 3x12
Control State
Unwinding Retrieving
5
We gave ACT-R Verbal Instructions on the Same
Unwind Strategy as Taught to the Children

• The instructions are stored in declarative memory
  and retrieved
• ACT-R applying them to 7x 3 38
• Instruction 1a Create image 38
• Instruction 2b Unwind-right 7x3
• Instruction 3b Change image to 38-3, this to
  35, and focus on 7x
• Instruction 2c Unwind-left 7x
• Instruction 4b Change image to 35/7, this to
  5, and focus on x
• Instruction 2a The answer is 5, key it.
• Initially instructions interpreted by general
  production rules.
• Eventually production compilation produces
  task-specific production rules.

6
ACT-R Modules The first 2 Seconds 7x338
7
ACT-R Modules The middle 2 Seconds 7x338
8
ACT-R Modules The last 2 Seconds 7x338
9
Learning over 6 Days of Experiment
10
Prefrontal/Retrieval BA 45/46 (x -40, y 21,
z 21)
11
Comments on the Good Experiment

• Virtue The model is not hand crafted but learns
  from instruction (albeit the instructions are a
  little hand-crafted to facilitate parsing).
• Virtue Because of this it comes close to
  predicting rather than postdicting. The two
  parameters estimated in the preceding model fit
  were the latency scale for retrieval and the
  visual encoding time.
• Virtue The use of imaging data (and there were
  lots more) provides strong converging evidence
  for the the assumptions about intermediate
  stages.
• Virtue We took the same model and used it to
  predict the learning of adult subjects of an
  artificial algebra -- including brain imaging
  data.
• Doubt However, what is being learned is a rather
  algorithmic skill that affords more-or-less
  complete up-front instruction.

12
The Ugly Experiment (And not so
Algorithmic) Analogous to Instruction in
Foersters Text
Pyramids There is a notation for writing
repeated addition where each term added is one
less than the previous For instance, 5 4 3
is written as 5 2 Since 5 4 3 12 we
would evaluate 52 as 12 and write 52 12 The
parts of 5 2 are given names 5 is the base
and reflects the number you start with 2 is the
height and reflects the number of items you add
to the base 5 2 is called a pyramid

13
The Students Tasks Evaluate The Following

• 5 3
• 5 4 3 2 14
• 10 4
• 10 9 8 7 6 40
• 8 1
• 8 7 15
• 3 4
• 3 2 1 0 -1 5
• 5 7
• 5 4 3 2 1 0 -1 -2 12
• 0 4
• 0 -1 -2 -3 -4 -10
• 13 0
• 13
• 1000 2000
• 1000 1 0 -1 -1000 0

14
More Student Problems Write Pyramid Expressions

• 6 5 4 3
• 63
• 9 8 7
• 92
• 1 0 (-1) (-2)
• 13
• x (x 1) (x 2) (x 3) (x 4)
• x4
• 20 (20 1) . (20 11)
• 2011
• 15 (15 1) . (15 x)
• 15x
• z (z 1) . (z y)
• zy

15
More Student Problems Find the Height x

• 6 x 15
• 6 5 4 15 -- x 2
• 10 x 55
• 10 9 8 7 6 5 4 3 2 1
  55 -- x 10
• 912 x 912
• x 0
• 19. 3 x -9
• 3 2 1 0 -1 -2 -3 -4 -5
  -9 -- x 8
• 100 x -101
• 100 1 0 -1 -100 -101
  -101 -- x 201

16
Final Student Problems Find the Base x

• x 2 15
• guess and check 765 18 654 15
  or
• x (x - 1) (x -2) 15 -- 3x - 3 15
  -- x 6
• x 1 15
• x 8
• x 4 35
• x 9
• x 6 35
• x 8
• x 6 0
• ?????
• x 3
• x 6 -7
• x 2

17
Pyramid Problems

• Designed on analogy to instruction on
  exponentials in the now-classic Foerster Algebra
  1 text (9th grade)
• Problems with controlling for prior knowledge
  with exponentials and testing with college
  students.
• 10002000 analogous to Foesters 11000
• Foerster has expression-writing problems using
  ellipses.
• Foester has solve-for-exponent problems analogous
  to our solve-for-height.
• There are no solve-for-base problems in Foerster
  analogous to our solve-for-base problems.
• Ran 6 algebra (A students) and 6 CMU students
  with protocols.
• Little difference so presenting average data.
• Took the exact model that we ran on algebra, gave
  it instruction, and ran it on this, and I am now
  in state of repair.
• The ugly is what is really interesting in this
  story -- what did not work in original model and
  how it might be repaired.

18
(No Transcript)
19
Six Things that the Old ACT-R Model Could Not Do

• Recognize when it is undertaking a task of too
  great a magnitude (a number of hours to solve
  10002000 with a calculator).
• Formulate a new generalization (negatives cancel
  positives).
• Recognize and represent abstract patterns
  (Ellipsis).
• Rework its knowledge to enable a different use.
• Spontaneously recognize the relevance of past
  problems.
• See the equivalence of different problem
  descriptions (pyramid and successive integer).

20
Comments and Plan Forward

• We could program a model that would do this all.
• Perhaps such a model would have fit the data.
• But the whole point of this enterprise is to
  avoid programming in solutions -- better ugly
  than bad.
• The old model just read the instructions and
  followed them in a general way and so avoided
  solution programming.
• It became ugly partly because the instructions
  did not cover solving for base or height (but
  only part of the problem).
• To deal with this we gave ACT-R Soar-like
  capacities to respond to impasses and reason
  about how it extend the instructions to new
  situation (will not discuss).
• I will discuss rather the metacognitive issues
  that are problematical for Soar as well (John
  Laird).
• Focus on 10002000, 100x -101, and x6 0.

21
Comments on 10002000

• Despite the fact that students had calculators no
  one even began to calculate the sum.
• Students averaged about half of their time in
  unproductive attempts before they tried a method
  that work.
• An unproductive path tried by many was to find an
  analogy to what they knew about factorial.
• Five students reasoned about simpler problems
  like 24.
• Others reasoned more abstractly.
• A number of students confirmed the answer (0) by
  a second method before giving it as their final
  answer.

22
First Metacognitive Challenge Posed by 10002000
Recognizing there is a Problem
No student begins to attempt to use their current
method. (a) This is despite the fact it
has worked well and been practiced for the last 7
problems. (b) Minimally this requires
that students recognize at the outset that they
will have to make 2000 additions and that this is
unreasonable. (c) However, neither of
these determinations are required by the
procedure that has been working well. (d)
So it requires some thinking in parallel with
following what is becoming a well-oiled
algorithm.
23
Second Metacognitive Challenge posed by
10002000 Accumulating Abstract Knowledge

• The students who reason abstractly retrieve two
  generalizations made while solving earlier
  problems
• There are positives and negatives that cancel
• The last number in the sum is the difference of
  the base and the height.
• But neither of these inferences was required by
  the algorithm they were successfully following.
• Again they had to be engaged in parallel thought
  not required by the task at hand.

24
100x -101 Recognizing that there is a Problem

• Takes place in the presence of a working (but
  invented) procedure of just adding numbers
  starting with the base until the answer is
  achieved and then counting how many added.
• Again no student even begins calculating the
  answer.
• This is despite the fact that they have
  successfully calculated the answers for 4
  problems including 912x 912 and 3x -9.
• Thus, they are clearly not responding to a
  superficial feature of the problem but
  nonetheless they are putting together (in
  parallel) the needed inferences sufficiently
  rapidly that they stop any attempt to go down
  their standard path.
• This is what my current model cannot do.

25
Comments on Solving 100x -101

• While most give evidence of being reminded of
  10002000, they are not faster -- median of 127
  sec for 100x -101 versus 72 seconds for
  10002000.
• One difficulty is that the reduce-to-simpler-probl
  em method is difficult because it is not so
  obvious what an analogous simpler problem would
  be -- no student poses something like 2x -3
  and tries to generalize.
• The second difficulty is that lacking a height
  they do not have an obvious way to state the last
  term in an abstract formula.
• Still, all successful solvers do wind up
  abstractly characterizing the addition and many
  use the fact that 100200 0 as a stepping
  stone.

26
Summing up A Partial Cure for Ugliness
Metacognitive Parallel Reflection

• While proceeding forth with a normal procedure
  students are reflecting on what they are doing.
• This allows them to recognize a too-difficult
  problem before they start the addition required
  by their normal procedure.
• This allows them to make generalizations about
  their problem solution not relevant to solving
  the current problem but which can be used later
  -- like the fact that the last term in the
  addition is base minus height.
• This also allows 7 of the 11 students to quickly
  (12 sec) solve x6 0, while the other 4 who
  apply guess-and-test take 78 sec -- current model
  predicts 82 sec.

27
Architectural Issue Raised by Parallel Reflection

• Does parallel reflection compete for resources
  with regular problem solving?
• A Soar or EPIC unlimited parallel model would say
  no.
• We could propose in ACT-R a separate
  metacognitive module like vision which runs
  without demands on the central system -- a little
  homunculus.
• However, I am willing to bet that it does compete
  and must interleave itself in open moments
  afforded by things like the perceptual-motor gaps
  and other pauses in processing.
• Therefore, I predict breakdown of such reflective
  successes as we take away from participants the
  spare time by dual tasking.

28
Thank You Questions or Reflections?