How the PSLC Members will Develop a Theory of Robust Learning

1
How the PSLC Members will Develop a Theory of
Robust Learning

• PSLC Directors Ken Koedinger and Kurt VanLehn

2
PSLC Vision

• Problem Not enough rigorous, real-world,
  generalizable results
• Solution LearnLab
• Technology social process resources
• New paradigm for learning experimentation
  theory
• Intellectual merit
• New understanding of robust learning
• Test of lab-based learning principles in real
  settings
• Use-inspired hypotheses to test in the lab
• Broad impact
• Wide use of new experimental paradigm
• More effective courses
• New evidence-based education

3
LearnLab Addresses Current Limitations of
Learning Experiments

• Rigorous studies with real content in real
  classrooms with real students
• Collect analyze fine-grain data over long
  durations
• Measure robust learning
• Long-term retention, transfer, accelerated future
  learning

4
The 5 Rs of LearnLab

• More Rigorous and Realistic Research Results on
  Robust Learning!

5
Outline for todays talk

• Motivation Why a theory?
• Meta What kind of theory?
• Initial theory
• Micro
• Macro
• Future work

6
Interacting Pieces of PSLC
Principles of Robust Learning
7
SLC-wide motivations

• Questionable accretion
• Compare textbooks contents
• Why no problem sets in textbooks?
• Compare to other applied sciences
• Medicine
• Agriculture

8
Outline for todays talk

• Motivation Why a theory?
• Meta What kind of theory?
• Initial theory
• Micro
• Macro
• Future work

Next
9
Feasible types of theory

• Newells
• A single, computational model
• An accurate simulation of a student
• Simons
• Concepts e.g., working memory
• Combine them to form explanations

10
Outline for todays talk

• Motivation Why a theory?
• Meta What kind of theory?
• Initial theory
• Micro
• Macro
• Future work

Next
11
Key Conceptual Premises Vocabulary

• Learning is partially decomposable
• Knowledge component a piece of acquired
  knowledge
• Concept, principle, production rule, schema,
  reasoning process, meta-cognition, ...
• Knowledge event an interval in time or a piece
  of instructional material where a knowledge
  component might be learned/used
• Knowledge component learning
• Construction Component is (re-)derived from
  perceptions and existing knowledge
• Feature validity Response-relevant or deep
  features acquired, irrelevant/shallow features
  ignored
• Strength Accumulated importance, ease of
  retrieval, efficiency of processing

12
Illustrations of knowledge components
False massless is missing

• The magnitudes of the tension forces exerted by
  the ends of a taut string are equal.
• A Chinese character, its pronunciation and its
  meaning
• The number of kisses exchanged when French people
  greet is meaningful
• If Im feeling a bit uncertain, attempt the step
  but if Im feeling quite uncertain, ask for hint.

Skills
Overly general
Meta
13
Illustrations of knowledge events

• Read definition
• Study example
• Use in proof
• Explain to another student
• Make error, get tutored, correct
• Start to write def, look up in text, finish

14
Basic knowledge events

• Description Verbal rule, explanation provided
• Self-explanation Verbal rule, explanation
  generated
• Example Instance of stimulus-response,
  situation-action, features-label provided
• Practice Stimulus/situation/ features provided,
  student generates response/action/ label

Passive Active
Explicit Description Self-explanation
Implicit Example Practice
15
How Research Clusters Produce Learning Outcomes

• Instructional treatments vary knowledge events to
  produce changes in construction, feature
  validity, or strength of knowledge components

Feature Validity
Construction
Strength
Knowledge Events Descriptions, Examples,
Self-Explanation, Practice
Refinement of Features
Dialogue
Co-Training
Fluency
16
Knowledge events over time
While studying example, tries to self-explain
fails looks in text succeeds
While solving problem, looks up example recalls
explanation maps it to problem
Recalls explanation slips corrects
Knowledge event duration
Solves without slips
Solves without slips
17
Illustrations of Construction

• Natural language understanding
• A surface always exerts a force that is
  perpendicular (normal, in mathematics) to the
  surface. It is called the normal force on the
  object.
• Analogy
• When an object presses on spring, the spring
  pushes back. A surface is like a very stiff
  spring. When on object pushes on it, it pushes
  back. This is called the normal force on the
  object.
• Induction Here are some normal forces

18
Illustration of feature validity
At rest 5 kg block 1 kg block Frictionless
What is big blocks velocity after falling 2 m?

• Surface features match training problems for
  Newtons second law
• Pulley, string, blocks
• Deep features match training problems for
  Conservation of Energy
• Two times, no friction forces, no applied forces,
  displacement, velocity

19
Micro learning processes results
Process Result
Construction Existance
Refinement of features Feature validity
Strengthening Strength
20
Why is a macro level necessary?

• Micro level describes learning processes, but not
  why they occur.
• What kinds of situations, instruction,
  backgrounds, etc. tend to elicit micro learning,
  and hence robust learning?
• Need a taxonomy

21
Outline for todays talk

• Motivation Why a theory?
• Meta What kind of theory?
• Initial theory
• Micro
• Macro
• Future work

Next
22
Original Research Clusters Pathways to Robust
Learning
Robust Learning
Foundational Skills
Sense-Making
Co-Training, Multiples
Refinement of Features
Dialogue
Fluency
23
Differentiating Processes Outcomes
Robust Learning
Sense-Making
Foundational Skills
Co-Training, Multiples
Refinement of Features
Dialogue
Fluency
24
Elaborating Missing Processes
Robust Learning
Outcomes Knowledge, reasoning learning
processes
Foundational Skills
Sense-Making
Instructional Processes
Multiple inputs, representations, strategies
Feedback, example variability, authenticity
Tutorial dialogue, peer collaboration
Schedules, part training
25
Instructional, learning, reasoning processes
Reasoning Learning Processes
Learning Processes
26
Illustrations from PSLC Studies

• Katz
• Liu Perfetti
• Pavlik

InstructionLearning ProcessOutcome
27
Katz Post-practice reflection

• Instruction After solving a physics problem,
  student answers reflection questions, e.g.,
• What is the basic approach?
• What if the block were moving up instead of down?
• Learning processes
• Construction, e.g., basic approach is
  Conservation
• Refinement, e.g., initial block direction is
  irrelevant
• Outcomes
• Existance of basic approach knowledge
  components
• Feature validity of many knowledge components
• Manipulation Dialogue vs. text remediation
• Dialogue increases frequency of construction
  refinement

28
Liu Perfetti Chinese character learning

• Task Correctly pronounce Chinese characters
• Instructional treatment Multiple inputs
• Show video of mouth movements in addition to just
  audio of sound of the character
• Other possible inputs outputs still of lips,
  pin yeng (written version of pronunciation),
  meaning
• Learning process Co-training
• Feature refinement
• Different media make different features salient
• Outcome
• Knowledge components with better feature validity

29
Pavlik Scheduling practice

• Task given French word, provide English word
• Instruction Vary training schedule
• Use ACT-R to compute sequence that optimizes
  costs benefits of training task difficulty
• Spacing between practice trials increases as
  student gets better at word pair
• Learning process
• Strengthening through practice
• Outcome
• Knowledge components with higher strength

30
Pathways lead to measures

• Robust learning pathways yield measurable
  outcomes through well-acquired knowledge
  components (including both domain meta
  knowledge components)

Pathways
Sense-making
Foundational Skills
Long-term retention Transfer Future Learning
Measures
31
Pathways lead to measures

• Robust learning pathways yield measurable
  outcomes through well-acquired knowledge
  components (including both domain meta
  knowledge components)

Pathways
Sense-making Rederivation
Foundational Skills Strengthening
Long-term retention Transfer Future Learning
Measures
32
Pathways lead to measures

• Robust learning pathways yield measurable
  outcomes through well-acquired knowledge
  components (including both domain meta
  knowledge components)

Pathways
Sense-making Rederivation Adaptation
Foundational Skills Strengthening Deep feature perception
Long-term retention Transfer Future Learning
Measures
33
Pathways lead to measures

• Robust learning pathways yield measurable
  outcomes through well-acquired knowledge
  components (including both domain meta
  knowledge components)

Pathways
Sense-making Rederivation Adaptation Self-supervised learning
Foundational Skills Strengthening Deep feature perception Cognitive headroom
Long-term retention Transfer Future Learning
Measures
34
Encourage Active Declarative Processing Through
Self-Explanation
Aleven, V. Koedinger, K. R. (2002). An
effective metacognitive strategy Learning by
doing and explaining with a computer-based
Cognitive Tutor. Cognitive Science, 26(2)
35
Problem Shallow knowledge low feature
validity

• Example of a knowledge component with low feature
  validity
• Looks-equal production rule
• If the goal is to find angle A and it looks
  equal to angle B and angle B is D degreesThen
  conclude that angle A is D degrees

36
Example of Shallow Reasoning
37
Explanation Condition
38
Problem Solving Condition
39
Assessing transfer Not Enough Info item
40
Assessing transfer Incorrect over-generalization
41
SE Study 2 Results
42
Mapping to PSLC terms

• What robust learning measures?
• What robust learning pathways?
• Sense-making or foundational skills
• What knowledge components changed from pre to
  post?
• What learning reasoning processes lead to
  measures?

43
Mapping to PSLC terms

• Robust learning measureTransfer
• What robust learning pathway(s)?
• Sense-making
• What knowledge components changed from pre to
  post?
• Higher feature validity on geometric reasoning
  components
• (More accessible of declarative knowledge?)
• What learning reasoning processes lead to
  measures?
• Deep feature perception adaptation?

44
Instructional, learning, reasoning processes
Reasoning Learning Processes
Learning Processes
45
Tutoring Help-Seeking

• Goal Foster long-term learner independence
• Model of ideal learning help-seeking behaviors

• Tutor this model
• Improve robust learning
• Long-term retention
• Transfer
• Accelerated future learning

46
Measuring Future Learning
47
Outline for todays talk

• Motivation Why a theory?
• Meta What kind of theory?
• Initial theory
• Micro
• Macro
• Future work

Next
48
Theory development via

• Discussions at research cluster meetings
• Reporting research
• Web pages
• Advisory board visit (Dec. 13 14, 2005)
• Annual progress report (winter, 2006)
• Site visit (spring, 2006)
• Planning
• Strategic plan (summer, 2006)
• Suggest calls for research projects
• Individuals clusters engage in all of the above

49
Evaluation of the PSLC theory development effort

• Members understand the terms/theory
• may or may not ascribe to it
• C-STPS interviews
• PSLC documents become more integrated
• C-STPS text analysis
• Members papers in open literature use PSLC
  terms/theory
• C-STPS text analysis
• Others use PSLC terms
• C-STPS interviews and text analysis

50
END
51
Limitations/missing pieces

• Other key learning-related processes
• Utility, attention, motivation, meta-cognition,
  perceptual chunking, declarative-procedural
  distinction
• Computational, cognitive, social/motiv
  considerations
• Science has formulas, where are ours?
• generalization gradient, power law of practice,
  spacing, transfer gradient, cognitive IRT
• Machine learning theorems
• The Psych of HCI type book
• Model Human Learner instead of Model Human
  Processor
• (see photo of white board EC meeting notes)

52
Illustration Self-explanation

• Given examples to study then problems to solve,
  self-explaining (SE) the examples improves
  problem solving (PS) immediately, after delay, on
  far-transfer problems. Probably increases
  learning on new tasks as well.

Sense-making Rederiving principles during SE encourages rederiving during PS Adapting principles during SE to apply encourages similar adapatation during PS Self-supervision Deciding to SE and finding that it makes PS easy (easier?) encourages SE later
Foundational Skills Using principles to SE strengthens them SE adds deep features to the examples Cognitive headroom Easy recall of parts facilitates SE of whole
Measures Retention Transfer Acceleration
53
PSLC Study Examples
InstructionLearning ProcessOutcome

• Katz Post-reflection dialog in physics leads
• to constructions that increase knowledge
  component existance, and
• to feature refinements that increase feature
  validity
• Liu Perfetti Audio lip video of chinese
  leads to co-training that increases feature
  validity
• Aleven Contiguous diagram elements labels
  leads to better refinement increase feature
  validity
• Pavlik Expanded spacing of practice of French
  vocabulary optimizes strengthening to strengthen

54
Aleven Contiguous labels enhance geometry tutor

• Instruction Givens answers in either
• a table beneath the diagram, or
• boxes in the diagram
• Learning process Refinement
• Placing data near visual referents increases
  frequency of refinement
• Outcome
• Feature validity

55
Some slides from 05 site visit
56
Pre-PSLC Pittsburgh Theory

• A mess of concepts
• Self-explanation
• Procedure vs. declarative memory
• Help-seeking behavior
• Competition (of 1st language with 2nd)
• Co-training
• Can generate several possible explanations/predict
  ions, but not evaluate their probability

57
How to improve the theory

• Talking together about our work
• Increase shared beliefs/concepts
• Build on each others results
• Grounding our terminology
• Theory-laden tools, e.g.,
• Data Shop
• Theory-laden tutoring systems
• Computational models, e.g.,
• Simulated students

58
How concrete and general will our theory become?

• Models of memory, attention,
• ACTR, Epic, etc.
• Models of skill acquisition
• Cascade, HS, Iccarus, etc.
• SimStudent project?
• Strategic plans framework
• Includes social/motivational
• Newells UTC book

59
Multi-directional payoffs
Cognitive psychology
Neuro-cognitive studies
Computational models of cognition
Intelligent tutoring systems
Social meta-cognitive studies
Machine learning