Cognitive Mastery Learning

1
Cognitive Mastery Learning
Albert Corbett HCI Institute Corbett_at_cmu.edu
Co-Director, with Ken Koedinger and John
Anderson, Pittsburgh Advanced Cognitive Tutor
(PACT) Center
2
Outline

• Cognitive Mastery Challenge
• Cognitive Mastery Impact
• Knowledge Tracing Assumptions
• Validating Knowledge Tracing
• Evaluating Cognitive Mastery
• Scaffolding Understanding

3
Cognitive Mastery Learning

• Model Tracing
• Yield 1 SD effect size in learning
• Twice as good as typical human tutor
• Half as good as best human tutors
• Cognitive Mastery Learning Challenges
• Can we go beyond improved average outcomes and
  help each student reach mastery of material?
• Will more and more problem solving practice
  achieve this?
• Can we individualize the curriculum so each
  student gets just the practice he or she needs?

4
Definitions

• Model Tracing Interpret students behavior by
  comparison with the student model provide
  feedback and advice
• Knowledge Tracing Infer students knowledge of
  the rules in the cognitive model, based on
  performance.
• Cognitive Mastery Individualize the curriculum,
  based on knowledge tracing, to enable student to
  master all cognitive rules in the lesson.

5
Cognitive Mastery

• Goal Individualize the problem sequence to
  provide each student just the problem-solving
  opportunities needed to master the material.
• Sufficient and Efficient Problem Solving
  Experience
• Two Questions
• What are the units of knowledge?
• How do we decide if the student has mastered
  them?

6
Cognitive Mastery

• Goal Individualize the problem sequence to
  provide each student just the problem-solving
  opportunities needed to master the material.
• Sufficient and Efficient Problem Solving
  Experience
• Two Questions
• What are the units of knowledge? Rules in
  Cognitive Model
• How do we decide if the student has mastered
  them? Bayesian Inference from Performance

7
Skill Meter Displays Learning State Probabilities
8
Outline

• Cognitive Mastery Challenge
• Cognitive Mastery Impact
• Knowledge Tracing Assumptions
• Validating Knowledge Tracing
• Evaluating Cognitive Mastery
• Scaffolding Understanding

9
Cognitive Mastery EffectivenessSummary

• Impact
• Accurately predict
• student test performance
• Increase test performance
• by about a letter grade

10
Mastery Learning Efficiency
40 more problems 14 more time 25 greater
accuracy 570 increase in mastery
Effect Size Cognitive Mastery vs. Fixed
Curriculum 0.65 Corbett, A.T. (2001). Cognitive
computer tutors Solving the two-sigma problem.
User Modeling Proceedings of the Eighth
International Conference, UM 2001, 137-147.
11
Outline

• Cognitive Mastery Challenge
• Cognitive Mastery Impact
• Knowledge Tracing Assumptions
• Validating Knowledge Tracing
• Evaluating Cognitive Mastery
• Scaffolding Understanding

12
Knowledge Tracing

• Goal For each cognitive rule, infer the
  students knowledge state from performance.
• Suppose a student has six opportunities to apply
  a rule and emits the following sequence of
  correct (1) and incorrect (0) responses. What
  can we conclude about whether the student has
  learned the rule?

1 0 1 0 1 1
13
Learning Assumptions

• Two-state learning model
• Each rule is either learned or unlearned
• In problem-solving a rule can make the transition
  from the learned to the unlearned state at each
  opportunity to apply the rule
• No forgetting - Rules do not make the transition
  from the learned state back to the unlearned
  state

14
Performance Assumptions

• If the rule is in the learned state there is some
  chance the student will slip and make a mistake.
• If the rule is in the unlearned state there is
  some chance the student will guess correctly.

15
Four Parameter Model
p(T)
Unlearned State
Learned State
p(L0)
p(G)
1-p(S)
correct
correct
Two Learning Parameters p(L0) Probability the
rule is in the learned state at time 0 (prior to
the first opportunity to apply the rule in
problem solving). p(T) Probability the rule will
make the transition from the unlearned state to
the learned state at each opportunity to apply
the rule Two Performance Parameters p(G) Probabili
ty the student will guess correctly if the rule
is in the unlearned state p(S) Probability the
student will slip (make a mistake) if the rule is
in the learned state
16
Knowledge Tracing

• Goal For each cognitive rule, infer the
  students knowledge state from performance.
• Suppose a student has six opportunities to apply
  a rule and emits the following sequence of
  correct (1) and incorrect (0) responses. What
  can we conclude about whether the student has
  learned the rule?
• Iterative process We update the estimate of the
  probability the student knows a rule at each
  opportunity to apply the rule.

1 0 1 0 1 1
17
Inferring Learning State

• Following each opportunity to apply a rule,
  the new probability estimate that the rule has
  been learned, p(Ln), is the sum of two
  probabilities
• (1) A revised estimate of the probability that
  the rule was already in the learned state, given
  the new evidence (correct or incorrect response)
• (2) the probability the student learned the rule
  at this opportunity if the student did not
  already know the rule.

p(Ln) p(Ln-1Rn) (1 -
p(Ln-1Rn))p(T)
Bayes Theorem
18
Knowledge Tracing Simulation
Assume P(L0) 0.3 p(T) 0.4 p(G)
.2 p(S) .1 Student performance 1 0 1 0
11 Attempt p(knew) p(Ln-1 ) p(learn now)
p(know) p(Ln ) n correct (Bayes
Theorem) (1 - p(Ln-1) T)
p(knew)p(learn) 1 1 0.66
0.14 0.80 2 0 0.33
0.27 0.60 3 1 0.87
0.05 0.92 4 0 0.59
0.16 0.76 5 1 0.93
0.03 0.96 6 1 0.99
0.00 0.99
19
Outline

• Cognitive Mastery Challenge
• Cognitive Mastery Impact
• Knowledge Tracing Assumptions
• Validating Knowledge Tracing
• Evaluating Cognitive Mastery
• Scaffolding Understanding

20
Predicting Student Performance

• Knowledge tracing estimates learning - an
  unobservable construct
• To validate knowledge tracing we need to generate
  performance predictions
• The probability a student will fire a production
  correctly at the nth opportunity in problem
  solving is

p(Cn) p(Ln-1)(1-p(S)) p(Un-1) p(G)
21
Student Modeling Validation(Lisp Programming
Tutor)
22
Performance Predictions

• To predict tutor performance
• Refine cognitive model rule set
• For each rule generate best-fitting estimates for
  the two learning parameters and two performance
  parameters
• To predict individual student quiz performance
• Estimate individual difference weights

23
Empirical Learning Curves Evidence of
Overgeneralization (Corbett Anderson,1995)
24
Empirical Learning Curves Evidence of
Overgeneralization
25
Student Modeling Validation(Lisp Programming
Tutor)
26
Predicting Quiz Performance

• Estimate individual Difference Weights
• Predict probability of completing exercise
  correctly (rather than probability of completing
  each step correctly)

Pp(Cgs) The probability a student will complete
an exercise correctly is the product of the
probabilities the student will complete each
successive goal correctly
27
Knowledge Tracing APT Lisp TutorPredicting
Student Test Performance

• Fit
• Actual 0.81
• Expected 0.86
• R 0.66
• MAE 0.10

Corbett, A. Anderson, J. (1995). Knowledge
tracing Modeling the acquisition of procedural
knowledge. User Modeling and User-Adapted
Interaction, 4, 253-278.
28
Knowledge Tracing Genetics TutorPredicting Test
Performance
27 Students 14 Cognitive
Rules FITS Actual
0.87 Actual 0.86
Expected 0.83 Expected 0.82
r 0.82
r 0.83 MAE
0.08 MAE 0.04
29
Outline

• Cognitive Mastery Challenge
• Cognitive Mastery Impact
• Knowledge Tracing Assumptions
• Validating Knowledge Tracing
• Evaluating Cognitive Mastery
• Scaffolding Understanding

30
Cognitive Mastery

• Students continue doing problems in each
  curriculum section until the probability that
  each cognitive rule is in the learned state
  exceed a criterion value (typically 0.95)

31
Mastery Learning Efficiency
60 more problems 5 more time 93 greater
accuracy 130 increase in mastery
32
Knowledge TracingPredicting Test Scores(Corbett
Anderson, 1995)

• Fit
• Actual 0.81
• Expected 0.86
• R 0.66
• MAE 0.10

33
Systematic Performance Overestimates

• Knowledge Tracing overestimates test performance
  by about 10
• Hypotheses
• Systematic overestimate in tutor also
• Motivation Shift
• Forgetting
• Transfer
• Corbett, A.T. and Bhatnagar, A. (1997). Student
  modeling in the ACT Programming Tutor Adjusting
  a procedural learning model with declarative
  knowledge. User Modeling Proceedings of the
  Sixth International Conference, UM 97, 242-254.

34
Outline

• Cognitive Mastery Challenge
• Cognitive Mastery Impact
• Knowledge Tracing Assumptions
• Validating Knowledge Tracing
• Evaluating Cognitive Mastery
• Scaffolding Understanding

35
Modeling the Problem GivensThe Challenge
append, cons, list
36
Modeling the Problem GivensAugmented Feedback
Write a function call that takes the arguments
(hut) ((shed) (tent)) and returns ((hut) (shed)
(tent))
(cons
37
Modeling the Problem GivensAugmented Feedback
Write a function call that takes the arguments
(hut) ((shed) (tent)) and returns ((hut) (shed)
(tent))
(cons
38
Flying Parentheses
Suppose we want to call a function with the
arguments (hut) ((shed)
(tent)) To construct the result ((hut) (shed)
(tent))
39
Flying Parentheses
Suppose we want to call a function with the
arguments (hut) (
(shed) (tent)) To construct the result ((hut)
(shed) (tent))
40
Flying Parentheses
Suppose we want to call a function with the
arguments (
(shed) (tent)) To construct the result
((hut) (shed) (tent))
(hut)
41
Flying Parentheses
Suppose we want to call a function with the
arguments
((hut)(shed) (tent)) To construct the result
((hut) (shed) (tent))
42
Results
Tutor Performance
Test Performance
43
Results Test Performance
Corbett, A.T. and Trask, H, (2000). Instructional
interventions in computer-based tutoring Dtial
impact on learning time and accuracy. Proceedings
of ACH CHI2000 Conference on Human Factors in
Computing Systems, 97-104.
44
Summary Cognitive Mastery

• Cognitive Mastery substantially increases student
  achievement scores
• Ultimately there are diminishing returns to doing
  more of the same type of problem
• Clarifying domain representations can foster
  understanding and further increase achievement
  scores
• Plan scaffolding can help students organize
  familiar knowledge more quickly, but may not
  yield higher asymptotic performance.