QA on Anderson et al. 1995 Intro to CTAT

1
QA on Anderson et al. 1995Intro to CTAT

• CPI 494 598 Jan 27, 2009
• Kurt VanLehn

2
Initial Framework
NEXT

• Step loop
• User interface
• Interpreting student actions
• Suggesting good actions
• Feedback and hints
• Task selection
• Assessment
• Authoring and the software development
• Evaluations
• Dissemination

3
Describe the user interface of the Lisp Tutor

• What do students type? What are the lt..gt things?
  Steps?

4
Describe the user interface of the Geometry tutor

• What does student do? Steps?

5
Initial Framework

• Step loop
• User interface
• Interpreting student actions
• Suggesting good actions
• Feedback and hints
• Task selection
• Assessment
• Authoring and the software development
• Evaluations
• Dissemination

NEXT
6
For the Lisp tutor

• What counts as a correct step?
• How did the tutor determine if the students step
  is correct?
• Would the tutor accept both
• (plus X 5) and (plus 5 X)
• (times X 0.5) and (divide X 2)
• How does it know?
• Can students choose their own variable names?

7
For the geometry tutor

• What steps count as correct?
• True but useless propositions?
• Top down vs. bottom up?
• How does the tutor determine if the students
  step is correct?

8
Initial Framework

• Step loop
• User interface
• Interpreting student actions
• Suggesting good actions
• Feedback and hints
• Task selection
• Assessment
• Authoring and the software development
• Evaluations
• Dissemination

NEXT
9
For the Lisp tutor

• When the student asks for a hint, how does the
  tutor determine what step to hint?
• If the tutor allowed students to keep their
  incorrect steps on the screen, what step would
  the tutor suggest when asked?
• What if the tutor did not give immediate feedback
  and were asked for a hint?

10
For the Geometry tutor

• How does it decide what step to suggest?
• What does ACT theory say about this policy?
• Would it be better to recognize the students
  plan and help them along by suggesting its next
  step?

11
Initial Framework

• Step loop
• User interface
• Interpreting student actions
• Suggesting good actions
• Feedback and hints
• Task selection
• Assessment
• Authoring and the software development
• Evaluations
• Dissemination

NEXT
12
For the Lisp tutor

• Does the normal version of the tutor give
  immediate feedback, delayed?
• What other kinds of feedback policies did they
  try? What were the results?
• How was hinting usually done?
• Did the tutor ever ask a question when asked for
  a hint?

13
For the Geometry tutor

• Does the normal version of the tutor give
  immediate feedback, delayed?
• What other kinds of feedback policies did they
  try? What were the results?
• How was hinting usually done?
• Did the tutor ever ask a question when asked for
  a hint?

14
Initial Framework

• Step loop
• User interface
• Interpreting student actions
• Suggesting good actions
• Feedback and hints
• Assessment
• Task selection
• Authoring and the software development
• Evaluations
• Dissemination

NEXT
15
Lisp tutor Assessment

• What is assessed? Motivation? Knowledge? What
  kind of ?
• What granularity?
• Give examples of what is assessed?
• Whats the name of the technique?
• When students enter correct step, incorrect step
  or hint-request, what happens to the assessment?

16

• IF the goal is to get the second element of the
  list
• THEN code car and set as a subgoal to pass to car
  an argument that is the tail of the list

17
Geometry tutor Assessment

• What is assessed?
• Give examples
• How is the assessed stuff different from Lisps?

18

• IF the goal is to prove two triangles are
  congruent
• THEN set as a subgoal to prove that corresponding
  parts are congruent.

19
Initial Framework

• Step loop
• User interface
• Interpreting student actions
• Suggesting good actions
• Feedback and hints
• Assessment
• Task selection
• Authoring and the software development
• Evaluations
• Dissemination

NEXT
20
Lisp tutor Task selection

• What are the tasks to be selected from?
• Are they authored or generated dynamically?
• Who does the selection? Tutor or student?
• What is the procedure for selection?
• What information does the procedure need?
• How does this procedure fit with ACT theory?

21
Geometry tutor task selection

• What are the tasks to be selected from?
• Are they authored or generated dynamically?
• Who does the selection? Tutor or student?
• What is the procedure for selection?
• What information does the procedure need?
• How does this procedure fit with ACT theory

22
Initial Framework

• Step loop
• User interface
• Interpreting student actions
• Suggesting good actions
• Feedback and hints
• Assessment
• Task selection
• Authoring and the software development
• Evaluations
• Dissemination

NEXT
23
Lisp stutor Evaluations

• List the evaluations done
• For each, is it Formative? Summative? Hypothesis
  testing?
• What are the results of each?

24

• a fixed set of programming exercises either with
  the tutor or in a standard LISP environment.
  Students using the tutor completed the exercises
  30 faster and performed 43 better on a
  posttest.
• using the tutor completed the exercises 64
  faster and scored 30 higher on posttests than
  did students using a standard LISP environment.

25
What did this graph show?
26
What did this graph show?
27
Geometry tutor Evaluations

• List the evaluations done
• For each, is it Formative? Summative? Hypothesis
  testing?
• What are the results of each?

28

• In Peabody HS, one standard deviation
• An ethnography
• Again, but only got positive result with teacher.