Automating Comprehension Questions Lessons from a Reading Tutor

1
Automating Comprehension QuestionsLessons from a
Reading Tutor
Albert Corbett and Jack Mostow Project LISTEN
(www.cs.cmu.edu/listen) School of Computer
Science Carnegie Mellon University Pittsburgh, PA
15213, USA corbett, mostow_at_cmu.edu
Funded by The U.S. Department of Education,
Institute of Education Sciences Grant R305B070458
to Carnegie Mellon University
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
2
Outline

• Evaluation Principle
• Case Study Intelligent Reading Tutor
• Desiderata - Tutorial Functions
• (Student Outcome Related)
• Desiderata - Generation Process Related

Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
3
Evaluation Principle

• Principle Evaluate automatically generated
  questions based on goals of the
  system in which they occur.

Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
4
Evaluation Principle

• Principle Evaluate automatically generated
  questions based on goals of the
  system in which they occur.
• Case Study
• Project LISTEN Reading Tutor
• Grade 1-3 reading level (ages 6-9)
• Introduce 2 Types of Automatically Generated
  Reading Tutor Questions
• Use as Examples to Introduce Desiderata

Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
5
Desiderata Measuring Outcomes Comprehension

• Automatically Generated Cloze Questions, e.g.,
• And the very next day, the ____ had turned into a
  lovely flower.
• grain lily walnut prepare
• correlate reliably with a standard measure of
  reading
• comprehension, (r 0.85)
• (Mostow, Beck, Bey, Cuneo, Sison, Tobin
  Valeri, 2004)

Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
6
DesiderataScaffolding Outcomes Comprehension

• Automatically Generated Generic wh- Questions,
  e.g.,
• When does this take place?
• in the present in the future in the
  past
• it could happen in the past I cant tell
• scaffold reading comprehension
• (Beck, Mostow Bey, 2004)

Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
7
Desiderata Checklist
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
8
Desiderata Checklist
Beck (2005) Engagement
Tracing Employs response time
and accuracy for cloze
questions to detect student
(dis)engagement
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
9
Desiderata Checklist
If we can assess student
comprehension, we can evaluate
interventions designed to improve
comprehension
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
10
Desiderata Checklist
e.g., provide immediate
feedback on
student answers to
scaffold comprehension
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
11
Desiderata Checklist
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
12
Desiderata Checklist
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
13
Desiderata Checklist
The Principle Tutorial Goal
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
14
Desiderata Checklist
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
15
Desiderata Checklist
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
16
Desiderata Checklist
(for multiple choice questions)
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
17
Desiderata Checklist
Desiderata 2, 3 and 4 enable 5
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
18
Desiderata Checklist
Weve done some initial work on Cloze question
difficulty e.g., word difficulty, of
distractors that violate constraints
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
19
Desiderata Checklist
e.g., types of inferences in comprehension
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
20
Desiderata Checklist
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
21
Desiderata Checklist
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
22
Desiderata Checklist
Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
23
Summary

• Principle Evaluate automatically generated
  questions based on goals of the system
  in which they occur.

Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
24
Summary

• Principle Evaluate automatically generated
  questions based on goals of the system
  in which they occur.
• Cloze questions And the very next day, the ____
  had turned into a lovely flower.
• grain lily walnut
  prepare
• Satisfy multiple assessment and evaluation
  tutoring goals
• Satisfy automatic question generation, answer
  generation, distractor generation, and scoring
  desiderata

Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA
25
Summary

• Principle Evaluate automatically generated
  questions based on goals of the system
  in which they occur.
• Cloze questions And the very next day, the ____
  had turned into a lovely flower.
• grain lily walnut
  prepare
• Satisfy multiple assessment and evaluation
  tutoring desiderata
• Satisfy automatic question generation, answer
  generation, distractor generation, and scoring
  desiderata
• When does this take place?
• Wh- questions in the present in the
  future in the past
• it could happen in the past I cant
  tell
• Satisfy comprehension scaffolding desideratum
• Satisfy automatic question generation desideratum

Workshop on the Question Generation Shared Task
and Evaluation Challenge September 25-26,
National Science Foundation, Arlington, VA