Group 3 Discussion

1
Group 3 Discussion

• Aravind Joshi, Donna Gates, Sandra Williams,
  Xuchen Yao, Sarah Luger
• Nov. 4-6 2011AAAI Symposium on Question Generation

2
Task 1 Evaluating QG from sentences/paragraphs

• What are the aspects important for evaluation?
• Agreed on using automatic judgment (BLEU style
  score as in MT)
• Use Amazon Mechanical Turk to generate as many
  questions as possible for each single sentence in
  the test corpus
• Say paying 0.10 to ask 1 turker to generate 2
  questions each for 5 sentences (10 questions in
  all). Assign 5 turkers to each sentence. We
  estimate generating 1000 questions for 100
  sentences with 100.

3
Task 1 Evaluating QG from sentences/paragraphs

• Should the two subtasks remain as they are or
  should we focus on one, or replace both, or
  modify any of them?
• for evaluation purposes, they should be
  separated.
• We have three opinions
• keep them as they are
• break the paragraph task into 3 subtasks
  general/medium/specific subtasks (or just
  general/specific).
• introduce topics in the input

4
Task 1 Evaluating QG from sentences/paragraphs

• Did you participate in QGSTEC in 2010? If not,
  what would encourage you to participate?
• we should advertise more on the paragraph task
• finding sponsor from companies (e.g. quora.com,
  suggested by Sarah)
• through UPenn summarization group (to promote
  their tools, Aravind promised to ask them)

5
Design a reliable annotation scheme/process

• If using Mturkers to judge the generated
  questions
• use a lot of them.
• If using Mturkers to generate questions, to
  prevent duplicate questions
• do 2-3 rounds, each round showing turkers all
  unique questions generated from the previous
  round.
• or get something out from the distribution of all
  kinds of turker questions (e.g., the kind of
  questions humans generate the most, etc.)

6
pretending we are the turkers

• Two sources of fuel are particularly important to
  ensure a healthy and efficiently functioning
  brain - oxygen and water.

7
Two sources of fuel are particularly important to
ensure a healthy and efficiently functioning
brain - oxygen and water.

• 2 What are the two sources of fuel to ?
• What two sources of fuel are important for brain
  health?
• What are the most important fuels for a
  functioning brain?
• Name two types of fuel for a healthy and
  efficient brain.
• Why does the brain require oxygen and water?
• How are oxygen and water important for your
  brain?
• How many sources of fuel are to ensure ?
• Is oxygen an important source ?
• Oxygen and what other fuel are essential for a
  healthy brain?

8
Round 2

• Oxygen and water are essential for functionality
  of what body organ?
• A functioning and healthy brain needs what fuels
  as input?
• How many sources were identified as fuel of
  brain?
• Water makes brain healthy and what else does too?
• Can the brain work without oxygen and water?
• Why does the brain need oxygen and water for its
  health?
• What fuels your brain?
• What would happen if your brain had less oxygen?
• To ensure a healthy brain, what do you need?
• Does brain need water and oxygen to function?

9
two rounds so far, our questions tell us

• In the 1st round
• the important pieces of information of the
  sentence (as least the WH questions)
• even though the syntax/structure has been
  changed, the key info is kept.
• The questions show the understanding of the
  turkers for the input
• In the 2nd round
• we start to pick out less important info
• we start to think about entities and relations
• we start to think of synonyms, hypernyms,
  paraphrasing.
• we start to think about causal relations,
  inference etc.

10
Other things we can do

• present paragraphs instead of sentences to turkers