JustinTime Interactive Question Answering

1
Just-in-Time Interactive Question Answering

• Sanda Harabagiu, PI
• John Lehmann
• John Williams
• Jeremy Bensley
• Paul Aarseth
• Language Computer Corporation

2
Overview

• Project Objectives
• Research Strategies
• System Design
• Accomplishments and Challenges
• Results
• Lessons Learned

3
Research Objective

• Address the interactive aspect of QA systems
• Design and implement a dialog shell that can
  be used with any QA system
• Extension of work in factual QA with dialog
  capabilities already tested in call center systems

4
Novelty

• Usage of just-in-time citations of other
  analysts similar QA sessions
• Just-in-time citations are provided by
  Just-In-Time Information Seeking Agents (JITISA)
• JITISA intelligent agents that proactively
  search and retrieve information that might be
  useful without requiring any action from the
  professional analyst
• Non-intrusive software agents that monitor the
  dialog contexts generated when different agents
  interrogate QA systems

5
Tasks Proposed
6
Real-life Dialogs

• Start with a complex question
• What is the situation in the Middle East?
• I am interested in Indias Prithvi ballistic
  missile program.
• Complex questions cannot be handled by current QA
  systems.
• Complex questions need to be decomposed in
  simpler questions
• What are the latest developments in the program?
• How many versions of the Prithvi are there?
• How many Prithvi missiles does India currently
  have?

7
Two Strategies

• User-centric
• Several users need to find the same information
• They use the QA system and ask lots of questions,
  taking many different dialog paths
• Analyze how they expressed their intentions by
  generating a riverflow of templates
• Information/knowledge-centric
• Generate a database of Frequently-Asked
  Questions
• Build a similarity metric
• Analyze only the dialog motivators, the
  decomposition into simpler questions and the
  generation of answers from various sources

8
Dialog Strategies

• Core Assumptions
• Sufficient documents relating to any question are
  available
• There is enough interest in the topics such that
  other users might have already posed many
  pertinent questions
• News source collections
• Documents from major newspapers with dates
• Web source collections
• Multiple specific queries used per domain, saving
  top 500 documents each time

9
Question Answer DataBase (QADB)

• Because domain is closed, we may be able to
  predict questions and collect answers
• How well can we cover the range of possible
  questions?
• Process
• 1. Split up topics between developers
• 2. Generate question and answer records
• 3. Rotate topics among developers

10
Generating a QADB
Complex Question Scenario Decomposition
Use docs to get more questions
Create Question
11
Pilot 1 QADB Population

• For 10 domains, collected 334 question records,
  each with answers from multiple sources
• Perform retrieval of answers by computing
  question similarity based on concepts

12
Pilot 2 QADB Population

• For 6 domains, collected 350 question records,
  each with answers from multiple sources
• Improved question similarity metric
• Domain-specific concepts
• Inverse frequency concept weighting

13
System Architecture
14
Decomposition Challenges

• As already shown, complex questions must be
  broken down in order that QA systems be able to
  handle them
• How can rational natural language questions be
  formed from a topic question?
• Of those, how can the interesting and meaningful
  questions be chosen?

15
Decomposition Data

• AQUAINT Spring 03 CNS topics used
• CNS-generated 140 decomposition questions from 12
  domains
• LCC researchers generated additional questions
  for 6 domains boosting total to 342 questions
• Decompositions averaged 60 questions for selected
  domains
• Combined with 350 questions from QADB

16
Analysis of Decompositions

• Human-generated decompositions analyzed
• Special focus on how sets of topic terms
  corresponded to interesting decompositions
• Topic How has Russia assisted North Koreas
  nuclear weapons program (NWP)?
• Specifically, what constitutes a NWP?
• What sort of assistance would be useful to a NWP?
• What assistance has Russia given to NWPs?
• What assistance has Russia given to North Korea
  for its NWP?

17
Question Relations

• QADB Questions manually organized into
  hierarchies to observe relations
• Questions related on several different levels
• Term level
• Information Target level
• Motivational level

18
Question Relevance

• Questions analyzed to see why they were found
  interesting and chosen
• Interesting defined a decomposed questions
  potential to supply information which leads to a
  greater understanding of the information goal
• Reoccuring features led to the identification of
  3 classes of questions

19
Interesting Question Classes

• What type of attack will terrorists next use
  against the US?
• State Exploration What is the state of USs
  homeland defenses?
• Instance Collection Which terrorist groups might
  want to attack the US?
• Evidence Discovery What is the level of
  technology that these terrorists currently
  possess and are developing?

20
Pilot 1 Results

• Assessors in graded our dialog based on several
  performance measures for each domain
• Scale 1-7 with 7 representing completely
  satisfied

21
Pilot 2 Observations

• Analysts asked much better formed questions
• Better directions?
• Influence of graphical interface?
• Analysts examined source documents
• Credibility?
• Explanatory context?
• Additional relevant information?
• Analysts liked system-proposed related questions
• Used this feature frequently and commented on it

22
Pilot 2 Results

• Assessors graded our dialog based on several
  performance measures for each domain
• Significantly less wizard for our system in
  Pilot 2 compared to 1
• Scale 1-7 with 7 representing completely
  satisfied

23
Lessons Learned

• User-centric is paramount
• Knowledge-centric is very important
• Automatically create interesting and relevant
  questions
• Need to find and validate answers
• Context is very important and complex
• More than reference resolution
• Logical relations between sequences of questions
  in a scenario that depend on how much is known
  about that scenario