Question Answering

1
Question Answering

• Changed User Assumptions
• User may not be one who built the KB
• User may not have used knowledge entry tools
• The Process of Question-Answering
• I Posing the question (problem setup)
• User can load the exam document as source
• II Problem-solving by AURA
• Decide which KB models to use
• Reasoning with those models
• III Answer presentation and explanation

2
I Posing the Question Approach

• Computer-processable language (CPL)
• sweet spot between formal logic and full NLP

Unrestricted natural language
Formal language
CPL
A ball falls from a cliff
?x?y B(x)? R(x,y)?C(y)
Consider the following possible situation in
which a ball first
too hard for the user
too hard for the computer

• Interactive validation/correction technologies
• Multi-modal input
• CPL, diagrams, equations interface with a single
  model

3
Understanding EnglishTechnical Challenges

• Resolving ambiguity (many sources!)
• Could the chemical be baking soda?
• Swimming pool water
• Dealing with incomplete/erroneous text
• The block starts from rest starts what?
• How long does it take for the object to hit the
  ground? instantaneous?
• Mapping to the KBs ontology
• A train collides with a wagon ?
  Collide-And-Stick
• An object falls from a cliff ? Free-Fall
• Reflecting back the systems understanding

4
Usability Challenges

• How easy will it be for me to learn CPL?
• How will I know if AURA has understood me
  correctly?
• If AURA misunderstands, will it be obvious to me
  how to fix things?

5
Aura
Enter Question
CPLgt
A ball is thrown with a horizontal velocity of
20 m/s from a cliff that is 125m above ground.
6
Aura
Enter Question
CPLgt
A ball is thrown with a horizontal velocity of
20 m/s from a cliff that is 125m above ground.

• Does this look okay?
• ASSUMPTIONS
• ? The throw occurs at the top of the cliff.
• ? The thrower is a person.
• FACTS
• ? A person throws a ball.
• ? A person throws a horizontal velocity.
• The horizontal velocity is 125 m above ground.
• OBJECTS
• EVENTS

Wrong!
Reinterpret
Rewriting Suggestions
Okay
7
Aura
Enter Question
CPLgt
A ball is thrown with a horizontal velocity of
20 m/s from a cliff that is 125m above ground.

• Does this look okay?
• ASSUMPTIONS
• ? The throw occurs at the top of the cliff.
• ? The thrower is a person.
• FACTS
• ? A person throws a ball.
• ? A person throws a horizontal velocity.
• The horizontal velocity is 125 m above ground.
• OBJECTS
• EVENTS

Reinterpret
Rewriting Suggestions
Okay
8
Aura
Enter Question
CPLgt
A ball is thrown from a cliff.
9
Aura
Enter Question
A ball is thrown from a cliff.
CPLgt
Does this look okay? ASSUMPTIONS ? The throw
occurs at the base of the cliff. ? The thrower
is a person. FACTS ? A person throws a ball. ?
The location of the person is the base of the
cliff. ? The location of the throw is the base of
the cliff. OBJECTS EVENTS
Bad asssumption!
Reinterpret
Rewriting Suggestions
Okay
10
Aura
Enter Question
A ball is thrown from a cliff.
CPLgt
Does this look okay? ASSUMPTIONS ? The throw
occurs at the base of the cliff. ? The thrower
is a person. FACTS ? A person throws a ball. ?
The location of the person is the base of the
cliff. ? The location of the throw is the base of
the cliff. OBJECTS EVENTS
Uncheck box and click Reinterpret
Reinterpret
Rewriting Suggestions
Okay
11
Aura
Enter Question
A ball is thrown from a cliff.
CPLgt
Does this look okay? ASSUMPTIONS ? The throw
occurs at the top of the cliff. ? The thrower
is a person. FACTS ? A person throws a ball. ?
The location of the person is the top of the
cliff. ? The location of the throw is the top of
the cliff. OBJECTS EVENTS
Reinterpret
Rewriting Suggestions
Okay
(clicks Okay)
12
Aura
Enter Question
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt
13
Aura
Enter Question
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground?
14
Aura
Enter Question
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground?
Ask!
Cancel
15
Technical Assessment

• Technology for
• Core language processing
• Incomplete/erroneous input
• Reflecting systems understanding
• Currently
• Base software is in use in other projects
• To do Extend and customize for AURA
• Customize to scientific language, questions
• Connect with AURAs core ontology

16
Usability Assessment

• Can users
• reformulate questions in CPL?
• recognize systems misunderstanding?
• easily correct the misunderstanding?
• Based on
• usability studies (5 hrs write, read studies)
• previous work with controlled languages
• Boeing simplified English
• To do
• Extensive user studies with live system

17
Multi-Element Questions
Comment This and the next slide hidden not
part of the main final presentation but
available as backup

• Multi-element questions
• multiple choice
• free-form with multiple parts
• User can invoke templates for these (see next
  slide)
• No significant new technical/usability issues
• use of templates avoids new language processing
  challenges

18
Aura
Enter Question
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground? (5
choices)
A.
3 seconds
B.
5 seconds
C.
6 seconds
D.
12 seconds
E.
25 seconds

• Does this look okay?
• CHOICES
• A. The duration of the fall is 3 seconds.
• ? B. The duration of the fall is 5 seconds.
• ? C. The duration of the fall is 6 seconds.
• ? D. The duration of the fall is 12 seconds.
• ? E. The duration of the fall is 25 seconds.

Reinterpret
Rewriting Suggestions
Okay
19
Question Formulation with Diagrams

• User can
• create diagrams containing objects, variables,
  and values
• (same as for knowledge formulation)
• load and fill in existing diagrams
• refer to objects in diagrams using English
• Diagrams can be
• Component diagrams made from pre-defined
  objects, connections, values
• Metric diagrams involving actual geometry and
  measurement.

20
Technical Usability Challenges(same as for
knowledge formation)

• Technical Challenges
• Spatial reasoning is very difficult
• Seamless integration with other knowledge
• Conveying the limits of diagrams to the user
• Usability Challenges
• How do I know when I should use a diagram?
• Will I be able to easily sketch what I want?
• How do I know if my drawing is adequate?

21
Aura
Domain
Kinematics
Objects
Enter Question
CPLgt
22
Aura
Domain
Kinematics
Objects
Enter Question
CPLgt
User has entered values for two known variables
and has shown the desired unknown using "?"
23
Aura
Domain
Kinematics
Objects
Enter Question
CPLgt
How long does it take the object to fall?
User has entered part of the problem using a
diagram and has entered a CPL question that
refers to the diagram.
24
Technical Usability Assessment

• Technical (as for knowledge formulation)
• Component diagrams
• Metric diagrams
• Spatial reasoning
• Usability
• Diagram entry
• Based on initial user trials (simulated system)
• To do More extensive tests with live system

25
Text Equations
Comment This and the next slide hidden not
part of the main final presentation but
available as backup

• Equations
• Use a keyword in text to indicate an equation
• Enter equation using equation calculator (same
  as for knowledge formulation)
• Technical
• Main challenge interfacing calculator to rest
  of KB
• Usability
• In initial read studies, users uniformly liked it

26
Aura
Enter Question
CPLgt
Is EQN a Lewis acid-base reaction?