A Day in the Life

1

• A Day in the Life
• Knowledge Formulation
• Question Answering
• Question Formulation
• Problem Setup
• Text only
• Text plus diagrams
• Text plus equations
• Entering question components
• Free form
• Multiple choice
• Problem Solving
• Method selection
• Reasoning
• Answer Presentation

Technical
Usability
Scenario
Scenario
Scenario
Technical
Scenario
Scenario
Scenario
Technical
Usability
Scenario
Usability
Scenario
Technical
Technical
Usability
Scenario
2
Innovative Technical Ideas

• Computer-processable language (CPL)
• sweet spot between formal logic and full NLP

Unrestricted natural language
Formal language
CPL
A ball falls from a cliff
?x?y B(x)? R(x,y)?C(y)
Consider the following possible situation in
which a ball first
too hard for the user
too hard for the computer

• Interactive validation/correction technologies
• Multi-modal input
• CPL, diagrams, equations interface with a single
  model

3
Technical Challenges

• Core language technologies
• want to cover many different linguistic phenomena
• Ambiguity resolution
• many sources ? many possible interpretations of
  input
• Semantic mapping
• transforming from linguistic concepts to KB
  concepts
• many-to-many mapping (more ambiguity)
• Reflection
• Replaying back to the user the systems
  understanding

4
Usability Challenges Status

• Can the user grasp and adequately use CPL
  (restricted English), when processing of the
  original text fails? Can he/she easily see how to
  express his/her conceptual model of the question
  in CPL?
• Approach develop and test guidelines for writing
  in CPL
• Status
• Kraka studies with two SMEs rated this as very
  promising (read only study)
• Iterative development and testing of guidelines
  with different SMEs (5 1 hr sessions, pen paper
  exercise) resulted in good re-expressions of the
  original questions in CPL (read/write study)
• Can the user grasp AURAs understanding of
  his/her input text, and recognize when AURA
  misunderstood?
• Approach fact-by-fact paraphrasing of AURAs
  model back to the SME
• Status Kraka studies with two SMEs rated this as
  very promising (read only study)
• Can the user easily realize what corrective
  action(s) to take if the system misundertood?
• Approach (i) on-line rephrasing assistance (ii)
  user can explicitly highlight errors and
  reprocess
• Status Not evaluated (really needs a live
  system)

5
Usability More Conclusions

• The SMEs preferred the CPL plus diagrams over
  pure textual versions
• The SMEs liked the feedback from the system
  telling them what the systems current model of
  the problem looks like.
• The SMEs suggested that different domains and
  topics within may call for different tools.
• Write test pending week of 12 June

6
Question AnsweringGuide to the Scenarios

• Question Formulation
• Problem setup
• Text only
• Text plus diagrams
• Text plus graphics
• Entering question components
• Free form
• Multiple choice
• Problem-Solving
• Method selection
• Reasoning
• Answer Presentation
• Ontological (descriptive) knowledge
• Rules
• Equations
• Graphical/Spatial
• Tables

7
(QF text only)
8
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
9
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
10
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
A ball is thrown with a horizontal velocity of
20 m/s from a cliff that is 125m above ground.
11
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
A ball is thrown with a horizontal velocity of
20 m/s from a cliff that is 125m above ground.

• Does this look okay?
• ASSUMPTIONS
• ? The throw occurs at the top of the cliff.
• ? The thrower is a person.
• FACTS
• ? A person throws a ball.
• ? A person throws a horizontal velocity.
• The horizontal velocity is 125 m above ground.
• OBJECTS
• EVENTS

Wrong!
Reinterpret
Rewriting Suggestions
Okay
12
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
A ball is thrown with a horizontal velocity of
20 m/s from a cliff that is 125m above ground.

• Does this look okay?
• ASSUMPTIONS
• ? The throw occurs at the top of the cliff.
• ? The thrower is a person.
• FACTS
• ? A person throws a ball.
• ? A person throws a horizontal velocity.
• The horizontal velocity is 125 m above ground.
• OBJECTS
• EVENTS

Reinterpret
Rewriting Suggestions
Okay
13
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
A ball is thrown from a cliff.
14
Aura
Document
Scenario
Diagram
Answer
Enter Question
A ball is thrown from a cliff.
CPLgt
Does this look okay? ASSUMPTIONS ? The throw
occurs at the base of the cliff. ? The thrower
is a person. FACTS ? A person throws a ball. ?
The location of the person is the base of the
cliff. ? The location of the throw is the base of
the cliff. OBJECTS EVENTS
Bad asssumption!
Reinterpret
Rewriting Suggestions
Okay
15
Aura
Document
Scenario
Diagram
Answer
Enter Question
A ball is thrown from a cliff.
CPLgt
Does this look okay? ASSUMPTIONS ? The throw
occurs at the base of the cliff. ? The thrower
is a person. FACTS ? A person throws a ball. ?
The location of the person is the base of the
cliff. ? The location of the throw is the base of
the cliff. OBJECTS EVENTS
Uncheck box and click Reinterpret
Reinterpret
Rewriting Suggestions
Okay
16
Aura
Document
Scenario
Diagram
Answer
Enter Question
A ball is thrown from a cliff.
CPLgt
Does this look okay? ASSUMPTIONS ? The throw
occurs at the top of the cliff. ? The thrower
is a person. FACTS ? A person throws a ball. ?
The location of the person is the top of the
cliff. ? The location of the throw is the top of
the cliff. OBJECTS EVENTS
Reinterpret
Rewriting Suggestions
Okay
(clicks Okay)
17
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) EVENTS Thro
w (propel with force) FACTS A person throws a
ball. The location of the person is the top of
the cliff. The location of the throw is the top
of the cliff.
A ball is thrown from a cliff.
CPLgt
CPLgt
18
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) EVENTS Thro
w (propel with force) FACTS A person throws a
ball. The location of the person is the top of
the cliff. The location of the throw is the top
of the cliff.
A ball is thrown from a cliff.
CPLgt
CPLgt
The height of the cliff is 125 m.
19
(QF free form question)
20
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground?
21
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground?
22
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a
duration. THING TO FIND The duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground?
Ask!
Cancel
23
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a
duration. THING TO FIND The duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground?
Answer
ltshort answer summarygt
ltAnswer tab on LHS (?) shows introspectable
explanationgt
24
Text Diagrams Technical Ideas

• Main points
• User can select palette of canonical objects
• Can drag, place, resize objects in canvas
• Can name them, connect them, attach properties
• User can name properties, and link them with
  existing property names (slots) in the KB
• System summarizes its understanding by listing
  objects and facts recorded so far
• Objects are in shared workspace, so user can also
  refer to them using CPL, including
• assigning properties
• asking questions

25
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
26
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
(select Diagram tab on LHS)
27
Aura
Document
Scenario
Diagram
Answer
Enter Question
Domain
Kinematics
CPLgt
Objects
User selects Kinematics and gets a menu of
diagrams.
28
Aura
Document
Scenario
Diagram
Answer
Enter Question
Domain
Kinematics
CPLgt
Objects
User selects a cliff diagram from the menu.
(Picks this)
29
Aura
Document
Scenario
Diagram
Answer
Enter Question
Domain
Kinematics
CPLgt
Objects
Diagram has active buttons for variables. User
can type in values and connect them to variables,
or connect them to variables in other objects.
30
Aura
Document
Scenario
Diagram
Answer
Enter Question
Domain
Kinematics
CPLgt
Objects
User has entered values for two known variables
and has shown the desired unknown using "?"
This diagram alone is sufficient for solving this
problem.
The diagram produces a KM model. Both text and
diagram can be used to produce a unified model.
31
QF Text equations
32
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
33
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
34
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
Is EQN a Lewis acid-base reaction?
35
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
Is EQN a Lewis acid-base reaction?
36
Aura
Document
Scenario
Diagram
Answer
Enter Question
CPLgt
Is EQN a Lewis acid-base reaction?
37
Aura
Document
Scenario
Diagram
Answer
Enter Question
Is Al(OH)3(s) OH-(aq) ? Al(OH)4-(aq) a Lewis
acid-base reaction?
CPLgt
Ask!
Cancel
38
QF multiple choice
39
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground?
40
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground? (5
choices)
41
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a
duration. THING TO FIND The duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground? (5
choices)
A.
B.
C.
D.
E.
42
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a
duration. THING TO FIND The duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground? (5
choices)
A.
3 seconds
B.
5 seconds
C.
6 seconds
D.
12 seconds
E.
25 seconds
43
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a
duration. THING TO FIND The duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground? (5
choices)
A.
3 seconds
B.
5 seconds
C.
6 seconds
D.
12 seconds
E.
25 seconds

• Does this look okay?
• CHOICES
• A. The duration of the fall is 3 seconds.
• ? B. The duration of the fall is 5 seconds.
• ? C. The duration of the fall is 6 seconds.
• ? D. The duration of the fall is 12 seconds.
• ? E. The duration of the fall is 25 seconds.

Reinterpret
Rewriting Suggestions
Okay
44
Aura
Document
Scenario
Diagram
Answer
Enter Question
SCENARIO SO FAR OBJECTS Ball (round
object) Cliff (steep high face of rock) Top of
Cliff (location) Person (human being) Ground
(flat surface of the earth) EVENTS Throw (propel
with force) Fall (descend freely under influence
of gravity) FACTS A person throws a ball. The
location of the person is the top of the
cliff. The location of the throw is the top of
the cliff. The cliff is 125 m high. The ball has
a horizontal velocity. The horizontal velocity is
20 m/s. Air resistance 0. The ball falls. The
fall starts at the top of the cliff. The fall
ends at the ground. The fall takes a
duration. THING TO FIND The duration.
CPLgt A ball is thrown from a cliff. CPLgt The
height of the cliff is 125 m. CPLgt The balls
horizontal velocity is 20 m/s. CPLgt There is no
air resistance. CPLgt How long does it take for
the ball to fall to the ground? (5
choices)
A. 3 seconds B. 5 seconds C. 6 seconds D. 12
seconds E. 25 seconds
Ask!
Cancel
45
Method Selection 1. Automatic Classification
automatic classification (recognition)
Object velocity 0
ObjectAtRest
automatic classification (recognition)
ObjectAtRest object-of Fall
path Air resistance
0
ObjectFallsFromRestNoAir
Hence the original Object inherits all the rules
from the ObjectFallsFromRestNoAir model
46
Method Selection 2. Heuristic Matching

• The Semantic Matcher matches the expanded
  question representation against Physics Models
  (authored by KF-SMEs)

ObjectFallsFromRestNoAir object (a Object) orig
in (a Place) destinat (a Place with (height
((a Height-Value with (less-than (
)))))) duration sqrt (2(the distance of Self)
/ g) distance (the height of (the origin of
Self)) - (the height of (the destinat of Self))
47
Problem Solving

• The reasoner backchains from the holy grail
  (question goal) through the new instantiation of
  the SME-authored model

ObjectFallsFromRestNoAir object Object1 origin
Cliff2 destinat TheGround duration sqrt
(2(the distance of Self) / g) distance (the
height of (the origin of Self)) - (the height
of (the destinat of Self))
48
Problem Solving

• The reasoner backchains from the holy grail
  (question goal) through the new instantiation of
  the SME-authored model

ObjectFallsFromRestNoAir object Object1 origin
Cliff2 destinat TheGround duration sqrt
(2(the distance of Self) / g) distance 125m
- (the height of (the destinat of Self))
49
Problem Solving

• The reasoner backchains from the holy grail
  (question goal) through the new instantiation of
  the SME-authored model

ObjectFallsFromRestNoAir object Object1 origin
Cliff2 destinat TheGround duration sqrt
(2(the distance of Self) / g) distance 125m
- 0m
50
Problem Solving

• The reasoner backchains from the holy grail
  (question goal) through the new instantiation of
  the SME-authored model

ObjectFallsFromRestNoAir object Object1 origin
Cliff2 destinat TheGround duration sqrt
(2(125m) / g) distance 125m
51
Problem Solving

• The reasoner backchains from the holy grail
  (question goal) through the new instantiation of
  the SME-authored model

ObjectFallsFromRestNoAir object Object1 origin
Cliff2 destinat TheGround duration 5s dista
nce 125m
52
Problem Solving

• The reasoner backchains from the holy grail
  (question goal) through the new instantiation of
  the SME-authored model

ObjectFallsFromRestNoAir object Object1 origin
Cliff2 destinat TheGround duration 5s dista
nce 125m
53
Problem Setup
For more advanced problems, a Problem Setup phase
is required. This will use rules to analyze the
problem and infer needed information.

• Stylized English
• (what is the time of a fall with height
  125 m)
• (5.049 SECOND)
• User must know the physical principle fall and
  the names of variables time and height.
• Actual problem

An object is thrown with a horizontal velocity of
20 m/s from a cliff that is 125 m above level
ground. If air resistance is negligible, the
time that it takes the object to fall to the
ground from the cliff is most nearly
The problem as stated by the user must be
converted to a correct and complete formal model
sufficient for problem solving.
54
Problem Setup Tasks

• Problem Completion fill in missing information,
  make assumptions.
• Determine what Physical Principles apply
• Set up Geometric Model, Situations
• Solve the problem
• Generate explanations

55
Problem Completion

• Inference of problem scale subatomic,
  macroscopic, astronomical
• Inference of environment earth-surface -gt earth
  gravity
• Inference of values assume rope is massless
• Other inferences
• a car moves horizontally in a straight line
• trains are heavier than cars

56
Determine Physical Principles
A minimal set of physical principles that
correctly model the problem must be selected
(using rules), based on

• Kinds of objects. Some objects imply the models
  to use.
• Interactions of objects
• The question and desired quantity
• Code words "smooth", "elastic"
• Scale gravity is not considered in atomic-scale
  problems

These rules will be challenging for SMEs to write
and debug! Rules may conflict or interact badly.
This is a continuing problem for research
physics!
57
Collect Influences
Although most AP problems are simple, sometimes
it is necessary to write an equation that
collects all influences of a certain kind.

• Forces free-body diagram
• Force symmetry some problems rely on symmetry of
  forces without knowing what the forces are.
• Other sums torque, momentum, energy, heat
• Superposition of fields and waves
• Conservation laws convert differential equations
  into conservation between two snapshots

58
Geometric Model
Physics requires modeling actual geometry

• Determine the (smallest) kind of geometry
  required
• Direction of geometry (e.g. horizontal or
  vertical)
• Special locations earth surface, starting point
  may be set to 0, greatly simplifying algebra.
• Directions of influences, e.g. gravity is
  downward
• Electromagnetic directions special rules for
  fields, currents, motions, forces.

59
Situations
A Situation is a period of time during which the
physics remains the same. Physics problems may
have multiple situations that are linked by
conservation laws (e.g. position,
velocity). Free-response problems often have
linked situations

• A ball is accelerated upward by a spring (Prin.
  FR 1)
• It leaves the spring and continues moving upward
• A batter hits the ball
• The ball moves ballistically
• The ball hits the ground

60
Equation Solving
Several kinds of equation solving are required

• Solving an equation for a desired variable
• Evaluating an expression (with units) to get a
  value
• Determining how an expression varies with a
  variable
• Numeric variation if length doubles and
  resistivity is 1/5 ...
• Form of variation inverse-square
• Qualitative if x increases, does y increase or
  decrease?
• Unit conversion
• Unit comparison Volt/meter vs. Newton/Coulomb
• Unit classification Is Joule/sec a unit of power?

61
Explanation Generation
The strong analytical models produced by Problem
Setup will support clear and cogent explanations.

• Situations provide a conceptual formulation of
  the problem in terms of physical principles and a
  time line.
• Geometric models support diagram generation.
• Equation solving can be traced (if desired) to
  show how the answer was derived.
• Rules can be used to justify the solution method,
  and ultimately could support tutoring.