Interactive Knowledge Capture for ProblemSolving Systems

1
Interactive Knowledge Capturefor
Problem-Solving Systems
Jim Blythe Yolanda Gil Jihie Kim www.isi.edu/ikc
ap
2
Outline

• Main elements of our approach
• Experiences and lessons from earlier work
  monolithic systems
• New work on the Calo project
• open systems

3
Acquiring procedure knowledge from users

• Each user can have unique requirements of a
  system
• Their requirements will change over time, perhaps
  frequently
• In travel planning, may be different for each
  trip
• Users need to be able to modify procedure
  representations in intelligent systems to address
  their needs
• Complements example-based learning

4
Example adding procedure knowledge to a travel
planning tool

• An assessment tool makes judgments about travel
  itineraries
• e.g., the airline should be United or American
• e.g., the hotel should be within walking
  distance, unless I am renting a car
• Need to add procedure knowledge to tell the
  system to make a new kind of judgment
• the hotel can cost up to 20 more than the
  government per diem rate for the city.
• or supporting procedures
• to estimate driving time, divide the distance
  by 55

5
Expects Support for procedure KAKey
Technologies

• Where does the user start?
• An acquisition wizard guides the user to start
  the KA process through a dialog, based on
  problem-solving methods.
• KA takes many steps users will be lost
• The acquisition wizard manages the process from
  end to end.
• Users dont know the computer language.
• An English-based procedure editor
• Users modify the English paraphrase of the formal
  representation.
• A search-based expression composer
• Suggests valid reformulations of user sentences.
• How do users ensure all the needed information is
  added?
• An interdependency analyzer understands which
  pieces of knowledge are used to solve a problem.

6
Main elements of our approach

• Meta-reasoning
• About how knowledge fragments are combined to
  solve a goal
• About missing information, effects of changes to
  KB
• About the dialog and context
• Putting knowledge in the users terms
• Generate text descriptions to hide the syntax
• Browse-and-replace interface for procedures
• Search-based reformulations of free text
• Test knowledge on real examples

7
Outline

• Main elements of our approach
• Experiences and lessons from earlier work
  monolithic systems using Expect
• New work on the Calo project
• open systems

8
Acquisition wizard

• Dialog with user to start the process.
• Some questions use menus or text input.
• Others use the English editor to refine
  procedural knowledge

9
English-based Procedure editor
NL description of method

• (multiply
• (obj (look-up
• (obj fsa-per-diem-hotel-rate)
• (for (r-city ?hotel))))
• (by 1.2))

Alternatives for selected text fragment
10
Test new knowledge immediately
Softools, developed under the DARPA AcT program
Each element is defined and checked using
constraints
11
Global Procedure Analysis Test
interdependencies, both rule-rule and rule-data
Problem-Solving Methods
Domain Ontology
...
(evaluate (obj coa) (wrt logistics))
r-location
port
INTERDEPENDENCIES
seaport
r-berths
airport
r-pols
...
... (r-location port)
r-piers
inland waterway seaport
maritime seaport
r-storage-area
... (r-berths seaport)
U new port Havana S I need to know if it is an
airport or a seaport U seaport S I need to know
the location and the berths
Interdependencies guide Knowledge Acquisition
12
When combined, the tools use information from
each other
Application
Acquisition wizard
Interdependency analyzer
Acquisition analyzer
Procedure editor
Expression composer
Instance editor
Relation/concept editor
13
The acquisition wizard

• Guides the user through the initial steps of
  adding new knowledge.
• Structures the knowledge to be added using
  default procedural knowledge.
• Questions are generated from a problem-solving
  theory.

14
Problem-solving theory for plan evaluation

• A hierarchy of generic types of plan judgments
  with default procedural knowledge.

judgment
global judgment
local judgment
bounds check
extensional check
upper bound
lower bound
positive
negative
completeness judgment
hotel cost judgment
Warn if the value is too large?
DEFINED check that the value is less than the
maximum value
ASK USER compute a maximum value for each object
15
Benefits of integration The acquisition wizard
and the method editor

• Each component receives information from the
  other that helps the user
• The wizard provides to the editor
• An initial version of the method, with the
  correct capability
• An expectation of the result type of the method
• The editor provides to the wizard
• A more detailed method result type
• Used to help classify the new task in the
  ontology

16
bounds check
upper bound
lower bound
Warn if the value is too large?
17
The expression composer

• Assists users in formalizing informal statements
  by suggesting composite expressions using terms
  the system understands.
• Uses an ontology of terms and relations, and
  synonyms derived from WordNet with extensions.
• Makes breadth-first forward search, matching
  keywords

18
Expression composer example

• User types max staging post landing
• Tool suggests find the maximum of the landing
  distance available of the runways of the forward
  staging post.

Function call
find
object
of
reformulation
landing-distance-available
maximum
Typed variable
runways
combining queries
?forward-staging-post
Information element
19
Integrated with the method editor

• Constructs compound terms in KB that include user
  terms and have desired type
• Anytime breadth-first search through space of
  terms

20
Outline

• Main elements of our approach
• Experiences and lessons from earlier work
  monolithic systems
• New work on the Calo project
• open systems

21
Tailor interactive task acquisition forthe Calo
office assistant

• A personal assistant that manages everyday tasks.
• Learns from experience, interacts with the user
  naturally through several channels.
• Large DARPA-funded project run by SRI.

New Facts Goals
2
(source commanders-vision new)
7
(ACHIEVE (satisfy Incoming-INs))
Process Library

• Spark is Calos task manager.
• (A PRS-based system developed by Myers et al.)

1
Process Execution
Facts Goals
6
External World
5
8
Goal2
Goal3
ACT8
ACT3
3
sleeping
sleeping
Intention Graph
Fact1
ACT2
4
normal
22
Approach and challenges in acquiring knowledge
within CALO

• Carry key elements forward
• Explicit reasoning about knowledge dependencies
  and acquisition context
• Expressing the procedures in the users terms
• Earlier work used Expect, a monolithic system
  using custom problem-solving system, language,
  methods, ontology,..
• Now use Sparks performance element and
  procedures, ontologies provided by several
  groups.

23
Example purchasing a laptop

• Spark manages the workflow as the user purchases
  a laptop.
• Choose a model, find bids, get authorization,
  track purchase
• During the process, Spark finds that the order
  cannot be completed, because a manager who must
  authorize the purchase is not available.
• User should be able to tell the system You dont
  need authorization when the cost is less than
  2000

24
Initial solution

• Make direct analysis of the Spark procedures
• Encapsulates a model of Sparks behaviour
• Requires additions to language, e.g. types,
  purpose of subtasks
• Combine the procedure analysis with the
  expression composer to help interpret user
  sentences.
• Dialog is currently implicit
• Currently provide follow-up questions based on
  analysis.

25
Response toYou dont need authorization when
the cost is less than 2000
Automatic generation of text from Spark procedure
definitions
Expression composer suggests valid condition
User can explore modifications based on different
assumptions
26
Tailor summary and future work

• Currently allow users to modify existing
  procedures, using global analysis and expression
  composer.
• Next
• Support defining new tasks, integrate with advice
  for Spark.
• Model dialog build templates for supporting
  theory, working with Allen/Ferguson on dialog
  model.

27
Summary

• Our work allows users to add procedure knowledge
  to both custom and pre-existing intelligent
  systems.
• Combination of global analysis and term
  reformulation can allow users and the system to
  reach shared understanding.
• Interactive knowledge capture complements
  example-based learning to allow intelligent
  systems to adapt.
• Evaluation strategies
• Ablation user studies with fixed and free tasks.
• Good results with Expect Constable, still to do
  with Tailor.