Knowledge Systems

1
(No Transcript)
2
Knowledge Systems

• Knowledge Systems use formal representations of
  knowledge to answer unanticipated questions with
  coherent explanations
• Knowledge System KB Q/A
  Explanation Generator Knowledge Acq.
  tools

3
Advances over Expert Systems

• Coverage of domain, not domain task
• Various modes of reasoning, well integrated
• Domain level explanation
• Rapid construction

U
Just how advanced are they?
4
Project Halo

• Long term build a Knowledge System encompassing
  much of the worlds scientific knowledge
• Short term assess current technologies
• Use a portion of the Advance Placement (AP)
  chemistry exam as a metric

Full support for Project Halo was provided by
Vulcan Inc, Seattle, WA
5
Challenges

• Systems must be robust in the face of widely
  varying, unanticipated questions.
• Explanations are as important as correctness.
• Hard-ball evaluation, aimed to expose weaknesses.
• New domain and short development time require
  using off-the-shelf KRR methods and systems.

It was not clear at the outset that these
challenges could be met.
6
Example Questions

• The spectator ions in the reaction of barium
  nitrate with sodium sulfate are what? (choices)
• Although nitric acid and phosphoric acid have
  very different properties as pure substances,
  their aqueous solutions possess many common
  properties. List some general properties of these
  solutions and explain their common behavior in
  terms of the species present.
• Explain why a solution of HClO4 and NaClO4 cannot
  act as a buffer solution.
• Sodium azide is used in air bags to rapidly
  produce gas to inflate the bag. The products of
  the decomposition reaction are what? (choices)

Questions were manually encoded in our formal
language
7
Because Questions Vary Widely

• we can not anticipate the questions, or even
  the type of questions, so a retrieval method
  wont do.
• A custom inference method wont do.
• The system must be capable of using its knowledge
  in unanticipated ways.

8
An Example Explanation

• What are the products of the given decomposition
  reaction?
• By definition, oxidation-reduction reactions
  occur when electrons are transferred from the
  atom that is oxidized to the atom that is
  reduced. We need to look for changes in the
  oxidation states of the elements in the reaction.
  
• In the reactants, the oxidation state(s) of the
  element Na is/are (1). In the product, the
  oxidation state(s) is/are (0)
• Therefore, the reaction causes a change in
  oxidation state.
• Therefore, this is an oxidation reduction
  reaction.
• By definition, a Binary Ionic-Compound
  Decomposition Reaction occurs when a binary ionic
  compound is heated.
• Therefore, this reaction is a Binary-Ionic
  Compound Decomposition reaction.
• In general, a Binary Ionic-Compound
  Decomposition Reaction converts a binary
  ionic-compound into basic elements.
• In this reaction, NaN3 reacts to produce Na and
  N2.
• The products of the decomposition reaction are
  (d) Sodium and nitrogen-g

9
Our KRR System

• KM KRL-like frame system with FOL semantics.
• able to represent
• classes, instances, prototypes
• defaults, fluents, constraints
• (hypothetical) situations
• actions (pre-, post-, and during- conditions)
• and reason about
• inheritance with exceptions
• constraints
• automatic classification (given a partial
  description of an instance, determine the classes
  to which it belongs)
• temporal projection (my car is where I left it)
• effects of actions
• KM answers questions by interleaving two types of
  inference
• Automatic classification
• Backward chaining

Details AAAI97
10
Structure of the Knowledge Base

• Two principal types of chemistry knowledge
• terms, e.g. binary ionic compound
• laws, e.g. problem-solving method for computing
  products of reactions of binary-ionic compounds
• Terms are encoded as definitions to enable
  automatic classification.
• Laws are encoded as rules to enable backward
  chaining.

Details KR04 (Barker, et.al.)
11
The Content of a Chemistry Law
Concentration of Solute Law Context a
mixture M such that volume(M) V liters
has-part(M) includes Chemical C such
that quantity(C) Q moles
concentration(C) Conc molar Input V, Q
Output Conc Method Conc ? Q/V
12
Knowledge Engineering Methodology

• Knowledge base built in 4 months
• Ontological engineering (4 person-months)
  designed representations, including structure of
  terms, laws, reactions, solutions, etc.
• Knowledge capture (6 person-months) consolidated
  70 textbook pages into 35 pages of terms and laws
• Knowledge encoding (15 person-months) coded in
  KM 500 types and relations, 150 chemistry laws
  and 65 terms. Compiled a large test suite which
  was run daily
• Explanation engineering (3 person-months)
  augmented the representations of terms and laws
  with templates

13
Results of Project Halo

• After 4 month development effort, the knowledge
  systems were sequestered and given a test
• 165 novel questions 50 multiple choice 115 free
  form response
• Questions translated from English to formal
  language by each team, then assessed for fidelity
  by Vulcan and team representatives

• Details AI Magazine (Winter 2004)
• www.projecthalo.com systems, Q/A, and
  analyses

14
Correctness

• Our systems correctness score corresponds to an
  AP score of 3 high enough for credit at UCSD,
  UIUC, and many other universities.
• Weve predicted scoring 85 after a 3 month
  follow-on project.

15
Explanation Quality
16
Error Analysis

• We analyzed every point lost. Most deductions
  were due to errors in domain modeling mistakes
  that domain experts would not make. (More later)
• Some errors were caused by technology problems.

Details KR04 (Friedland, et.al.)
17
Problems Due to KRR Technology

• Explanations too verbose e.g. passages repeated
  multiple times with only small variations
  graders expected a general statement that covered
  them all. Requires explanation planning
• Questions that require reasoning about our
  representations
• Calculate the pH of a particular substance.
  Explain why the result is unreasonable.
• Explain the difference between the subscript 3
  and the coefficient 3 in 3HNO3.
• Explain when and why its OK for a particular
  chemistry method to use an equation that only
  approximates the true answer.

18
Reasoning about Relevance
Hydrofluoric acid is a weak acid, Ka 6.8 x
10-4, and yet it is considered to be a very
reactive compound. For example, HF dissolves
glass. The major reason it is considered highly
reactive is (a) It is an acid. (b) It forms
H3O. (c) It dissociates. (d) It readily forms
very stable fluoride compounds. (e) It is a weak
electrolyte. All five statements are true. The
question requires that the system reason about
which of the multiple true statements is
most relevant to the claim.
19
Bottom Line

• Halo I was a rigorous evaluation of current
  Knowledge System technology.
• In general, the systems were more capable than
  Vulcan expected.
• The major hurdles to building a Knowledge System
  for science are errors (in domain modeling) and
  cost (10K/page).