J' William Murdock David W' Aha Leonard A' Breslow

1
Assessing Elaborated Hypotheses An Interpretive
Case-Based Reasoning Approach

• J. William MurdockDavid W. AhaLeonard A.
  Breslow
• Intelligent Decision Aids Group
• Navy Center for Applied Research in AI
• Naval Research Laboratory, Code 5515
• Washington, DC 20375
• lastname_at_aic.nrl.navy.mil
• http//www.aic.nrl.navy.mil/AHEAD

ICCBR03 Trondheim, Norway
2
AHEAD Analogical Hypothesis Elaboration for
Activity Detection
Objective Help analysts understand and trust
hypotheses about threats

• Models Functional (TMK) models encode how
  terrorist actions are performed and what they are
  intended to accomplish.
• Reasoning Interpretive case-based reasoning
  process
• Retrieve Given a hypothesis and a set of TMK
  models of terrorist activities, retrieve the
  best-matching model.
• Propose Given the matched model and the
  hypothesis evidence, generate the best-matching
  instantiation of that model.
• Justify Given this instantiation, analyze the
  evidence to create arguments (pro and con) that
  explain why that evidence matches (or mismatches)
  that instantiation.

3
Context Evidence Extraction Link Discovery
Intelligence Database
Evidence Extraction
Classified Reports
Other Analyst Tools
Hypotheses Arguments
News Reports
AHEAD
Hypotheses
Privacy/Security
Transactions
Link Detection
Pattern Learning
Threat Group (e.g., terrorists, organized crime)
People
4
AHEAD Functional Architecture
1. Retrieve FIRE Analogy Server
Link Discovery Tools
Hypothesis/Model Mapping
Case-Base (Models)
2. Propose Trace Extractor
Evidence Extraction Tools, Existing
Knowledge-Bases
Model Trace
Other Intelligence Analysis Software
3. Justify Argument Generator
GUI
5
AHEADs Interpretive CBRReasoning Methodology

• Retrieve Case
• A structured probe is extracted from the input
  hypothesis
• MAC/FAC (Forbus Gentner, CogSci 1991) uses the
  probe to retrieve an index for a TMK model
• Retrieved index is used to directly access the
  TMK model
• Propose Solution
• Hypothesis is reformatted as a planning input
  state
• Modified SHOP2 (Nau, et al., JAIR 2003) uses the
  model to generate an annotated plan that
  approximates the hypothesis
• Annotations based on queries to evidence
  databases (supporting evidence, contradictory
  evidence, no evidence)
• Justify Solution
• Where evidence matches the effects of actions in
  the plan, AHEAD produces an argument for the
  hypothesis
• Where evidence is missing or contradicts the
  plan, AHEAD produces an argument against the
  hypothesis

6
TMK Models Generalized Cases

• Task-Method-Knowledge (TMK) models are
  qualitative, functional representations of
  processes
• Task Encodes what a process does (arranged in
  hierarchies)
• Method Encodes how a process works (i.e.,
  decompose tasks)
• Knowledge Tasks and methods are described in
  terms of their effects on knowledge
• AHEADs TMK models are based on HTNs from SHOP2
  (U. of Maryland)
• with functional annotations that describe the
  purposes of actions
• with external queries to evidence databases
• SHOP2 finds an optimal match to the given
  hypothesis
• Larger scale may need near optimal matching

7
TMK Model Illustrative Example
Industry Takeover
Target Industry
makes
controls
Business Takeover
Business Selection
Mafia
groupMembers
makes
output
controls
provided
resists
makes
Intimidate
Surrender
MurderForHire
Target Employee
Target Business
ceo
makes
dead
8
AHEAD Functional Architecture Structured Argument
A hypothesis is a potential conclusion about what
happened in the world.
An argument against asserts that some part of the
relevant model was not performed.
An argument for is a asserts that some part of
the relevant model was performed.
Hypothesis
...
...
Argument For
Argument Against
Argument Against
Argument For
Other arguments against simply indicate a lack of
evidence.
Evidence
Evidence
Evidence
Some arguments against have explicit links to
evidence that supports them.
All arguments for are supported by references to
the original evidence.
Dashed lines in this figure indicate a connection
plus a qualitative confidence level.
9
Illustrative ExampleHypothesis Evidence
Evidence
Hypothesis
Hypothesis1
Yuri Volkov
Grigory Pridantsev
IndustryTakeover
Event 2
Murder
Subhypothesis1
Subhypothesis2
Late April, 1999
Event 1
MurderForHire
May 17, 1999
Industry1
Mafiya Group 1
Unidentified Gunman
Unidentified Gunman
Report 1
Report 2
10
AHEAD GUI
Interface is imbedded within a web browser.
Statement of the hypothesis
Hierarchical presentation of arguments for
Red and black icons are used to indicate
qualitative certainty levels.
Hierarchical presentation of arguments against
Hyperlinks to original sources for facts
A key allows users to quickly see what each icon
means.
Buttons allow navigation among different claims
within a given data set.
11
Pilot Study
Output Ratings (1-10) of the credibility of the
hypothesis and the users confidence.
Hypothesis Rating
1-10
Confidence Rating
Subjects
1-10
Both types of information are run through the
AHEAD GUI
On some tests a subject gets full arguments
(including hypotheses, and relevant pieces of
evidence).
GUI
On other tests the user gets only the hypothesis
and all of the evidence
Condition 1
Condition 2
12
Methodology Details

• Six hypotheses to each of six subjects
• Subjects were not experts in organized crime or
  intelligence analysis (future work)
• Arguments were produced by hand, following the
  process in the AHEAD functional architecture
• Users filled out responses for each hypothesis
• How valid is this hypothesis? (1-10)
• How confident are you of your hypothesis validity
  assessment? (1-10)
• How much time did you spend studying this
  hypothesis?
• Users were also asked general summary questions

13
Hypotheses

• Russian organized crime domain
• Based on evidence produced by a simulator
  developed for the EELD program (IET, 2002)
• Produces ground truth about some simulated
  events
• Randomly corrupts and omits portions of the
  information
• Resulting evidence is an input for the hypothesis
  generation systems and AHEAD
• Some hypotheses produced by SCOPE/iGEN (Eilbert,
  2002)
• Others were ground truth from the simulator

14
Metrics

• Elapsed time (want lower)
• Confidence (want higher)
• Error in Judgment (want lower)
• How far the user was from being correct
• judged validity - scaled absolute validity
• Error in Confidence (want lower)
• How well the users reported confidence
  corresponded to actual accuracy
• Loses points for answers that wrong answers with
  high confidence or right answers with low
  confidence
• (10 - confidence) - error in judgment

15
Results
16
Current/Future Work

• Automated evaluation of hypotheses
• Using arguments for/against as features by which
  to judge
• Multi-strategy evaluation
• Domain-specific heuristics
• Domain-independent heuristics
• Machine learning
• Case-based reasoning (within the larger
  interpretive CBR process)
• Useful for filtering hypotheses for the user
• Tighter integration with hypothesis generators
• Web services, SOAP, WebDAV, SQL, DAML, blah,
  blah, blah
• Necessary for large scale evaluation of automated
  performance
• Larger scale user studies
• Domain experts, automatically generated
  arguments, real world data, enough users to get
  more statistically significant results, more
  challenging baselines

17
Summary
AHEAD Analogical Hypothesis Elaborator for
Activity Detection

• Theme Elaborating terrorist threat hypotheses to
  aid comprehension and use by human analysts.
• Input Hypotheses regarding a potential threat
• Output Argument for and/or against the
  hypotheses
• Approach Interpretive CBR process
• Evaluation Users with arguments were
• Faster
• More confident
• More accurate
• More reliable in their judgments of confidence