HPKB briefing

1
Information Extraction and Transport,
Inc. with Veridian PSR, GMU / IB3, and Paul
Cohen under the direction of Murray Burke,
DARPA RKF Evaluation Issues August 2, 2000
2
TKCP Spec

• TKCP spec V1.1 issued
• Y1 content ECB Chapter 7 (most)
• SQs issued
• ECB
• Komachi
• IET
• Chapters 5 6 (portions)
• Chapter 7 (reserved)

3
Question Ground Rules

• IETs V1.1 spec is liberal
• w.r.t. KRR issues
• w.r.t. source difficulty ratings
• Teams to identify KRR red flags / issues
• Textbook content
• SQ-introduced issues SQ version space
• Accepted SQs
• Rejected SQs
• Consensus ontology of question KRR issues
• Spec includes initial issues assessment for SQs
• Initial feedback milestones Nov 3, Dec 1

4
Pump Priming

• No viable alternatives
• Knowledge for its own sake
• Prerequisite, target, fundamental primitives,
  common sense
• Concepts, relations, axioms
• Knowledge for tools sake
• Lexical, grammatical
• Ground rules
• Right division of labor over KEs, SMEs
• Anticipated RKF solutions not ready yet

5
E2E Evaluation Strategy
IETs SME Lab
Checkpoint KB
Checkpoint KB
KB Delta
Process Modeling
Tools
Test Taker (SME)
Integration Teams
KB Prior
KB
Gatekeeper KE
Jun 2000 Evaluation Begins
Aug 2000 Evaluation Ends
Sep 2000 Specs Ready
Integrator KEs
6
Experimental Conditions

• User type KE vs. SME
• KR tool analogy, sketch, dialog
• TQ descriptive, model-based, what-if
• Pump priming primitives, prerequisite