Interactive Knowledge Acquisition Tools: A Tutoring Perspective

1
Interactive Knowledge Acquisition ToolsA
Tutoring Perspective

• Yolanda Gil
• Jihie Kim
• USC/Information Sciences Institute
• August 9, 2002

2
Motivation Investigate Synergies between
Instructional Systems and Acquisition Tools
SOFTWARE
USER
?
Instructional System
teaches
Good Tutoring Principles
Acquisition Tool
Good Learning Principles
teaches
?
3
Our Previous Work in Knowledge AcquisitionThe
EXPECT Project at USC/ISI

• EXPECT architecture for knowledge-based systems
  exploits highly declarative representations
• Swartout Gil, KAW-95, Gil Melz, AAAI-96
  Blythe et al, IUI-01
• http//www.isi.edu/expect
• Research focus interactive knowledge acquisition
  (KA) tools that help end users to develop
  knowledge bases
• Deriving models of knowledge interdependencies to
  detect knowledge gaps and errors Kim Gil,
  AAAI-99 Kim Gil, IUI-2000 Kim Gil,
  AAAI-2000
• KA dialogue scripts to guide users by following
  up on effects of complex changes Gil Tallis,
  AAAI-97 Tallis Gil, AAAI-99 Tallis,
  IJHCS-2001
• Exploiting background theories to understand how
  new knowledge fits Blythe, IJCAI-2001 Blythe,
  AAAI-02

4
EXPECT A User-Centered Framework for Developing
KBSs
EXPECT
Ontologies and Method libraries
Knowledge Base
Method instantiator
Domain ontologies
Domain ontologies and factual knowledge
CYC/Sensus Upper
Problem solving methods
Evaluations and Critiques
Plans (PLANET)
Domain dependent KBS
Evaluation PSMs
Resources (OZONE)
KA tools

Interdependency Model (IM)
KBS compiler
EMeD
PSMTool
NL Editor
Dialogue plans (KA Scripts)
Knowledge-Based System
Instrumentation
5
Brief Overview of Representative KA Tools (I)

• CHIMAERA McGuinness et al 2000
• Acquisition of concepts, relations, instances
• Diagnoses faulty definitions
• EXPECT Blythe et al 2001
• Acquisition of problem solving knowledge
• Exploits dialogue scripts, interdependency
  models, bg k
• INSTRUCTO-SOAR Huffman Laird 1995
• Acquisition of task models in Soar
• Situated NL instruction is mapped to PSCM Newell
  et al. 1991
• KSSn Gaines Shaw 1993
• Acquisition of concepts, rules, data
• Based on personal construct psychology Kelly
  1955
• PROTOS Bareiss et al 1990
• Acquisition and classification of new cases
• Learning indexes to categories

6
Brief Overview of Representative KA Tools (II)

• SALT Marcus McDermott 1989
• Acquisition of constraints and fixes for
  configuration design
• Exploits Problem Solving Method/ Task
  (Role-limiting approach)
• SEEK2 Ginsberg et al. 1985
• Acquisition of rules
• Uses verification and validation techniques
• SHAKEN Clark et al. 2001
• Acquisition of process models
• User interaction based on concept maps Novak
  1977
• TAQL Yost 1993
• Acquisition of SOAR rules
• Editor for high level language for PSCM Newell
  et al. 1991
• TEIREISIAS Davis 1979
• Acquires and classifies new cases
• Learning indexes to categories

7
Open Challenges in KA

• Users remain largely responsible for the
  acquisition process
• Decide where, what, when, how, why to enter
  knowledge
• System checks errors, may have some short-term
  acquisition goals
• Ideally, KA tools should have student-like
  skills
• Formulate and pursue learning goals
• Keep track of lessons and progress
• Assess how much they are learning and how useful
  k is
• If teacher is not so great, still capable of
  learning

8
Instructional Systems and Acquisition Tools What
Are the Synergies?
SOFTWARE
USER
?
Instructional System
teaches
Supplement Students limitations
Good Tutoring Principles
Acquisition Tool
Good Learning Principles
Supplement Teachers limitations
teaches
?
9
Tutoring and Learning Principles Relevant to KA
Kim Gil, ITS 02 (I)
Teaching/Learning principle Tutoring literature
Start by introducing lesson topics and goals Atlas-Andes, Meno-Tutor, Human tutorial dialog
Use topics of the lesson as a guide BEE, UMFE
Subsumption to existing cognitive structure Human learning, WHY, Atlas-Andes
Immediate Feedback SOPHIE, Auto-Tutor, Lisp tutor, Human tutorial dialog, human learning
Generate educated guesses Human tutorial dialog, QUADRATIC, PACT
Keep on track GUIDON, SHOLAR, TRAIN-Tutor
Indicate lack of understanding Human tutorial dialog, WHY
10
Tutoring and Learning Principles Relevant to KA
Kim Gil, ITS 02 (II)
Teaching/Learning principle Tutoring literature
Detect and fix buggy knowledge SCHOLAR, Meno-Tutor, WHY, Buggy, CIRCSIM
Learn deep model PACT, Atlas-Andes
Learn domain language Atlas-Andes, Meno-Tutor
Keep track of correct answers Atlas-Andes
Prioritize learning tasks WHY
Limit the nesting of the lesson to a handful Atlas
Summarize what was learned EXCHECK, TRAIN-Tutor, Meno-Tutor
Provide overall assessment of learning knowledge WEST, Human tutorial dialog
11
Five Main Functions of KA Tools
KNOWLEDGE ACQUISITION BACKEND
ASSIMILATE INSTRUCTION
TRIGGER GOALS
USER INTERFACE
PROPOSE STRATEGIES
PRIORITIZE GOALS STRATEGIES
PRESENTATION DESIGN
Knowledge Base
12
Guidance Exploited by KA Tools
Guidance from Knowledge Base
KNOWLEDGE ACQUISITION BACKEND
Problem Solving Task Knowledge
Domain Knowledge
ASSIMILATE INSTRUCTION
General Background Knowledge
TRIGGER GOALS
USER INTERFACE
Example Cases
PROPOSE STRATEGIES
Guidance from Meta Knowledge
Knowledge Repres. Model
PRIORITIZE GOALS STRATEGIES
Diagnosis Debugging Principles
PRESENTATION DESIGN
Tutoring Learning Principles
13
Tutoring and Learning Principles in KA Tools
Basic Conceptual Framework
KNOWLEDGE ACQUISITION BACKEND
USER INTERFACE
ASSIMILATE INSTRUCTION
Operational Principles
Knowledge Editor
TRIGGER GOALS
General Tutoring Learning Principles

• Dialogue
• Goals Strats
• State
• Suggestions
• History

PROPOSE STRATEGIES
PRIORITIZE GOALS STRATEGIES
PRESENTATION DESIGN
Knowledge Base
14
Tutoring and Learning Principles Implicit in KA
tools
Design Presentation
Trigger Goals
Assimilate Instruction
Tutoring/Learning principle
Prioritize Goals Strats
Propose Strategies
EXPECT, SEEK2
Introduce topics goals
SALT
EXPECT
SEEK2
SALT
Use topics of the lesson as a guide
PROTOS, SALT
PROTOS
Subsumption to existing cog. structure
TEIREISIAS
EXPECT
TEIREISIAS
INSTRUCTO-SOAR
PROTOS
Immediate feedback
EXPECT
TEIREISIAS
Generate educated guesses
Keep on track
INSTRUCTO-SOAR
INSTRUCTO-SOAR
Indicate lack of understanding
EXPECT,CHIMERA
TAQL
Detect and fix buggy K
Learn deep models
Learn domain language
SEEK2
Keep track of answers
EXPECT
Prioritize learning tasks
Limit the nesting of lessons
Summarize what is learned
KSSn
Assess learned knowledge
15
Tutoring and Learning Principles in KA Tools

• Observation Some learning and tutoring
  principles are used in some aspects of the
  dialogue by some tools
• Opportunity Incorporate principles more
  thoroughly in all aspects of the dialogue
• Observation These principles are implicit in the
  tools code and thus are limited
• Opportunity Exploit declarative representations
  of learning state, goals, and strategies

16
SLICK (Skills for Learning and Interactively
Capture Knowledge)
USER INTERFACE
KNOWLEDGE ACQUISITION BACKEND
SLICK Dialogue Manager
Proactive Dialogue Window
Active Acquisition Goals Strategies
Awareness Annotations
Tutoring Learning Principles
KB State
Dial. History
17
Conclusions

• Analysis of existing KA tools shows they use
  tutoring/learning principles
• Sparsely
• Implicitly
• Current capabilities of KA tools can be improved
  by
• Representing tutoring/learning principles
  declaratively
• Organizing the dialogue around lesson topics
• Keeping track of how knowledge improves through
  dialogue
• Exposing what knowledge has been assimilated and
  what areas need improvement or testing
• Assessing their competence and confidence on
  question answering