Knowledge Acquisition and Validation

1
Chapter 11

• Knowledge Acquisition and Validation

2
Knowledge Engineering (KE)

• Art of bringing the principles and tools of AI
  research to bear on difficult applications
  problems requiring experts' knowledge for their
  solutions
• Technical issues of acquiring, representing and
  using knowledge appropriately to construct and
  explain lines-of-reasoning
• Art of building complex computer programs that
  represent and reason with knowledge of the world
• (Feigenbaum and McCorduck 1983)

3
Knowledge Engineering (KE)

• Narrow perspective knowledge engineering deals
  with knowledge acquisition, representation,
  validation, inferencing, explanation and
  maintenance
• Wide perspective KE describes the entire process
  of developing and maintaining AI systems
• We use the Narrow Definition
• Involves the cooperation of human experts
• Synergistic effect

4
Knowledge Engineering (KE)

• KE involves the cooperation of human experts in
  the domain
• A major goal in KE is to construct programs that
  are modular in nature so that additions and
  changes can be made in one module without
  affecting the workings of other modules

5
Knowledge Engineering (KE)
Process Activities

• Knowledge Acquisition
• acquisition of knowledge from human experts,
  books, documents, sensors, or computer files
• Knowledge Validation
• verify and validate until its quality is
  acceptable
• Knowledge Representation
• preparation of a knowledge map and encoding the
  knowledge in the knowledge base
• Inferencing
• software enable the computer to make inferences
  based on the knowledge
• Explanation and Justification
• design and programming the ability to answer
  questions.

6
Knowledge Engineering Process
Source of Knowledge (Experts, others)
Knowledge Validation (Test Cases)
Knowledge Acquisition
Encoding
Knowledge Base
Knowledge Representation
Explanation Justification
Inferencing
7
Scope of Knowledge
Knowledge Sources

• Documented (books, manuals, etc.)
• Undocumented (in people's minds)
• From people, from machines
• Knowledge Acquisition from Databases
• Knowledge Acquisition Via the Internet

8
Knowledge Acquisition Difficulties

• Problems in Transferring Knowledge
• Expressing Knowledge
• Transfer to a Machine
• Number of Participants
• Structuring Knowledge

9
Knowledge Acquisition Difficulties
Other Reasons

• Experts may lack time or not cooperate
• Testing and refining knowledge is complicated
• Poorly defined methods for knowledge elicitation
• System builders may collect knowledge from one
  source, but the relevant knowledge may be
  scattered across several sources
• May collect documented knowledge rather than use
  experts

10
Knowledge Acquisition Difficulties
Other Reasons

• The knowledge collected may be incomplete
• Difficult to recognize specific knowledge when
  mixed with irrelevant data
• Experts may change their behavior when observed
  and/or interviewed
• Problematic interpersonal communication between
  the knowledge engineer and the expert

11
Overcoming the Difficulties

• Knowledge acquisition tools with ways to decrease
  the representation mismatch between the human
  expert and the program (learning by being told)
• Simplified rule syntax
• Natural language processor to translate knowledge
  to a specific representation
• Impacted by the role of the three major
  participants
• Knowledge Engineer
• Expert
• End user

12
Overcoming the Difficulties

• Critical
• The ability and personality of the knowledge
  engineer
• Must develop a positive relationship with the
  expert
• The knowledge engineer must create the right
  impression
• Computer-aided knowledge acquisition tools
• Extensive integration of the acquisition efforts

13
Required Knowledge Engineer Skills

• Computer skills
• Tolerance and ambivalence
• Effective communication abilities
• Broad educational background
• Advanced, socially sophisticated verbal skills
• Fast-learning capabilities (of different domains)
• Must understand organizations and individuals

14
Required Knowledge Engineer Skills

• Wide experience in knowledge engineering
• Intelligence
• Empathy and patience
• Persistence
• Logical thinking
• Versatility and inventiveness
• Self-confidence

15
Knowledge Acquisition Methods
An Overview

• Manual
• Semiautomatic
• Automatic (Computer Aided)

16
Knowledge Acquisition Methods
Manual Methods - Structured Around Interviews

• Process (Figure 11.4)
• Interviewing
• Tracking the Reasoning Process
• Observing
• Manual methods slow, expensive and sometimes
  inaccurate

17
Knowledge Acquisition Methods
Manual Methods
Experts
Elicitation
Coding
Knowledge Engineer
Knowledge Base
Documented Knowledge
18
Knowledge Acquisition Methods
Semiautomatic Methods

• Support Experts by allowing them to build
  knowledge bases with little or no help from KE
• Help Knowledge Engineers by allowing them to
  execute the necessary tasks

19
Knowledge Acquisition
Expert-Driven
Computer-aided (interactive) interviewing
Coding
Knowledge Base
Experts
Knowledge Engineer
optional interactions
20
Knowledge Acquisition Methods
Automatic Methods

• Experts and/or the knowledge engineers roles
  are minimized (or eliminated)
• Induction Method (Figure 11.6)

21
Knowledge Acquisition
Induction-Driven
Knowledge Base
Case histories and examples
Induction system
22
Interviews

• Most Common Knowledge Acquisition Face-to-face
  interviews
• Interview Types
• Unstructured (informal)
• Semi-structured
• Structured
• The knowledge engineer slowly learns about the
  problem
• Then can build a representation of the knowledge
• Knowledge acquisition involves
• Uncovering important problem attributes
• Making explicit the experts thought process

23
Unstructured Interviews

• Seldom provides complete or well-organized
  descriptions of cognitive processes because
• The domains are generally complex
• The experts usually find it very difficult to
  express some more important knowledge
• Domain experts may interpret the lack of
  structure as requiring little preparation
• Data acquired are often unrelated, exist at
  varying levels of complexity, and are difficult
  for the knowledge engineer to review, interpret
  and integrate
• Few knowledge engineers can conduct an efficient
  unstructured interview

24
Structured Interviews

• Systematic goal-oriented process
• Forces an organized communication between the
  knowledge engineer and the expert
• Procedural Issues in Structuring an Interview
• Interpersonal communication and analytical skills
  are important

25
Table 11.1
Procedures for Structured Interview

• The knowledge engineer studies available material
  on the domain to identify major demarcations of
  the relevant knowledge.
• The knowledge engineer reviews the planned expert
  system capabilities. He or she identifies targets
  for the questions to be asked during the
  knowledge acquisition session.
• The knowledge engineer formally schedules and
  plans (using a form) the structured interviews.
  Planning includes attending to physical
  arrangements, defining knowledge acquisition
  session goals and agendas, and identifying or
  refining major areas of questioning.

26
Table 11.1
Procedures for Structured Interview

• The knowledge engineer may write sample
  questions, focusing on question type, level and
  questioning techniques.
• The knowledge engineer ensures that the domain
  expert understands the purpose and goals of the
  session and encourages the expert to prepare
  prior to the interview.
• During the interview the knowledge engineer
  follows guidelines for conducting interviews.
• During the interview the knowledge engineer uses
  directional control to retain the interview's
  structure.

27
Interviews
Summary

• Are important techniques
• Must be planned carefully
• Results must be verified and validated
• Are sometimes replaced by tracking methods
• Can supplement tracking or other knowledge
  acquisition methods

28
Recommendation

• Before a knowledge engineer interviews the
  expert(s)
• Interview a less knowledgeable (minor) expert
• Helps the knowledge engineer
• Learn about the problem
• Learn its significance
• Learn about the expert(s)
• Learn who the users will be
• Understand the basic terminology
• Identify readable sources
• Next read about the problem
• Then, interview the expert(s) (much more
  effectively)

29
Tracking Methods

• Techniques that attempt to track the reasoning
  process of an expert
• Most common formal method
• Protocol Analysis

30
Protocol Analysis

• Protocol a record or documentation of the
  expert's step-by-step information processing and
  decision-making behavior
• The expert performs a real task and verbalizes
  his/her thought process (think aloud)

31
Table 11.2
Procedure of Protocol Analysis

• Provide the expert with a full range of
  information normally associated with a task.
• Ask the expert to verbalize the task in the same
  manner as would be done normally while
  verbalizing his or her decision process and
  record the verbalization on tape.
• Make statements by transcribing the verbal
  protocols.
• Gather the statements that seem to have high
  information content.
• Simplify and rewrite the collected statements and
  construct a table of production rules out of the
  collected statements.
• Produce a series of models by using the
  production rules.

32
Table 11.3 Protocol Analysis
33
Observations
Other Manual Methods

• Observations Observe the Expert Work
• Special case of protocols
• Expensive and time-consuming
• Difficulties
• experts advise several people and several domain
  simultaneously
• observations cover all the other activities as
  well
• large quantities of knowledge

34
Observations
Other Manual Methods

• Case analysis
• Critical incident analysis
• Discussions with the users
• Commentaries
• Conceptual graphs and models
• Brainstorming
• Prototyping
• Multidimensional scaling
• Johnson's hierarchical clustering
• Performance review

35
Expert-driven Methods

• Knowledge Engineers Typically
• Lack Knowledge About the Domain
• Are Expensive
• May Have Problems Communicating With Experts
• Knowledge Acquisition May be Slow, Expensive and
  Unreliable
• Can Experts Be Their Own Knowledge Engineers?

36
Expert-driven Systems
Approaches

• Manual
• Computer-Aided (Semiautomatic)

37
Approaches
Manual Method Expert's Self-reports

• Problems with Experts Reports and Questionnaires
• 1. Requires the expert to act as knowledge
  engineer
• 2. Reports are biased
• 3. Experts often describe new and untested ideas
  and strategies
• 4. Experts lose interest rapidly
• 5. Experts must be proficient in flowcharting
• 6. Experts may forget certain knowledge
• 7. Experts are likely to be vague

38
Benefits

• May provide useful preliminary knowledge
  discovery and acquisition
• Computer support can eliminate some limitations

39
Approaches
Computer-aided

• To reduce or eliminate the potential problems
• REFINER - case-based system
• TIGON - to detect and diagnose faults in a gas
  turbine engine
• Other
• Visual modeling techniques
• New machine learning methods to induce decision
  trees and rules
• Tools based on repertory grid analysis

40
Repertory Grid Analysis (RGA)

• Techniques, derived from psychology
• Use the classification interview
• Fairly structured
• Primary Method
• Repertory Grid Analysis (RGA)
• A grid is a scale or a bipolar construct on which
  elements are placed within gradations

41
How RGA Works

• The expert identifies the important objects in
  the domain of expertise (interview)
• The expert identifies the important attributes
• For each attribute, the expert is asked to
  establish a bipolar scale with distinguishable
  characteristics (traits) and their opposites
• The interviewer picks any three of the objects
  and asks What attributes and traits distinguish
  any two of these objects from the third?
  Translate answers on a scale of 1-3 (or 1-5)

42
How RGA Works

• Step 4 continues for several triplets of objects
• Answers recorded in a Grid
• Expert may change the ratings inside box
• Can use the grid for recommendations

43
Table 11.4 RGA Input for Selecting a Computer
Language
44
Table 11.5 Example of a Grid
45
Knowledge Engineer Support
Knowledge Engineers Tasks

• Advise the expert on the process of interactive
  knowledge elicitation
• Set up and manage the interactive knowledge
  acquisition tools
• Edit the unencoded and coded KB
• Set up and manage the knowledge-encoding tools
• Validate application of the KB
• Train clients

46
Knowledge Engineer Support
Knowledge Acquisition Aids

• Special Languages
• Editors and Interfaces
• Explanation Facility
• Revision of the Knowledge Base
• Pictorial Knowledge Acquisition (PIKA)

47
Knowledge Engineer Support

• Integrated Knowledge Acquisition Aids
• PROTÉGÉ-II
• KSM
• ACQUIRE
• KADS (Knowledge Acquisition and Documentation
  System)
• Front-end Tools
• Knowledge Analysis Tool (KAT)
• NEXTRA (in Nexpert Object)

48
Knowledge Acquisition Objectives
Computer-aided or Automated

• Increase the productivity of knowledge
  engineering
• Reduce the required knowledge engineers skill
  level
• Eliminate (mostly) the need for an expert
• Eliminate (mostly) the need for a knowledge
  engineer
• Increase the quality of the acquired knowledge

49
Knowledge Acquisition Method
Selecting an Appropriate KA

• Ideal Knowledge Acquisition System Objectives
• Direct interaction with the expert without a
  knowledge engineer
• Applicability to virtually unlimited problem
  domains
• Tutorial capabilities
• Ability to analyze work in progress to detect
  inconsistencies and gaps in knowledge
• Ability to incorporate multiple knowledge sources
• A user friendly interface
• Easy interface with different expert system tools
• Hybrid Acquisition - Another Approach

50
Knowledge Acquisition
KA from Multiple Experts

• Major Purposes of Using Multiple Experts
• Better understand the knowledge domain
• Improve knowledge base validity, consistency,
  completeness, accuracy and relevancy
• Provide better productivity
• Identify incorrect results more easily
• Address broader domains
• To handle more complex problems and combine the
  strengths of different reasoning approaches
• Benefits And Problems With Multiple Experts

51
Handling Multiple Expertise

• Blend several lines of reasoning through
  consensus methods
• Use an analytical approach (group probability)
• Select one of several distinct lines of reasoning
• Automate the process
• Decompose the knowledge acquired into specialized
  knowledge sources

52
Multiple Expert Configurations

• Individual Experts
• Primary and Secondary Experts
• Small Groups
• Panels

53
Knowledge Analysis

• Producing the Transcript
• Interpreting the Transcript
• Analyzing the Transcript

54
Producing the Transcript

• Should produce a complete and exact transcript of
  the recorded session.
• In some situation, an exact transcript may be
  produced for only certain sections of the session.

55
Producing the Transcript
Guidelines for producing a transcript
Heading
Passage

• sessions date
• location of session
• attendees
• major theme of the session
• project title

• tape counter number
• paragraph index number
• name of person speaking

56
Guidelines for Interpreting a Transcript

• Identify the key pieces of knowledge, the
  chunks.
• Use handwritten notes taken during the session to
  aid in identifying the key pieces of knowledge.
• If a word processor is used in transcribing the
  information, then the important information can
  be noted by using italics, underlining, or
  bolding techniques.
• If a typewritten version of the transcript is
  produced, highlight the important information
  with a pen.
• Label each piece of identified information with
  the type of knowledge it represents.
• Identify any issues that need further
  clarification.

57
Guidelines for Interpreting a Transcript

• Record each new piece of information with other
  similar pieces of information already discovered.
• Reference each new piece of information to its
  source.
• Relate the piece of information to other recorded
  information in some graphical fashion.
• Review the body of knowledge collected with the
  expert to confirm the knowledge structures.
• Highlight those areas that need to be pursued and
  use them in designing the next knowledge
  elicitation session.

58
Structuring the Knowledge Graphically

• Cognitive Maps
• Inference Networks
• Flowcharts
• Decision Trees

59
Cognitive Maps
Unknown
Age
Employees
Salary
10k-100k
Yes, No
Research, Design
Job
Bonus
Managers
Engineers
Salary
Salary
30k-60k
50k-80k
45
24
27
Age
Age
Age
32k
Salary
Salary
Mary
Bob
Jane
Salary
40k
65k
Job
Bonus
Job
Design
Research
Yes
60
Inference Network
Rule 7
Weather Prediction is Rain
AND
Rule 10
Rule 8
Wind Conditions Indicates Rain
Barometric Pressure Falling
Temperature Moderate
OR
Rule 9
Wind Gusty
Wind Direction From East
60ltTemplt80
Wind Speed gt 5 Knots
61
Flowcharts
Consult Specialist
N
Ask Questions Related to Hypothesis
Obtain Initial Data
Blood Disease Diagnosis
Can Form Hypothesis
N
Hypothesis Right
N
Y
Run Tests to Confirm Hypothesis
Y
Tests Confirm Hypothesis
Prescribe Remedy
62
Decision Tree
63
Validation Verification of the Knowledge Base

• Quality Control
• Evaluation
• Assess an expert system's overall value
• Analyze whether the system would be usable,
  efficient and cost-effective
• Validation
• Deals with the performance of the system
  (compared to the expert's)
• Was the right system built (acceptable level of
  accuracy?)
• Verification
• Was the system built "right"?
• Was the system correctly implemented to
  specifications?

64
Measures of Validation
65
Measures of Validation
66
Measures of Validation
67
To Validate an ES

• Test
• The extent to which the system and the expert
  decisions agree
• The inputs and processes used by an expert
  compared to the machine
• The difference between expert and novice
  decisions
• (Sturman and Milkovich 1995)