COM362 Knowledge Engineering

1
KE Methodologies and Knowledge Acquisition

• John MacIntyre
• 0191 515 3778
• john.macintyre_at_sunderland.ac.uk

2
Content

• Expertise and the Role of Experts
• The Process of Knowledge Acquisition
• Defining the Domain
• Knowledge Acquisition Techniques
• Interviews/Verbatim Protocols/Repertory
  Grids/Rule Induction/Case Studies
• Advantages and Disadvantages
• Interviews Re-visited

3
Expertise in Software

• Definitions of expert systems
• A program which has a wide base of knowledge in
  a restricted domain, and uses complex inferential
  reasoning to perform tasks which a human expert
  could do
• Wellbank, 1983
• A computer system which is designed to help
  people with tasks involving uncertainty and
  imprecision, and which require judgements and
  knowledge
• Hart, 1992

4
The Role of Experts

• Major source of knowledge for KE
• Have a body of knowledge unfamiliar to the layman
• Have a proven track record
• Demonstrate
• Effectiveness
• Efficiency
• Awareness of limitations
• Versatility

5
How Do Experts Help?

• Provide information
• specific items of knowledge
• human text book
• Problem solving
• show how the problem is solved
• describe when extra information is needed
• Explanation
• to support answers or advice
• increase confidence, enhance learning

6
Preparatory Work

• Understand the problem
• talk informally to people
• watch people working
• Background reading
• text books and manuals
• Review existing documentation
• in-house case histories, examples
• Familiarise with key personnel
• name, job title, location, responsibilities

7
Knowledge Acquisition

• Crucial stage in KBS development
• Involves
• Elicitation
• Analysis
• Interpretation
• Some fundamental questions need to be answered to
  apply to real problems

8
Questions

• What is the relationship between knowledge and
  language?
• How can we characterise different domains of
  knowledge?
• What constitutes a theory of human
  problem-solving?

9
The Process of KA

• Mining those jewels of knowledge out of their
  heads one by one
• Feigenbaum McCorduck, 1983
• Involves
• Employing a technique to elicit data
• Interpreting data
• Construction of a model or language which
  describes the knowledge which is determined by
  the data

10
Defining the Domain

• Knowledge must be extracted from experts and
  built into a conceptual model
• Requires agreement between KE and experts on some
  fundamental questions
• What are the inputs and outputs?
• Which inputs cause difficulties?
• How are the problems and solutions characterised?
• What sort of knowledge is used?

11
Knowledge Domains

• KBS work within a restricted problem domain
• Need a formal means of representation to allow
  knowledge acquisition in different domains
• Consider the following domain examples

12
Domain 1 Mathematics

• Has a formal language representation and
  reasoning
• Problematic to see how sub-sets of these
  representations map onto real problems
• Knowledge engineer must create a language to
  describe the mapping process

13
Domain 2 Medicine

• No formal language
• Functionality described in terms of physical laws
• Subject to error
• Knowledge engineer must try to formalise the
  domain

14
Domain 3 Software

• No single stable formal language
• No coherent underlying theory
• Knowledge engineer must try to acquire knowledge
  to support reasoning, and formalise the domain

15
Problem Solving

• Generic set of tasks for problem-solving in KBS
• Interpretation inferring situation descriptions
  from sensor data
• Prediction inferring likely consequences of a
  given situation
• Diagnosis inferring system malfunctions from
  observations
• Design configuring objects under constraints

16
...continued

• Planning designing actions
• Monitoring comparing observations to plan
  vulnerabilities
• Debugging prescribing remedies
• Repair executing a plan to administer a
  prescribed remedy
• Instruction diagnosing, debugging, and repairing
  student behaviour
• Control of system behaviour

17
But...

• Same task in different domains may require the
  solution of a different type of problem
• In practice, many real problems require the
  solution of sub-sets of other problems
• Some problems may in fact be related to the KBS
  or AI approach to previous solutions

18
Problems with Humans!

• Fixation or preservation effects
• Weakness in counterfactual reasoning
• Failure to acknowledge negative evidence for a
  hypothesis
• Data sampling or acquisition problems
• Lack of resolution of complex interactions
• Knowledge engineer must navigate around these in
  knowledge acquisition

19
KA Techniques

• A number of well established techniques
• Differing degrees of formality
• Often a blend of techniques is best
• Can be time-consuming and complex
• The KA Bottleneck

20
Interviews

• Three categories
• Unstructured
• Semi-structured
• Structured
• Three phases in an interview
• Introduction
• Question/Answer
• Closing Summary and Review

21
Disadvantages

• Tendency to hear what we want to hear instead of
  what is being said
• Time consuming
• More complex subjects require high levels of
  concentration
• Interview fatigue
• Tend to extract basal knowledge
• Tend to miss heuristics of the experts knowledge

22
Verbatim Protocols

• Expert thinks aloud through a series of
  simulated examples
• Can be documented or audio/video recorded
• Useful in eliciting broad structure of the
  experts knowledge, and also how it is applied

23
Disadvantages

• Knowledge engineers role is passive
• Tend only to obtain a general overview of the
  experts knowledge
• Usually therefore must be combined with other KA
  techniques
• Success is dependent upon case(s) chosen

24
Repertory Grids

• Devised by George Kelly (psychologist)
• Expert must contribute elements to a grid
• Each element is an entity in the real world
• Expert must express a characteristic or trait
  which differentiates between entities
• Expert supplies opposites and traits which are
  added to the grid

25
Repertory Grids

• Useful technique for eliciting components in the
  domain, and their relationships to one another
• Widely used in automatic KA systems

26
Repertory Grids Example
27
Disadvantages

• Number of elements can become very large
• Only elicits the results of problem-solving
  exercises

28
Rule Induction

• Usually carried out by software
• Therefore a knowledge representation schema must
  have been worked out first
• Expert supplies the software with examples
• Software then formulates rules based on the
  examples
• Process is one of hypothesis formulation based on
  observation

29
Rule Deduction Example

• FACT children start school as soon as possible
  after their fifth birthday
• NEW CASE John is at school
• DEDUCTION John is at least five years old
• NEW CASE Peter is three years old
• DEDUCTION Peter is not at school

30
Rule Induction Example

• Name Age At School?
• John 7 Yes
• Peter 3 No
• Sarah 8 Yes
• Sam 2 No
• Jane 9 Yes
• Induced rule Children must be at least 7 to go
  to school

31
Rule Induction

• Bottom up approach, data dependent
• Uses generalisations to formulate rules
• Reduces the amount of input from the expert
• Commercial examples - the ID3 algorithm, written
  by Ross Quilian

32
Disadvantages

• Sufficient documentation essential
• Extreme values often produced, which can bring
  the whole area of KBS into question
• Noisy or incomplete data can reduce the ability
  of the software to induce rules
• Training set must be selected with care, which
  can be difficult practically

33
Case Studies

• Uses an existing, documented case
• Expert attempts to describe the means of solution
• Cases chosen specifically to elicit deep
  knowledge in the domain
• Useful technique for identifying the experts
  considerations in making decisions

34
Disadvantages

• May not be an appropriate case study with
  sufficient documentation in the domain
• Knowledge elicited is usually not sufficient for
  KBS development
• Success dependent upon case(s) selected

35
Introspection

• Expert is asked to discuss sensations, memories,
  feelings and images he/she experiences in solving
  a problem
• Can be used to extract a global strategy for
  problem solving in the domain
• Three approaches
• Retrospective scenario
• Forward scenario
• Critical incident scenario

36
Disadvantages

• Knowledge elicited may not be truly
  representative of the decision-making process
• Some knowledge will probably stay in the experts
  subconscious
• Success dependent on case(s) elected

37
Conclusions

• The most difficult and time-consuming part of KBS
  development
• Involves the reduction of a large body of
  knowledge to a precise set of facts and rules
• The major bottleneck in KBS development

38
Interviews Revisited

• Interviewing is by far the most common technique
  in KA
• Other techniques (eg repertory grids) can be very
  effective
• Combination of approaches is often best
• PREPARATION!

39
Preparing the KE

• Preparation ESSENTIAL
• Preparation for the Knowledge Engineer
• Review general subject matter
• Review details relating to the application
• Establish the interview procedures and schedules
• If possible, observe experts at work

40
Preparing the Experts

• Preparation for the experts
• Be sure of the primary goals of the application
• Consider how they use their expertise to solve
  the application problem(s)
• Familiarise with the Knowledge Engineer
• Be convinced of the need for the application

41
Degree of Structure

• Decide early on the degree of structure, and
  agree with domain expert
• Do not fluctuate between structured and
  unstructured techniques, and dont let the expert
  wander
• Unstructured interviews tend to result in
  impartial knowledge which is difficult to analyse
• More popular with experts!

42
Interview Techniques

• Two-on-one interviewing can be useful
• Interchanging roles of questioner and note-taker
• Speeds the process up
• Use three questions iteratively to focus the
  interview
• What do you do next?
• What does that mean?
• Why do you do that?

43
Interview Techniques

• Be specific, not general
• Empathise with the expert
• try to be natural and supportive
• Dont interrupt!
• Most of the talking should be done by the expert
• Dont use alien tools
• only use tools which are normal for the job

44
Interview Techniques

• Observe how the expert uses knowledge
• not only facts, theories and heuristics
• also knowledge manipulation
• usually not explicitly stated, therefore needs
  careful observation and recording
• can be elicited by contrasting experts behaviour
  with the intended users behaviour to determine
  expert difference

45
Interview Techniques

• Can use techniques borrowed from psychology
• 20 Questions
• Laddered Grid
• Card Sort
• Techniques should
• be simple to understand and implement
• have simple supporting material
• be reasonably brief

46
Recording

• Accurate recording of questions and answers
  essential
• Can be
• Written notes
• Audio recordings
• Video recordings
• Experts can become intimidated
• Ethical issues

47
Multiple Experts

• Initially, separate interviews if possible
• Brainstorming
• Concensus decisions
• Debriefing

48
Problems in Interviews

• Question bias
• Truth and accuracy
• Misunderstanding
• Wandering
• Interruptions
• Inaccurate recording
• Good preparation and procedures can avoid these
  problems