TESTING AND EVALUATION

1
TESTING AND EVALUATION
Contents 1. Evaluation, validation,
verification 2. Testing and evaluation of ES and
DSS 3. Knowledge base verification (consistency
and completeness) 4. Testing and evaluation of DSS
2
(No Transcript)
3

• Are the conclusion given by the system correct?
• Is the system reliable and does it give
  consistent conclusions?
• Is the system robust?
• Is the input data easily available?
• Is the reasoning appropriate?
• Is the output relevant, timely, and meaningful?
• Is the system easy to use?
• Does the system integrate successfully with
  the organisational setting in which it is going
  to be used?

4

• Evaluation
• Objective is to asses a system overall
  value is it usable, efficient, cost-effective,
  etc.
• Validation
• Deals with the performance of the system
• Verification
• Checking whether the system is correctly
  implemented to its specifications.

5

• Testing and evaluation of ES
• Types of knowledge that can be changed
• inference rules
• concepts
• control strategies
• Testing and evaluation of DSS
• Objective is to improve effectiveness of
  decision making. How to measure it?

6
Knowledge Base Verification Logical verification
of the rules can detect problems. 1. consistency
2. completeness
7

• 1. Checking for consistency
• Redundant rules equivalent condition parts and
  one or more of the conclusions are the same
• Rule 1 IF firm's net working capital gt 0,
• AND trend in net working capital is
  negative
• THEN solvency rating is low
• Rule 2 IF trend in net working capital is not
  positive,
• AND firm's net working capital is positive
• THEN solvency rating is low
• difficult maintenance

8

• Conflicting rules equivalent condition parts
  and conflicting conclusions
• Rule 1 IF solvency rating is average,
• AND profitability rating is average
• THEN apply for a secured loan
• Rule 2 IF solvency rating is average,
• AND profitability rating is average
• THEN apply for an unsecured loan
• concluding variable may belong to a
  multi-valued attribute

9

• Subsumed rules same conclusions, one of the
  rules contains additional constraints
• Rule 3 IF firm's net working capital gt 0,
• AND trend in net working capital is
  negative
• AND firm's currency ratio lt 2
• THEN solvency rating is low
• Rule 1 IF firm's net working capital gt 0,
• AND trend in net working capital is
  negative
• THEN solvency rating is low
• Rule 1 is subsumed in Rule 3

10

• Unnecessary IF conditions same conclusions, IF
  condition in one rule is in conflict with the IF
  condition in the other rule
• Rule 4 IF firm's net working capital lt 0,
• AND trend in net working capital is
  negative
• THEN solvency rating is low
• Rule 1 IF firm's net working capital gt 0,
• AND trend in net working capital is
  negative
• THEN solvency rating is low
• two rules can be combined into one
• Rule 4 IF trend in net working capital is
  negative
• THEN solvency rating is low

11

• Circular rules chaining of the rules in the set
  forms a cycle
• Rule 5 IF management competence is good,
• AND financial credit rating is good
• THEN overall credit rating is good
• Rule 6 IF overall credit rating is good,
• AND trend profitability is very good
• THEN management competence is good
• Rule dependency

R1 IF A and B THEN C R2 IF B and C THEN
D R3 IF C and D THEN A
12

• 2. Checking for completeness
• Missing rules due to
• unreferenced attribute values
• missing combinations of condition attribute
  values

3?3 combinations conflicting rules? missing
combinations (Average,Average) (marginal,
Average)
13
The number of rules may be less than the number
of possiblecombinations 1. Disjoint
conditional parts IF Management competence is
high or average AND financial analysis rating
is high THEN creditworthiness is high 2.
Meaningless combinations of values
14

• Modification techniques
• Reordering of rules
• Adding extra conditions to the IF part of the
  rule (specialisation)
• Weakening the IF part of the rule
  (generalisation)

15

• Testing and Evaluation of DSS
• System performance response time,
  availability, reliability, data entry output
  format, usage time, users, man-machine interface.
• Task performance aspect
• the quality of the decision
• decision making time spent to make a decision,
  alternatives, etc.
• user perception trust, satisfaction,
  understanding
• Business opportunities
• cost-benefit analysis
• Evolutionary Aspects
• How well the system can adapt to changes in
  environmental factors, problem characteristics
  and decision making behaviour.

16
Exercise. Given the following rule set (R1) IF
A OR B THEN D (R2) IF A THEN E (R3) IF E THEN
D (R4) IF D OR E THEN B What types of consistency
problems do we encounter in this rule set?
17
References 1. Knowledge-based Decision Support
Systems, With Applications in Business, 2nd
Edition, M. Klein, L. Methlie, Wiley,
1995. 2. Decision Support Systems and
Intelligent Systems, Fifth Edition E.Turban, Jay
Aronson, Prentice Hall, 1998.
18
SUMMARY

• Evaluation, validation and verification are
  introduced.
• Verification of the knowledge base can detect
  consistency and completeness problems.
• Consistency problems can be caused by redundant
  rules, conflicting rules, subsumed rules,
  unnecessary IF-conditions, and circular rules.
• Completeness problems can be caused by missing
  rules due to unreferenced attribute values, or
  missing combinations of attribute values.
• Testing and evaluation of ES and DSS are
  different.