TESLA:

1
Long-term environmental thinking
TESLA A Guide to Evidence Support Logic
from data to decisions
2
Outline
Outline

• Why use ESL?
• Decision Making
• Uncertainty
• Building a model
• Analysis

3
Why use ESL?
4
Why ESL?
Why use ESL?
Decisions based on expert judgment typically
involve processing data from a wide range of
sources. Supporting evidence may be complex,
conflicting, incomplete or uncertain, or only of
partial relevance. Assessing these materials to
derive defendable decisions demands an auditable
process that captures all the evidence and how it
fits together. This can also help to ensure that
the outcome is not unduly biased by favorite
evidence.
This presentation introduces Evidence Support
Logic (ESL), a methodology to simplify the
decision-making process. The method has been
implemented in the TESLA software.
5
Why ESL?
Why use ESL?
ESL is intended to support decision makers and
modelers in their sense-making when faced with
extensive information processing requirements. It
facilitates quantification of the level of
confidence that can be placed in hypotheses. ESL
involves systematically breaking down the
question under consideration into a logical
hypothesis model whose elements expose basic
judgements and opinions relating to the quality
of evidence associated with a particular
interpretation or proposition.
6
Why ESL?
Why use ESL?
Even with the help of ESL, decision-making can be
a daunting process. ESL is often used within a
facilitated process whereby an expert panel
identifies and assesses supporting evidence for
relevant hypothesis underpinning the decisions to
be made. At Quintessa we have a number of
experienced consultants who can act as
facilitators during the decision-making process
we know what questions to ask and what to do with
the answers.
7
Decision Making
8
Decisions
Decision Making

• An example decision is
• Is this site suitable for a nuclear waste
  repository?
• The decision is based upon whether a number of
  underlying hypotheses are correct or incorrect.
  Example hypotheses are
• Groundwater flux estimates are correct.
• A waste canister will not leak.
• The water chemical data is of adequate quality.

9
Uncertainty
10
Uncertainty
Uncertainty

• Uncertainty is a key issue in decision making.
  Types of uncertainty are
• Fuzziness (F) imprecision of definition
• Incompleteness (I) what we dont know
• Randomness (R) lack of a specific pattern

11
Uncertainty
Representing Uncertainty in Decisions
Closed world approach

• Represents everything in a particular system.
• Every concept is true or false with no undefined
  or inconsistent states.
• Information is complete all and only the
  relationships that can possibly hold are those
  implied by the given information.

6 sided dice All faces equal All outcomes are
known
Traditional approach does not allow for
incompleteness
Open world approach

• Represents partial knowledge.
• Concepts can be true, false, unknown and
  inconsistent with degrees of uncertainty between.
• Information is sparse, incomplete and possibly
  inconsistent.

?
Unknown number of sides Unknown values Unequal
face characteristics
?
12
Uncertainty
Representing Uncertainty Mathematically
Classical two-value logic

• Lack of knowledge not differentiated
• May lead to false assertions

0.70
0.30
Probability hypothesis is true P(H)
Probability hypothesis is false 1-P(H)
Evidence based three-value logic

• Honest about what is not known
• Allows better analysis of how to tackle the
  remaining uncertainty

Evidence For Evidence Against Uncertainty 1
13
Building a Model
14
Model
Building a Model

• Building a decision model comprises of three
  steps
• Development of a logical hierarchical model
  breaking down a single decision statement into a
  number of underlying hypotheses.
• Parameterisation of the model identification of
  sources of evidence.
• Propagation of evidence through the model
  providing an ultimate assessment for the root
  hypothesis and overall decision.
• All three stages can be completed with the aid of
  TESLA.

15
Model
1. Hierarchical Model Construction
An expert panel decides upon root hypothesis,
representing the decision. Collecting evidence
supporting or refuting this is difficult.
Therefore the panel agrees on sub-hypotheses
which break the root hypothesis down.
The model can be entered directly into TESLA
(right).
16
Model
1. Hierarchical Model Construction
Collecting evidence for these sub-hypotheses is
still difficult. Therefore the panel continues
to break them down into further sub-hypotheses,
until a point is reached where it is felt that
the collection of evidence is relatively
straightforward. Including experts from all
aspects of the decision is imperative to develop
a balanced, fully representative model.
17
Model
2. Elicitation and Parameterisation
The experts collect evidence supporting and
refuting each of the bottom-level (leaf)
hypotheses. Values giving the proportions of
evidence can be entered directly into TESLA the
amount of uncertainty is calculated automatically
from the unassigned proportion of evidence.
18
Model
2. Elicitation and Parameterisation
To aid the assignment of supporting and refuting
evidence, the following are useful

• Each hypothesis has an associated criterion for
  success e.g. for the hypothesis The pH is
  suitable, the criterion might be 6ltpHlt7.
• Linguistic judgements can be translated to
  numerical values using utility functions.
• This approach ensures a consistent and auditable
  process.
• It enables the identification of conflicts.
• It allows initial individual inputs which can be
  revised and fine tuned.

19
Model
2. Elicitation and Parameterisation

• The experts should ask themselves
• How much information would I ideally wish to
  have?
• Compared with that, how much information do I
  actually have?

Complete 100 Nearly complete 83 More than
half 67 About half 50 Less than
half 33 Very little 17 None 0
Knowledge base
Linguistic judgements can be mapped onto a linear
numerical scale
20
Model
2. Elicitation and Parameterisation
From the information available
How much faith do I have in the validity of the
evidence (data quality and reliability of
interpretation)?
How much supporting evidence is there?
Total 100 A lot 83 Quite a lot 67 Average 50
Not a lot 33 Very little 17 None 0
Total 100 A lot 83 Quite a lot 67 Moderate 50
Not a lot 33 Very little 17 None 0
Face value
Quality
21
Model
2. Elicitation and Parameterisation
Knowledge base More than half 67 Face value
of supporting evidence A lot 83 Quality of
evidence Not a lot 33 Amount of supporting
evidence 0.67 0.83 0.33 0.18
0.18
Process must be repeated for each lowest-level
(leaf) hypothesis and for the refuting evidence.
22
Model
2. Elicitation and Parameterisation
Sources of evidence for each individual
hypothesis can be recorded in TESLA, providing a
full audit trail. Files can even be embedded into
the tree, to be accessed at a later date.
23
Model
2. Elicitation and Parameterisation
In addition to deciding upon evidence values for
each leaf hypothesis, the expert panel must also
consider three other concepts Sufficiency,
Dependency and Necessity. Sufficiency This can
be thought of as a weighting (or relevance) for
each sub-hypothesis. Again, a simple linear
mapping can be used to convert a linguistic
interpretation into a numerical value (between 0
and 1).
24
Model
2. Elicitation and Parameterisation
Sufficiency - Example The simple example below
shows how sufficiency can be used to weight the
relevance of data. It also shows how sufficiency
is displayed in the TESLA interface by a number
to the left of the evidence values.
Sufficiency
25
Model
2. Elicitation and Parameterisation
Sufficiency Example (2) Sometimes the
sufficiency relating to supporting evidence may
be different to that of refuting
evidence. Advanced ESL is an optional extra for
TESLA. With this add-on, the user can specify
different values of sufficiency for the
supporting and refuting evidence.
Sufficiency of evidence for
Sufficiency of evidence against
26
Model
2. Elicitation and Parameterisation
Dependency Dependency describes the degree of
overlap or commonality in the sources of evidence
between sub-hypotheses. A simple linguistic
mapping can be used during the elicitation
process
0
0.25
1
0.5
0.75
Independent Low
Moderate High
Totally dependent
The dependency must lie between 0 (independence)
and 1 (total dependence).
27
Model
2. Elicitation and Parameterisation
Dependency - Example The simple example below
shows how dependency can be used to reflect
overlapping data. With standard ESL, only one
value of dependency can be set for sibling
sub-hypotheses. This is indicated in TESLA by a
number underneath the parent hypothesis.
Dependency
28
Model
2. Elicitation and Parameterisation
Dependency Example (2) Advanced ESL is an
optional extra for TESLA. It allows the user to
specify different values for the dependency
between groups of sibling hypotheses.
In our example, there is a lot of overlap between
the national and local forecasts but little
between the forecasts and looking out of the
window.
29
Model
2. Elicitation and Parameterisation
Necessity Sometimes it will be the case that if a
sub-hypothesis is not true, the parent can never
be true either, even if other sub-hypotheses
indicate that this may be the case. For example,
planning permission is required to build on a
site. The site may be perfect in every other
sense but if planning permission is not granted
the build cannot go ahead. There are just two
possible values for necessity true or false.
30
Model
2. Elicitation and Parameterisation
Necessity Example Necessity is demonstrated in
the simple example below. If the horse does not
enter the race, there is no possible way for it
to win even if it is completely fit and has
excellent form. Necessity is indicated in TESLA
by a grey background.
It is most likely that the horse will not run
thus evidence from the non-necessary
sub-hypotheses is ignored.
Necessity
31
Model
2. Elicitation and Parameterisation
Populating a model with values for supporting and
refuting evidence, sufficiency, dependency and
necessity can be quite complicated. A large
number of people may be involved in the process,
many may not have a mathematical background. It
is therefore extremely important that a competent
facilitator takes charge of the elicitation
panel. This facilitator will know which leading
questions to ask the experts to extract the
relevant information to populate the tree.
32
Propagation
3. Propagation of Evidence
Once evidence has been collected and entered for
each of the lowest-level hypotheses, it is
propagated up the tree to the original (root)
hypothesis. This is achieved with a mathematical
algorithm. It is important to note
that Propagation of evidence happens behind the
scenes in TESLA the user does not need to be
aware of the algorithm or have any understanding
of mathematics to use the software. If you wish
to view the algorithm, please click here
Maths
33
Propagation
3. Propagation of Evidence
A negative uncertainty indicates conflicting
evidence.
34
Propagation
3. Propagation of Evidence

• Evidence for and against is propagated from the
  information entered at the bottom of the tree to
  the hypothesis at the top.
• Sufficiency affects the propagation by weighting
  the contribution of each sub-hypothesis.
• Dependency affects the propagation by accounting
  for double-counted sources of evidence.
• Necessity affects the propagation by ignoring
  other sub-hypotheses if a necessary
  sub-hypothesis is false.
• The mathematics of the propagation algorithm are
  handled behind the scenes by TESLA and need not
  concern the average user.

35
Analysis
36
Analysis
Analysis
TESLA provides a number of graphical
visualisation tools to help the user analyse the
decision.

• The tornado plot identifies regions where small
  changes in evidence values (i.e. reducing the
  uncertainty) have a big impact on the overall
  result.
• The ratio plot provides a summary of the
  distribution of evidence in the model. It
  indicates hypotheses which are true or false
  beyond reasonable doubt. It also highlights
  areas of high uncertainty in the model.
• The uniform evidence plot is used to demonstrate
  the outcome of the model in extreme situations.

Examples of these plots are shown on the next
pages.
37
Analysis
Tornado Plot
The tornado plot depicts a sensitivity analysis
of the model. Evidence values for each leaf
hypothesis are perturbed. The impact of this on
the overall result is shown on the graph as a
percentage.
In our example, small changes in the evidence for
the safety assessment indicating suitability have
a large impact on the overall result,
highlighting this as an area for further research.
38
Ratio Plot
The ratio plot summaries how evidence is
allocated for each hypothesis in the model. It
is a plot of uncertainty (or conflict) against
the evidence ratio (evidence for/evidence
against).
Markers are plotted against a coloured
background. Each colour has a different meaning,
as shown here. For a hypothesis to be true
beyond reasonable doubt its marker must lie in
the dark green region.
39
Uniform Evidence Plot
This plot shows what the outcome of the model
would be if all the leaf hypotheses were assigned
the same evidence values. This is most useful
for testing extreme cases e.g. if all evidence
is against with none supporting, or the reverse
situation.
In some cases, complete evidence against may not
result in the root hypothesis being false beyond
reasonable doubt, due to the weightings assigned.
40
Summary
41
Summary
Summary

• Uncertainty is important in making informed
  decisions.
• A decision can be simplified by breaking it down
  into a number of sub-hypotheses, creating a tree.
• An elicitation panel with a competent facilitator
  leading it is imperative to the production of a
  balanced decision.
• Evidence supporting/refuting each bottom-level
  hypothesis is collected by the expert panel.

42
Summary
Summary (continued)

• Data sources can be logged in TESLA for a full
  audit trail.
• Evidence values are propagated up the tree by a
  propagation algorithm.
• Knowledge of the mathematical details of the
  algorithm is not required to make sense of the
  decision.
• Graphical analysis tools help the user to
  identify areas where reducing uncertainty will
  affect the outcome of the decision, and areas
  where it will have little effect, thus targeting
  research.

43
Propagation Algorithm
44
Maths
Propagation Algorithm
There is no need for the average user of TESLA,
or anybody on the elicitation panel, to
understand the mathematics behind the propagation
algorithm. However, for those who have a
mathematical background and are interested in the
mechanics of the method, the next few slides will
explain the derivation of the algorithm.
45
Maths
Propagation Algorithm
Mathematically, the evidence supporting a
hypothesis H is represented as the probability of
H being true P(H). Similarly, the evidence
refuting a hypothesis is represented as the
probability of H being false. Each of these types
of evidence is propagated through the tree
separately, using the same propagation algorithm.
We will derive the algorithm for the supporting
evidence. Uncertainty is not propagated, but is
calculated at each stage using the rule Evidence
For Evidence Against Uncertainty 1
46
Maths
Propagation Algorithm
Consider a hypothesis H which has two
sub-hypotheses, E1 and E2. The probabilities of
each of these being true are P(E1) and P(E2). We
can represent the relationship between the
hypotheses using a Venn diagram
P(H)
The probability of H being true is given by the
overlap between it and the circles representing
the probabilities of the sub-hypotheses (outlined
in white).
H
P(E1)
P(E2)
47
Maths
Propagation Algorithm
This relationship is written mathematically using
intersections, n. The intersection between A
and B, AnB, is simply the overlap between A and
B. Therefore the probability of hypothesis H
being true is given by the expression P(H)
P(HnP(E1)) P(HnP(E2)) P(Hn(P(E1)nP(E2))
We dont want to count the overlap between the
sub-hypotheses twice, hence the subtraction of
this term.
P(AnB) is simply the probability of A
intersecting B.
48
Maths
Propagation Algorithm
If A and B overlap, the dependency is defined as
B
A
Dependency
In words, it is the fraction of the smallest
object that overlaps with the other object.
If r 0 A and B are mutually exclusive,
i.e. AnB 0. r 1 A and B are
totally dependent, i.e. AnB A and/or AnB B.
49
Maths
Propagation Algorithm
Using the concept of dependency, we re-write our
algorithm for the probability of H being true
as P(H) P(HnP(E1)) P(HnP(E2)) r
min(HnP(E1), HnP(E2)). Here the dependency is
given by the expression
We shall see later that in fact this expression
is never used and a numerical value is simply
assigned to r.
50
Maths
Propagation Algorithm
We will now use conditional probabilities to
simplify our expression further. A conditional
probability, P(AB), is the probability of A
given that B has already occurred. Mathematically
it is expressed as
The propagation algorithm can thus be rewritten
in the form P(H) P(HE1)P(E1) P(HE2)P(E2)
r min(P(HE1)P(E1), P(HE2)P(E2)).
51
Maths
Propagation Algorithm
P(H) P(HE1)P(E1) P(HE2)P(E2) r
min(P(HE1)P(E1), P(HE2)P(E2)) Note how the
conditional probabilities in the first two terms
act as weightings for P(E1) and P(E2). In ESL
these quantities are known as the sufficiency
and are written as wi, where i is the index of
the sub-hypothesis. The sufficiency is
represented in the Venn diagram by the overlap
between P(Ei) and H.
If wi 0 There is no overlap wi
1 The overlap is complete
52
Maths
Propagation Algorithm
For two sub-hypotheses, the propagation algorithm
is therefore P(H) w1P(E1) w2P(E2) r
min(w1P(E1), w2P(E2)). This can be extended to n
sub-hypotheses, giving
53
Maths
Propagation Algorithm
The sufficiency, wi, and dependency, r, cannot be
calculated. Instead they are assigned numerical
values by the elicitation panel. Details of this
process are given in the main body of this
document.
54
Maths
Propagation Algorithm
A third quantity, the necessity, is also
described in the main body. Here we describe how
the propagation algorithm is affected by a
necessary sub-hypothesis.

• If the evidence refuting a necessary
  sub-hypothesis is less than 50, the propagation
  algorithm is used as normal.
• If it is greater than 50 and the propagation
  algorithm calculates the evidence refuting the
  parent hypothesis to be smaller than this value,
  the propagation algorithm is not used. Instead
  the evidence values for the necessary
  sub-hypothesis are copied directly to the parent.
• If it is greater than 50 and the propagation
  algorithm calculates the evidence refuting the
  parent hypothesis to be greater than this value,
  the propagation algorithm is used as normal.

55
Maths
Advanced Evidence Support Logic (AESL)
AESL is an optional extra for TESLA. It extends
the ESL methodology in the following ways

• Values for the knowledge base (k(Ei)), quality of
  evidence (q(Ei)) and face value of evidence
  (a(Ei)) can be input directly, giving
• P(Ei) k(Ei)q(Ei)a(Ei)
• Supporting and refuting evidence can be given
  separate values of sufficiency.
• Dependency can be set for subsets of sibling
  hypotheses, rather than a single value for all.
• The propagation algorithm can be overridden for
  any parent hypothesis by setting Any
  sub-hypothesis sufficient or All sub-hypotheses
  necessary. This is explained on the next slide.

56
Maths
Advanced Evidence Support Logic (AESL)
Any sub-hypothesis sufficient Setting this
propagation method override on a parent
hypothesis means that the maximum evidence value
of the sub-hypotheses is taken and copied
directly to the parent, i.e.
All sub-hypotheses necessary Setting this
propagation method override on a parent
hypothesis means that the minimum evidence value
of the sub-hypotheses is taken and copied
directly to the parent, i.e.
57
Maths
Click below to return to the main presentation.
Return