DG Joint Research Centre

1
DG Joint Research Centre
Formal and informal approaches to the quality of
information in integrated assessment Stefano
Tarantola January 24-25, 2002 Laxenburg,
Austria
2
Tools for Extended quality assurance
Information used as input to policy-making is
complex, uncertain and disputed.
Established guidelines eg NUSAP and model
Pedigree schemes for the quality assurance of the
decision process.
Silvio.funtowicz_at_jrc.it
3
To set the frame
Leamer, 1990 (economist) I propose a form
of organised sensitivity analysis in which a
neighborhood of alternative assumptions is
selected and the corresponding interval of
inferences is identified.
4
To set the frame
Conclusions are judged to be sturdy only if the
neighborhood of assumptions is wide enough to be
credible and the corresponding interval of
inferences is narrow enough to be useful.
Edward E. Leamer, 1990 Sensitivity Analysis
would help, in Modelling Economic Series, Edited
by CWJ Granger, Clarendon Press, Oxford. Chair in
Man. Bus. Econ., UCLA
5
To set the frame
We move one step further after characterising
the interval of inferences (using e.g. the
statistical variance),
Input factors should be interpreted in sensu
lato - alternative assumptions, - choice of
model, - algorithmic alternatives, -
poorly-known data...
6
The Case Study incineration vs. landfill
(Austria 1994)
Robustness assessment fails the interval of
the inference is too wide
No defensible choice is possible given the
uncertainties.
7
The Case Study incineration vs. landfill
(Austria 1994)

A
B
Space of the assumptions
Output uncertainty
8
Settings for the sensitivity analysis
To validate or invalidate assessments
GSA used to show that the uncertainty in the
decision on whether to burn or dispose solid
waste depends on the choice of the system of
indicators, and not on the quality of the
available data. Money should not be spent to
improve quality in data, but to reach a
consensus on the proper system of indicators.
Tarantola et al., in Saltelli et al. Eds, (2000)
Sensitivity Analysis John Wiley
V(Y)VE(YXi)EV(YXi)

9
Settings for the sensitivity analysis
Problem simplification and dialogue optimisation
We look for those uncertain factors that have
negligible influence on the output. These can be
fixed to the most plausible value within their
range of variation. The dimensionality of the
input space is then reduced.
Useless discussing about the use of different
architectures to build the composite indicator,
when these do not influence the result.
10
Settings for the sensitivity analysis
Output uncertainty reduction
Joint use of UA and GSA (iterative procedure).
Perform UA and get a confidence interval for the
output
If it is unacceptably large, acquire better
knowledge on the most important factors. Perform
UA again to check ...
It the output quality exceeds the requirements,
the specifications on the input quality can be
relaxed, starting from the less important
factors.
Crosetto and Tarantola (2001) Int J Geogr Inf
Science
11
Bibliography
1 Saltelli, A., K. Chan, M. Scott, Editors,
2000, Sensitivity analysis, John Wiley Sons
publishers, Probability and Statistics
series.2 Saltelli, A., Chan, K., Scott, M.
Eds., 1999, Special Issue on sensitivity
analysis, Computer Physics Communications, 117.
3 Saltelli A., Tarantola S., and Chan K.,
1999, A quantitative, model independent method
for global sensitivity analysis of model output,
Technometrics, 41(1), 39-56. 4 Saltelli A.,
Tarantola S., Campolongo F., 2001, Sensitivity
analysis as an ingredient of modelling,
Statistical Science, 15(4), 377-395.