Ottawa, 79 November 2005 http:farmweb'jrc'cec'eu'intci 1

1
Robustness analysis for Composite
Indicators Michaela Saisana michaela.saisana_at_j
rc.it European Commission Joint Research Centre
Ispra, Italy Composite Indicators
Workshop Ottawa, 7-9 November 2005
2
A measure of a countrys present environmental
quality and capacity to maintain or enhance
conditions in the years ahead.
www.yale.edu/esi
3
Environmental Systems
Human Vulnerability
5 Components
21 Indicators

• Air Quality
• Biodiversity
• Land
• Water Quantity
• Water Quality

• Environmental Health
• Basic Human Sustenance
• Natural disaster vulnerability

Environmental Stresses

• Reducing Air Pollution
• Reducing Ecosystem Stress
• Reducing Population Growth
• Reducing Waste and Consumption Pressures
• Reducing Water Stress
• Natural Resource Management

Global Stewardship
Social and Institutional Capability

• Participation in International Collaborative
  Efforts
• Greenhouse Gas Emissions
• Reducing Transboundary Environmental Pressures

• Environmental Governance
• Eco-Efficiency
• Private Sector Responsiveness
• Science/Technology

4
The ESI is the equally weighted average of the 21
indicators
aggregated into
5
Main issues in the construction of the ESI
Selection criteria Country size (population gt
100000 area gt 5.000 km2) Variable coverage
(country excluded if observed less than 45/76
variables) Indicators coverage (country excluded
if no information for 19/21 indicators)
Variable standardization for cross-country
comparison (with GDP, population, land area,
mammal species,), Z-scores Variable
transformation for outliers (very small/large
values) 2 step procedure - before imputation
of missing values
logarithmic transformation
- after imputation
transform back to the original
scale unless too skewed
Imputation of missing data Markov Chain Monte
Carlo method for 18.6 of missing cells
in the dataset
6
Results
7
China ESI ranking 133 GDP/capita
4344 Variable coverage 72 Missing variables
imputed 1 (pesticide consumption per hectare of
arable land (stress)) Not included 3 (waste
recycling rate (stress), salinized area due to
irrigation as
of tot. arable land (stress), Down Jones
Sustainability group index (capacity))
8
(No Transcript)
9
15 indicators out of 21 are below the mean
10
Is the Index ROBUST ?
Difference is 6.6
11
Uncertainty and Sensitivity Analysis
How sensitive ESI ranking is to changes in its
structure
ESI Score
Non-compensatory aggregation (non-compensatory
multicriteria)
5 components
Aggregation at the components level
21 indicators
Weighting schemes (budget allocation 17 experts)
76 variables
Imputation (variance from the 30 imputations)
12
Uncertainty and Sensitivity Analysis
How sensitive ESI ranking is to changes in its
structure

• How do the ESI ranks compare to the most likely
  rank under all scenarios?
• What is the optimal scenario for each country?
• Which are the most volatile countries?
• What are the major sources of volatility in the
  ranking?

• Saisana M., Nardo M., Saltelli A. (2005) 2005
  Environmental Sustainability Index, Appendix A.
  (www.yale.edu/esi)

13
Sensitivity of 2005 ESI
How do the ESI 2005 ranks compare to the most
likely ranks?

• Top 35 countries have very low volatility
• For 90 out of 146 countries (in blue) the
  difference between the ESI rank and the most
  likely (median) rank under all scenarios is less
  than 10 positions.

14
Results on 2005 ESI
Leaders

• Top 35 countries have very low volatility
• Congo and Mali volatility gt60 positions (Congo
  could have been ranked among the top 10 under a
  different scenario!)
• Note ESI 2005 more politically correct than ESI
  2001 US now rank 45. It ranked 11 in 2001.

15
Results on 2005 ESI
Middle performers
Laggards
Belgium is the most volatile of the EU25
16
Which are the most volatile countries and why
Volatile difference between the best and the
worst possible rank
17
Why this volatility?
18
(No Transcript)
19
What is a countrys best rank under all
assumptions ?
20

• China most influential factors
• Experts weighting
• Interaction between imputation and
• experts weighting

21
Knowledge Economy Indices
Sub-indicators selected by DG RTD (reviewed and
assessed by experts)
Investment
Sub-indicators
Type of knowledge
Weight Gross D. expend.on RD/cap
(GERD) Knowledge creation 2/24 Number of
researchers/cap (RES) Knowledge creation
2/24 New ST
PhDs/cap (PhD) Knowledge creation
4/24 Total education
spending (GDP)(TES) Knowledge creation
diffusion 4 3/24 Life-long
learning/cap (LLL) Knowledge diffusion
human capital 3/24 E-government
(eGOV) Knowledge diffusioninfo infrast.
3/24 Gross fixed capital formation (GFCF)
Knowledge diffusion new embedded tech.3/24
22
Imputation of missing data
Investment
21 data Missing in 2002
Trend model least squares polynomial regression
t-test for the estimates of the std for
regression coefficients
23
Acknowledging assumptions
SELECTING INDICATORS Inclusion- Exclusion of one
indicator-at-a-time IMPUTATION Trend model -
least squares polynomial regression t-test for
the estimates of the std for regression
coefficients to impute the 38 data gaps (1997
2002) WEIGHTING 1. Equal weights 2. Conceptual
model 3. Benefit-of-the-doubt AGGREGATION 1.
Linear 2. Non-compensatory multi-criteria
approach
24
Investment 2002 Uncertainty analysis results
SE has a 70 probability to be the top country
investing in the KE
25
(No Transcript)
26
KE Indices Sensitivity analysis results
(Sobol method)
First order capture individual impact Total
effect capture interactions/synergies
27
Sensitivity analysis as a tool to identify
thresholds
PhDFR,2002 N (6428,476) TESUK,2002 N
(4.52,0.17)

• Regardless of the changes in the other factors
  (imputed values, aggregation, weighting, set of
  indicators)
• France will not fall behind the 6th position if
  the estimated (expected) number of PhD students
  was 7200.
• UK will not fall behind the 8th position if the
  estimated value for TES in 2002 at 4.52 is the
  correct one.

BE FR AT UK
Selected countries rank versus two important
imputed values.
PhD in FR, 2002
Total education spending in UK, 2002
28
Conclusions

• The two KBE Indices have provided the KEI
  project with a suitable scenario for the
  methodological considerations and analysis.
• St. error of the imputed data lt10 of the
  estimated mean.
• Half of the EU countries show compensational
  effects in the aggregation (ES, FR, DE, IT, NL,
  AT, UK)
• Six countries (PT, UK, ES, FR, IT, SE) are
  sensitive to the set of weights.
• On average (Monte- Carlo), though, the two
  indices are robust to the assumptions related to
  imputation, aggregation and weighting.
• They are significantly influenced by the
  exclusion of one-indicator at a time (desirable
  property! i.e. different aspects)
• This type of analysis can provide useful
  arguments and insights in the debate on the
  construction of composite indicators.

29
Technology Achievement Index
10 uncertain input factors for the analysis
30
Uncertainty analysis Results
The values of the composite indicator are
displayed in the form of confidence bounds
Blue original TAI Red median of Monte Carlo
TAI
31
Uncertainty analysis Results
Significant difference TAI median of
MonteCarloTAI
A few countries show significant overlap and
therefore the ranking is unclear
Blue original TAI Red Monte Carlo TAI
32
In cases where partial overlapping between two
countries occurs, the difference in the TAI
values for that pair of countries can be further
analyzed in via sensitivity analysis
65 of the area
33
?Uncertainty Analysis Sensitivity analysis ?
Moving from uncertainty to sensitivity analysis
Each slice is the fractional contribution of a
factor or group of factors to variance of the
output
34
Our recommended practice is based on two
fractional variance indices one is a first
order effect one factor influence by itself
The other is a factors total influence
inclusive of all interaction with other factors
35
Sensitivity analysis - which uncertain input
affects the difference Singapore
Netherlands? Fixing non influential factors
36
Which uncertain input affects the difference
Singapore Netherlands?
Further analysis (e.g. scatter-plots) reveals
that MC-TAI favours Singapore when high weights
are assigned to Enrolment sub-indicator, for
which Singapore is much better than the
Netherlands, and/or to Electricity sub-indicator,
for which Singapore is marginally better than the
Netherlands.
37
(No Transcript)
38
(No Transcript)
39
Inference from sensitivity analysis for the
difference Singapore Netherlands?

• The weights for Electricity and Enrolment and
  the type of weighting approach (BA or AHP) are
  important
• The selection of the normalisation method does
  not affect the output variance.
• All the input factors, taken singularly, explain
  52 of the output variance. The remaining 48 is
  explained by interactions among the factors.
• The trigger for the weighting scheme has a
  strong interaction with other factors, mainly
  with the weights for Electricity, Enrolment and
  Exports.
• The high value of STi for the weight of Export
  tells us that it cannot be fixed in spite of its
  low Si .

40
A concluding remark
This could guide the use of the appropriate
test Requirements and Assumptions Normal
distribution? Known variances? Equal or
not? Need to compare distributions Are these
scores statistically different from each other?
41
Sources for SA book (2000), ? primer (2004)
? ?free software