Normalisation

1
Composite indicators The art of mixing apples
and oranges by Stefano Tarantolastefano.taranto
la_at_jrc.it Seminar composite indicators
development and guidance Firenze, 15-16 October
2007

2

• Outline
• Context and controversy
• Steps to build CIs
• Open issues in building CIs
• From Saltelli, A., 2007, Composite indicators
  between analysis and advocacy Social Indicators
  Research, 81 , 65-77.

3

• Context and controversy

4
Context
A composite indicator measures multi-dimensional
concepts (e.g. competitiveness, well-being,
environmental sustainability, etc.) which cannot
be captured by a single indicator. Ideally, a
composite indicator should be based on a
theoretical framework, which allows individual
indicators to be selected, combined and weighted
in a manner which reflects the dimensions of the
phenomenon being measured. Nardo, M. M.
Saisana, A. Saltelli and S. Tarantola (EC/JRC),
A. Hoffman and E. Giovannini (OECD), Handbook On
Constructing Composite Indicators Methodology
And User Guide, OECD Statistics Working Paper
JT00188147, STD/DOC(2005)3.
5
Google search
How many of them are around? Searching
composite indicators
6

• Among structural indicators being developed by
  the EC we find
• Price convergence between EU Members States
• Healthy Life Years
• Biodiversity
• Urban population exposure to air pollution by
  ozone and
• Urban population exposure to air pollution by
  particles (PM10)
• Consumption of toxic chemicals
• Generation of hazardous waste
• Recycling rate of selected materials
• Resource productivity
• E-business index
• Can you guess how many of these are composite?

7
See (http//www.oecd.org/publications/)
8

• The pros of composite indicators
• Can summarise complex or multi-dimensional
  issues in view of supporting decision-makers.
• Easier to interpret than trying to find a trend
  in many separate indicators.
• Facilitate the task of ranking countries on
  complex issues in a benchmarking exercise.

9
The pros of composite indicators Can
assess progress of countries over time on complex
issues. Reduce the size of a set of indicators
or include more information within the existing
size limit. Place issues of country
performance and progress at the centre of the
policy arena. (Advocacy) Facilitate
communication with general public (i.e. citizens,
media, etc.) and promote accountability.
(Advocacy)
10

• while composite indicators cons are
• May send misleading policy messages if they are
  poorly constructed or misinterpreted.
• May invite simplistic policy conclusions.
• May be misused, e.g., to support a desired
  policy, if the construction process is not
  transparent and lacks sound statistical or
  conceptual principles.

11

• and (cons)
• The selection of indicators and weights could
  be the target of political challenge. (a CI could
  exacerbate disagreement rather than focus minds)
  
• May lead to inappropriate policies if dimensions
  of performance (that are difficult to measure)
  are ignored.

(if one dislikes the policy under discussion,
he/she will most likely invoke one or more of
these)
12
A quote from the JRC
ltlt it is hard to imagine that debate on the
use of composite indicators will ever be settled
official statisticians may tend to resent
composite indicators, whereby a lot of work in
data collection and editing is wasted or
hidden behind a single number of dubious
significance. On the other hand, the temptation
of stakeholders and practitioners to summarise
complex and sometime elusive processes (e.g.
sustainability, single market policy, etc.) into
a single figure to benchmark country performance
for policy consumption seems likewise
irresistible. gtgt Saisana M., Saltelli A.,
Tarantola S. (2005) Uncertainty and Sensitivity
analysis techniques as tools for the quality
assessment of composite indicators, Journal of
the Royal Statistical Society - A, 168(2),
307-323.
13
A quote from Sharpe
ltltThe aggregators believe there are two major
reasons that there is value in combining
indicators in some manner to produce a bottom
line. They believe that such a summary statistic
can indeed capture reality and is meaningful, and
that stressing the bottom line is extremely
useful in garnering media interest and hence the
attention of policy makers. The second school,
the non-aggregators, believe one should stop once
an appropriate set of indicators has been created
and not go the further step of producing a
composite index. Their key objection to
aggregation is what they see as the arbitrary
nature of the weighting process by which the
variables are combined.gtgt Andrew Sharpe, 2004,
Literature Review of Frameworks for
Macro-indicators, report of the Centre for the
Study of Living Standards, Ottawa, CAN.
14
Last quote from OECD/JRC
ltltComposite indicators are much like mathematical
or computational models. As such, their
construction owes more to the craftsmanship of
the modeller than to universally accepted
scientific rules for encoding. As for models, the
justification for a composite indicator lays in
its fitness to the intended purpose and the
acceptance of peers (Rosen, 1991) gtgt.
The quality of a CI is its fitness for purpose
i.e. its ability to attract public to discuss
the problem at hand, meeting stakeholders
requirements and no more.
15
two more pros Constructing/underpinning
narratives for lay or literate audiences. (later
in this talk)
16
and Comparing effectively complex dimensions
with one another.
Good environmental results correlate
significantly with good governance. Policy
emphasis at the national and global levels on
establishing the rule of law, eliminating corrupti
on, promoting a robust policy dialogue, and
setting up effective regulatory institutions
appears fully justified.
17

• Steps to build CIs

18
From the handbook. Step 1. Developing a
theoretical framework What is badly defined is
likely to be badly measured Excerpt For
example, the Growth Competitiveness Index (GCI)
developed by the World Economic Forum is founded
on the idea that the process of economic growth
can be analysed within three important broad
categories the macroeconomic environment, the
quality of public institutions, and technology."
19
From the handbook. Step 1. Developing a
theoretical framework After Step 1. the builder
should have A clear understanding and
definition of the multidimensional phenomenon to
be measured. A nested structure of the
various sub-groups of the phenomenon. A list
of selection criteria for the underlying
variables, e.g., input, output, process.
Clear documentation of the above.
20
The Canadian Council on Learning develops the
Composite Learning Index (http//www.ccl-cca.ca/)
Learning to Know Learning to Know involves
developing the foundation of skills and knowledge
needed to function in the world. This includes
literacy, numeracy, general knowledge and
critical thinking. Learning to Do Learning to
Do refers to the acquisition of applied skills.
It can encompass technical and hands-on skills
and knowledge, and is closely tied to
occupational success. Learning to Live
Together Learning to Live Together involves
developing values of respect and concern for
others, fostering social and inter-personal
skills, and an appreciation of the diversity of
Canadians. This area of learning contributes to a
cohesive society. Learning to Be Learning
to Be refers to the learning that helps develop
the whole person?mind, body and spirit. This
aspect concerns personal discovery,
self-knowledge, creativity and achieving a
healthy balance in life. Pillars from
Jacques Delors Task Force UNESCO's
International Commission on Education for the
Twenty-first Century.
21
Step 2. Selecting variables A composite
indicator is above all the sum of its parts
Excerpt The strengths and weaknesses of
composite indicators largely derive from the
quality of the underlying variables. While
the choice of indicators must be guided by the
theoretical framework for the composite, the data
selection process can be quite subjective as
there may be no single definitive set of
indicators.
As in mathematical models
22
Step 2. Selecting variables After Step 2. the
builder should have Checked the quality of the
available indicators. Discussed the strengths
and weaknesses of each selected indicator. Made
scale adjustments, if necessary. Created a
summary table on data characteristics, e.g.,
availability (across country, time), source, type
(hard, soft or input, output, process).
23
Step 3. Multivariate analysis Analysing the
underlying structure of the data is still an art
After Step 3, the builder should have
Identified sub-groups of indicators or groups of
countries that have common statistical features

24
Step 4. Imputation of missing data. The idea of
imputation could be both seductive and dangerous
Step 5. Normalisation of data Avoid adding up
apples and oranges Step 6. Weighting and
aggregation The relative importance of the
indicators can become the substance of a
negotiation
25
Step 7. Robustness and sensitivity Sensitivity
analysis can be used to assess the robustness of
composite indicators Step 8. Links to other
variables Composite indicators can be linked to
other variables and measures
26
Statistical test of robustness by simultaneously
changing all uncertain factors
Performance index
Italy
Greece
Spain
Model averaging as we dont trust one model to
make inference, we use more models
27
Step 9. Back to the details De-constructing
composite indicators can help extend the analysis
Step 10. Presentation and dissemination A
well-designed graph can speak louder than words
28
The practices described in the handbook are
already being applied to existing composite
indicators, e.g. the 2006 EPI from Yale,
Columbia, WEF, JRC and to the Canadian Learning
Index .
29

• Open Issues in CI building

30
(1) A composite constructed on the basis of
underlying indicators with high internal
correlation is likely to give a very robust CI,
whose values and ranking are moderately affected
by changes in the selection of weights, the
normalisation method and other steps involved in
the analysis.
Open issues in CI Building 1 Variables
correlation
31
(2) When building composite indicators using
automated tools such as factor analysis, one
seeks to obtain a set of totally uncorrelated new
variables. While this can be a powerful tool to
benchmark countries performance, or to produce
e.g. leading or lagging synthetic indicators, the
interpretation in terms of original variables
becomes more difficult.
Open issues in CI Building 2 Variables
correlation
32
(3) At the same time, it would be very difficult
to imagine a composite indicator made of truly
orthogonal component indicators.(4) We would
consider the existence of correlation among the
attributes of an issue as a feature of the issue,
not to be corrected for through re-weighting of
components. However, if two attributes are
linked via a causal entailment, there might be
reasons for using only one attribute to
characterise the issue.
Open issues in CI Building 3, 4 Variables
correlation
33
Example The European Innovation Scoreboard
USPTO and EPO indicators are kept both in the
summary innovation index, though they are
correlated at 0.97 -legal differences in the
two systems US data cover grants, EPO data count
applications. Granted patents are a subset of all
patent applications and do not reflect the total
innovative capacity. - The year of the grants
has nothing to do with the point in time when the
invention took place it reflects the processing
capacity of the patent office. It is only
possible to draw a picture of the situation five
to six years before the actual point in time,
which is only seldom comparable to the actual
situation. Cont.
Open issues in CI Building 3, 4 Variables
correlation
34
Open issues in CI Building 3, 4 Variables
correlation
USPTO is a national patent office. A country has
a certain home advantage at the national office
US applicants at USPTO are overrepresented in
relation to applicants from other countries.
This home advantage does not find its
equivalent for European applicants at the EPO,
as there are still many national patent offices
all over Europe that receive a large number of
important and innovative patent applications.
So it seems fair to include both EPO and USPTO
patent data in the innovation index, which
explains the still increasing gap between US and
EU in innovation performance
Rainer Frietsch Fraunhofer ISI, Germany Comments
on the European Innovation Scoreboard 2005,
February 2005
35
(5) Weights are customarily conceived as
importance measures. In additive aggregations
weights are substitution rates (wi/wj ratio
of substitution of indicator i with indicator
j.
Open issues in CI Building 5 Aggregation
methods
This implies a compensatory logic, i.e. the
possibility of renounce to one point of indicator
i with wi/wj points for indicator j .
However, when one is not willing to compensate
(i.e. trade literacy with GDP per capita), a
multi-criteria approach can be used (Munda and
Nardo, 2003).
36
(6) The approach uses a Condorcet ranking
procedure and produces country rankings instead
of country scores. The ordering is based on
pair-wise comparison of countries across all the
indicators.
Multi-criteria approach
A B C
ABCABACBC ACB BAC BCA CAB CBA
Countries A, B, C Outscoring matrix
A B C
A B C
25 17 13
5 10 7
Take the combination with the highest value as
the best
37
(7) With this approach no compensation occurs.
To exemplify, a country that does marginally
better on many indicators comes out better than a
country that does a lot better on a few ones,
because the approach does not compensate
deficiencies in some dimensions with outstanding
performances in others. Michaela -?
Multi-criteria approach