Methodological Issues for Assessing Democratic Governance

1

• Methodological Issues for Assessing Democratic
  Governance
• Facilitators Ken Mease, Francois Roubaud
• and Mustafa Khawaja

2
Generic Issues

• How can we get data?
• What data can we get?
• How to use the data?
• Golden Rules

3
How is the information gathered?

• Participatory group discussions with less
  focused questions and more opportunity for free
  thinking, but cannot be representative
• Mail-in Survey it needs educated target (can
  read write) and existence of safe system in
  returning the responses
• Internet surveys need to be carefully targeted
  so that the desired population is reached and
  represented

4
How is the information gathered?

• Face-to-Face Interviewer Surveys Respondents will
  need to feel comfortable without fear or favour
• Desk study to find out the so-called hard
  facts about countries. What information was
  included and excluded?
• Expert coding of narrative reports a coding done
  by academics or other experts on a range of
  primary source material

5
What are the primary data sources/ types?

• Mainly the indicators are based on four
  primary types of data
• Standards, codes and treaties
• Events-based data
• Narrative reports
• Surveys

6
Samples What is the sample population?

• For any data source it is important to be clear
  about the sample used to provide the data.
• It is crucial to indicate that results are
  representative or not representative of the
  desired population
• There is a potential trade-off between a smaller
  (and cheaper) sample, or a larger one which would
  (may) be more representative. This is
  particularly important if data needs to be
  disaggregated (for example by region, income
  group, age, gender, race, religion).

7
Surveys, Different types of sampling

• Targeted aims to obtain results from a specific
  group of the population.
• General population surveys of the general
  population can (if carried out properly) claim to
  be representative.
• Specific geographic areas it is also possible
  to do surveys in specific locations.

8
Questions Specific to particular people or
applicable to all?

• Will broad coverage questions be understood by
  the majority of typical people?
• Example from the Afrobarometer survey 2005
  Overall, how satisfied are you with the way
  democracy works in your country?
• Narrow coverage questions, however, may be
  designed for more detail or for particular
  sub-populations.

9
Questions Specific to particular people or
applicable to all?

• Example question from WGA survey of
  well-informed persons How well defined is the
  separation of powers between the judiciary,
  legislature and executive in your country? (pg.
  86)
• If the survey is owned by the surveyor,
  questions will be tailored more to the owners
  needs.

10
What data can we get?
11
What data can we get? Objective measures

• Objective measures are constructed from
  indisputable facts. Typical examples of these
  might include the signature of treaties,
  financial measures and the existence of
  particular bodies
• Internal perceptions- are results based upon the
  views of respondents from within the country.
• External perceptions - are results based upon
  assessments made by non-residents (often experts)
  of the country.

12
What data can we get? Perception verses Reality

• When looking at perception measures it is
  important to bear in mind a number of factors.
• There may be a time lag.
• Perceptions are founded upon events which people
  remember.
• Hence perception and reality can be different.

13
How to use the Data
14
Discrete scales and additive measures

• Many available indicators provide ratings for
  countries on a pre-defined scale. The result is
  that the full range of over 200 countries in the
  world needs squeezing into a very limited set of
  possible ratings. This can lead to variance
  truncation
• Additive measures where scores are added
  together. This is normal for marking
  examinations, etc. However, it becomes a problem
  when applied to absolute standards.

15
Composite and Aggregate indicators

• Strictly speaking, a composite indicator is one
  which combines different things into a single
  measure. This is also often called an Index.
• A well known example of this would be the Human
  Development Index that combines education data
  with infant mortality rates and many other
  development indicators .

16
Aggregate indicators

• An aggregate indicator is one which combines
  different measures of a similar concept into a
  single measure.
• A widely cited example of this is the Governance
  Matters Indicator, which draws together 31 data
  sources into six composite indicators. (see page
  54).
• A scale is a set of numerical values assigned to
  subjects, objects, or behaviors for the purpose
  of quantifying the measuring qualities.

17
Indicators and data sources
Sec 1
Sec 2
Sec 3
Sec 4
Sec 5
Sec 6
Source 1
Source 3
Source 2
Source 4
18
Checklist for indicator attributes

• Validity - does the indicator measure what it
  purports to measure?
• Reliability - can the indicator be produced by
  different people using the same coding rules,
  questionnaire,or source material?
• Measurement bias is there any systematic
  measurement error?
• Transparency are the details available
  concerning the production of the indicator?

19
Checklist for indicator attributes

• Representativeness - what is the nature of the
  survey sample of individuals?
• Variance truncation - the degree to which scales
  force observations into indistinguishable
  groupings
• Information bias - what sources of information
  are being used?
• Aggregation problems - for combined scores, to
  what degree are aggregation rules logically
  consistent or overcomplicated?
• Ordering and wording of the questions

20
Three Golden Rules Rule 1 - Use a range of
indicators

• There is no single governance indicator which
  captures the subtleties and intricacies of
  national situations in a manner that enables
  global, non value-laden comparison.
• The key is a balanced set of indicators

21
Golden Rule 2 Use an existing indicator as a
first question not a last

• As an indicator becomes more detailed, it is more
  likely to point towards actions which could lead
  to an improvement in the result.
• Yes/No Asking the basic question, does a
  problem exist? (Be careful)
• Number After determining that a problem exists,
  determining the size of the problem (How?).
• Percentage Put the problem into context.
• Significance Use a significance test to examine
  whether the problem is evenly spread or certain
  groups face more of a problem.

22
Golden Rule 3 Understand an indicator before you
use it

• This is perhaps the most crucial rule of all,
  since by using an indicator you can be considered
  to be implicitly endorsing it, including its
  methodology and normative assumptions.

23

• Thank You!!!