The EVA Barometer

1
The EVA Barometer

• Signe Ploug Hansen
• Special Adviser at the unit for methodology and
  IQA
• The Danish Evaluation Institute (EVA)
• sph_at_eva.dk

2
Overview of the presentation

• What is a barometer good for?
• Why did EVA go for a barometer?
• What is the content of the EVA Barometer?
• What kind of information does a barometer
  provide?
• What does it take to use the barometer concept?
• How to get started?

3
What is a barometer good for?

• Can summarise external (as well as internal)
  feedback by providing performance indicators
• Provides you with overall expressions of the
  performance of your agency
• Represents a strong communicative tool for
  informing external stakeholders about the
  performance
• Validates the information you get from individual
  surveys

4
Why did EVA go for a barometer?

• Reduce information overload created by a growing
  number of internal and external feedback surveys
• Benefit from the fact that many isolated surveys
  had much overlapping in terms of themes
• Achieve an overall view of EVAs performance in
  terms of these common themes
• Be able to provide clear and simple information
  about the performance of the agency

5
The content of the EVA barometer 1/2

• The seven themes
• The quality of information provided by EVA
• EVAs abilities in terms of cooperation
• EVAs abilities in terms of process management
• EVAs knowledge abilities in terms of
  evaluation methods
• EVAs sector specific knowledge and
  understanding
• The usefulness of the quality assurance process
• The quality of the final evaluation report.

6
The content of the EVA barometer 2/2

• Questions of the theme information (translated
  versions)
• How satisfied or dissatisfied were you with
  EVAs information about
• the process of the project?
• the content of the project?
• the method(s) applied?
• your workload?
• Possible answers
• Satisfied, Largely satisfied, Largely
  dissatisfied, Dissatisfied

7
What kind of information does a barometer
provide?The information theme as an example
Receivers of EVAs quality assurance surveys
Identical questions regarding information
Percentage of positive answer (satisfied
largely satisfied) (ex)
Barometer
External experts
1 2 3 4
80
Information. Barometer 83
Quality assured institutions/programs
1 2 3 4
83
Givers of assignments
1 2 3 4
87
8
What does it take to use the barometer concept?

• A number of equally relevant surveys focusing on
  the same themes related to the performance of the
  agency
• Several questions related to each theme in each
  of the surveys
• Identical questions in each of the surveys
• Some years of stability concerning the content of
  the surveys and thus the activities of the agency
• Methodological and technical competencies to
  construct valid surveys and analyse the data

9
How to get started?

• Identify relevant surveys for the barometer
• Identify relevant themes
• Formulate a minimum of four questions related to
  each theme.
• Ensure that the questions make sense in all the
  surveys by testing them among potential
  respondents
• Start using the new questions in all the relevant
  surveys at the same time