The evaluation of publicly funded research activities An overview

1
The evaluation of publicly funded research
activities An overview

• Luis Sanz Menéndez
• OECD Symposium
• Madrid 3 July 2008

2
Issues for debate

• Implications of the use of evaluation as a
  management tool
• Changing evaluation "objects".
• systems and policies,
• programmes,
• research 'collectives',
• individuals and
• institutions.

3
Evaluation as a management tool

• Characterizing evaluation activities- objects-
  aims- approaches - articulation with decision
  making processes
• Different objects (Chabbal's terminology (1987))
• Actors individuals research collectives,
• Operators institutions, procedures, programmes
  and agencies/services, Policies

4
Evaluation objects changing focus

• From the beginning actors at the core of the
  process Researchers and research teams,
  research results and projects Organizing the
  Peer review process
• OECD and the evaluation of national systems Peer
  reviews and indicators
• The eighties and the focus on research
  operators Programmes at the core of new
  methodological developments
• The nineties (1) and the fashion of
  privatisation Focusing on the performance of
  research institutions
• The nineties (2) up to the fashion of
  'excellence' and the growing between multiple
  missions and one sided criteria of performance

5
Evaluation as a management tool

• Different aims 1- AuditsObjective compliance
  to pre-established rules, often
  administrative.Main effect sanctions. 2-
  Assessment of Performance Problems measures,
  differentiating outputs from outcomes.
  Approaches "Summative"/"ex-post". Effects
  hierarchy/positioning, gratifications/
  rewards.3- Relevance of action/activities
  Objective the adequate "implementation
  structures". Approach "pro-active" (focusing on
  the definition of future action).Effects
  changes/adaptations in the course of action4-
  Appropriateness (overall strategy) Objective
  discussing the aims (against the changing
  environment) and/or their translation into goals
  and objectives. Effects redefinition of the
  action and of its course.

6
Evaluation as a management tool

• Approaches to evaluation- evaluation process
  versus tools mobilized- "characterization" (of a
  situation) versus "judgment"- relations between
  approaches and couples "object/aim" --gt typical
  articulations
• The articulation with decision making processes-
  2 different approaches support to the boss or
  as a means for collective learning.- 4 major
  constraints --gt - Timing ---gt to feed back in
  the decision making process - Relevance ---gt to
  address issues at stake - Robustness ---gt to
  resist critics from evaluees - Credibility
  ---gt to be taken up by stakeholders decision
  makers

7
Objects approaches Typical articulations
Object/ Aims/ Approach/ Target Issues Process A
ctors Performance Scientometrics Projects
Quality Peer Reviews System Performance Indica
tors Appropriateness Advisory
councils Operators Performance Audits Relevanc
e Renewed schemes Appropriateness approaches
Source Callon, Laredo and Mustar (eds), 1997,
the Strategic Management of Research and
Technoloy, Paris Economica International
8
Issues for debate related to 'objects'

• National policies (OECD role)
• The evaluation of operators
• Approaches adopted and problems raised by the
  evaluation of 'research collectives' (labs,
  groups, centers, institutes). RAE in UK INSERM
  in France, CSIC in Spain, etc.
• Problems raised around the evaluation of
  individuals CNRS in France CNAEI in Spain SNI
  in México, etc
• The specific case of the strategic management of
  research institutions how to go from the micro
  to the meso level?