Group 3: Evaluating Research

1
Group 3 Evaluating Research

• January 24- 2001
• Southampton Workshop

2
Why evaluate?

• Holds the researchers accountable to the donor.
• Encourages researchers and funders are
  accountable downward to communities
• Improves the quality and impact potential of
  future research.
• Provides justification for further funding
• Identifies unexpected added value (that
  researchers hadnt realized)
• Provides donors with justification to their
  higher-ups on the value of the research

3
Why NOT to evaluate?

• Opportunity costs alternative use of funds
• Some consider research a waste of money what
  then of funds used to evaluate research?

4
Types of RH Research

• Basic research for improving knowledge
• Operations research
• --Large scale testing systems
  (population-level outcomes)
• --Smaller scale testing elements within a system
  (program-level)
• Drug trials

5
Alternative methodologies for evaluating research

• Case study good for learning about good
  practices and generating lessons learned 
• Systematic review of a portfolio of projects on a
  set of indicators
• Continuous reporting of results (quarterly
  reports)
• Assessment by external evaluation team
• Biblio-metric assessment number publications,
  type of publication, and citations.
• Audit to evaluate impact over time

6
Recommended principles for evaluating research
(a start)

• 1. Clarify the expectations for evaluation at the
  onset of the research.
• 2. Make evaluation proportional to the cost of
  the research
• 3. Make evaluation constructive, not punitive.
• 4. Recognize that there is an element of
  irrationality
• need to remain flexible
• 5. Evaluate different types of research on
  different criteria

7
Other issues

• Is this a concern of the rich countries only?
  Do poor countries have the luxury to spend
  additional funds on evaluating research? 
• There seems to be a dichotomy of useful and
  subjective versus more mechanical but more
  systematic methodologies.
• The impact of the research may not occur until
  years later if too much time elapses, it becomes
  more difficult to attribute change to the
  research.
• -- Alternatively, one can ask has the research
  been USEFUL-- used in further research, cited, etc

8
Other issues

• Possible problems with inter-project peer review
  
• groups collude to evaluate each other favorably
  or
• peers know that they will be competing in the
  future, so they are hesitant to reveal
  information about the inner workings or
  weaknesses of the program.

9
Other questions

•  What are the differences if one has a program
  versus a project?
• What are the pathways that are likely to lead to
  the desired outcome for different types of
  research in different environments?
• What do we mean by monitoring, evaluation, and
  impact?