Methods of Rapid Evaluation and Assessment

1
Methods of Rapid Evaluation and Assessment

• Evaluators Call July 31, 2007

2
Overview

• Types of rapid appraisal/assessment
• Types of rapid evaluation
• Commonalities
• Differences
• Discussion of how we might be able to employ
  these methods into our own work

3
Rapid Evaluation and Assessment Methods

• Rapid evaluation and assessment methods
• Quick, generally inexpensive study methods often
  used to provide an understanding of constraints
  and facilitating factors (i.e., lessons learned)
• Formal (not-so-rapid) assessment methods
• More time-consuming and generally more expensive
  often used to provide evidence of effectiveness
  and include quantitative behavioral surveys and
  facility assessment

4
Rapid Evaluation and Assessment Methods

• Types of rapid appraisal/assessment
• Rapid assessment (RA)
• Rapid ethnographic assessment (REA)
• Participatory rural appraisal (PRA)

5
Rapid Evaluation and Assessment Methods

• Rapid assessment (RA)
• RA is often used to generate information about
  health and social problems to aid in the design
  of culturally appropriate interventions for
  health and social problems
• RA is usually
• Team based
• Participatory
• Ethnographic inquiry (alternative to prolonged
  fieldwork)

6
Rapid Evaluation and Assessment Methods

• Rapid ethnographic assessment (REA)
• REA is very similar to RA
• Key differences
• REA tends to make use of a more limited range of
  research methods
• REA is more exclusively focused on exploring
  indigenous understandings of health issues

7
Rapid Evaluation and Assessment Methods

• Participatory rural appraisal (PRA)
• Principally employed in developing countries in
  the areas of natural resource management,
  agriculture, poverty, and food security
• Emphasis is on information gathering as a process
  with defined characteristics, including
• Community involvement
• Holistic and systematic approach
• Multidisciplinary and interactive methods
• Flexible responses
• Emphasis on communication and listening skills
• Visual display of information

8
Rapid Evaluation and Assessment Methods

• Types of rapid evaluation
• Real-time evaluation (RTE)
• Rapid-feedback evaluation (RFE)
• Rapid evaluation methods (REM)

9
Rapid Evaluation and Assessment Methods

• Real-time evaluation (RTE)
• RTE developed in the 1990s in response to the
  proliferation of humanitarian crises
• RTEs use a mixed-methods approach
• Involves semi-structured interviews, site visits,
  a limited number of in-depth interviews, focus
  groups, and reviews of secondary documentation

10
Rapid Evaluation and Assessment Methods

• Rapid-feedback evaluation (RFE)
• RFE involves the use of existing program data to
  make a quick, preliminary assessment of program
  performance
• Wholeys RFE model consists of five steps
• 1. Collection of existing data on program
  performance
• 2. Collection of new data on program performance
• 3. Preliminary evaluation
• 4. Development and analysis of alternative
  designs for full-scale evaluation
• 5. Assisting policy and management decisions
• Some evaluators have argued that you can skip
  step 4 depending on the existing data.

11
Rapid Evaluation and Assessment Methods

• Rapid evaluation methods
• Developed by WHO to assess the quality of health
  care services, identifying operational problems,
  and assisting managers in taking corrective
  action
• Data are collected by means of clinic exit
  interviews with patients, health staff
  interviews, observations of staff task
  performance, community and staff focus group
  discussions, review of clinic records, checking
  of facilities, and household interviews

12
Common Methods

• Quantitative data
• Collected through mini surveys and the review of
  existing data sets
• Qualitative data
• Formal and informal interviews with key
  informants/stakeholders
• Focus groups and community group interviews
• Naturalistic observations
• Mapping is also commonly used

13
Common Processes

• Participatory
• Usually involves community members/ target
  population in the framing of study
• Team-based
• Usually involves the collaboration of team
  members throughout process (from planning and
  data collection to the interpretation of
  findings)
• Iterative
• Usually involves the analysis of data while they
  are still being collected and the use of
  preliminary findings to guide decisions about
  additional data collection

14
Key Differences

• Assessment/Appraisal types (RA, REA, and PRA)
• Typically focus on assessing a situation or
  context within a defined geographic area
• Use ethnographic methods to capture the
  dimensions of local cultures relevant to the
  problem
• Evaluation types (RTE, RFE, and REM)
• Typically focus on projects, programs, or systems
• Concerned with the thoughts and behaviors of
  actors with projects, programs and systems
• Do not pay as much attention to cultures outside
  of the projects, programs, or systems being
  investigated

15
Advantages of Rapid Methods

• Low cost
• Quickly conducted
• Flexible (to explore new ideas)

16
Disadvantages of Rapid Methods

• Usually relate to specific communities or
  localities (difficult to generalize)
• Less valid, reliable, and credible than more
  formal methods
• Although McNall et al (2007) provides methods for
  countering this

17
How can we make use of these methods

• Are AETCs currently using these methods?
• Would you consider incorporating these methods
  into your current evaluation activities?
• Where and how could AETCs use these methods?

18
Resources

• McNall, M., Foster-Fishman, P.G. Methods of Rapid
  Evaluation, Assessment, and Appraisal. American
  Journal of Evaluation, 28 (2), 151-168.
• Morin, S., Maiorana, A., Koester, K., Sheon, N.,
  Richards, A. "Community Consultation in HIV
  Prevention Research A Study of Community
  Advisory Boards at 6 Research Sites." JAIDS, 33,
  513-520
• http//www.rapidassessment.net
• http//www.who.int/docstore/hiv/Core/Index.html