Measuring Outreach Effectiveness

1
Measuring Outreach Effectiveness

• Web4Dev November 2006
• Alex McKenzie - amckenzie_at_worldbank.org
• Knowledge Evaluation Capacity Development
• Independent Evaluation Group World Bank

2
Key Points

• Share IEGWBs approach for aligning web/online
  practice with mandate and business strategy
• Share our experience running several outreach
  campaigns, discuss metrics
• Measuring effectiveness and impact
• Would appreciate hearing about your own
  experience and/or feedback

3
Business Strategy

• IEG is an independent unit that assesses the
  effectiveness of World Bank Groups interventions
• Well regarded for its evaluation capabilities and
  reports
• Combines Accountability Learning
• To be effective, IEG wants to be an influential
  and reliable source of knowledge
• New Outreach and Communications Strategy
• Target key audiences
• Embed outreach throughout the evaluation process
• Accelerate learning, encourage utilization
• Measure results
• Decided on a campaign-based approach, 10 to 12
  major evaluation products per year.
• Multi-pronged outreach campaigns media, print,
  online, in-person, translations, etc.
• Piloted many new ideas, mainstreamed what worked

4
Key Audiences

• Systematic effort to identify audiences has led
  to two sets core campaign-specific
  (specialists)
• Identify decision-makers influencers
• Identify key network nodes
• Evolving audience criteria from static
  classifications, to others based on opportunity
  or time-sensitive demand
• Customize products to audience whenever
  economically feasible (e.g. conference CDROMs,
  translations, etc.)

5
Embed in Evaluation Process
6
Encourage Utilization
7
Encourage Utilization
8
Measuring Results

• Results-Based Management causal chain

9
Measuring Results Causal Chain
Source Strengthening EAPs Results
Orientation, April 2004
10
Measuring Results - Outputs
11
Measuring Results - Outputs
12
Measuring Results - Outcomes

• Intermediate Outcomes perception survey of IEG
  evaluation products, covering
• Readership
• Quality
• Influence on Bank processes and policies (similar
  for external clients)
• Utilization
• Annual Review of Operations Evaluation (AROE)
  2006, to be released in November/December.
• Source IEGs Annual Review of Operations
  Evaluation (AROE). See http//www.worldbank.org/i
  eg/aroe

13
Measuring Results - Outcomes

• Survey targeted individuals defined as intended
  audience 4285 internal (WBG) and 2759 external
  contacts. Results are indicative for respondents
  (22) but are not generalizable to population

14
Perception of Quality Staff/EDs (E4)
15
Perception of Quality External Clients (E13)
16
Influence Staff (E8)
17
Influence External Clients (E15)
18
Staff Use of Evaluations (E9)
19
External Clients Use of Evaluations (E16)
20
Measuring Results Survey Implications

• IEG provides good services to its main client,
  the Board. External audiences are also highly
  satisfied with the quality of IEG evaluations and
  considered them to be influential.
• However external audiences would like IEG to
  broaden its external consultation during
  evaluations.
• Bank staff see IEG products useful with deepening
  their understanding of a given subject.
• However they dont see them as useful in
  day-to-day work. IEG needs to focus on
  influencing ongoing and future operations.

21
Measuring Results Survey Implications

• All audiences suggest
• High value placed on timeliness to provide
  information when decisions need to be made
• Deepening analysis through more consultation and
  attention to context
• Focus on operational value of findings
• A call for action for IEGs outreach program
• 4-page briefs
• Customized findings brief, upstream, via email
• More use of non-print media
• Balance lecturer approach with conversations

22
Thoughts Going Forward

• "What information consumes is rather obvious it
  consumes the attention of its recipients. Hence,
  a wealth of information creates a poverty of
  attention."
• Herbert Simon, Nobel laureate

23
Your Experience/Feedback

• What has worked in your organization? (audience
  analysis relevance/timeliness measuring
  outputs, outcomes, impacts, etc.)
• Choices related to channels? High quality
  content vs. open/collaborative models
• Outreach models used? Worthwhile consultants
  advice?
• What challenges do you see ahead?