Blueprint for Building an Evaluation Plan

1
Building an Evaluation Plan
William Caspar Graustein Memorial Fund Discovery
Initiative Measuring the Outcomes of Discovery
Strategies Workshop
OMG Center for Collaborative Learning July 14,
2005
2
What is an Evaluation Plan?

• It is a narrative description of
• The focus of your evaluation
• Desired outcomes and outcome measures
• Methods/techniques that you will be using to
  collect and understand the data that you will
  need
• Who will be responsible
• The workplan and timeline

3
Lets Construct

• Heres a checklist of the primary components
• Introduction and Background
• A description of the project, strategy or
  activity that you are evaluating
• Research Questions
• Questions that you think you need answers to in
  order to understand the impact of and to improve
  your work

4
Lets Construct

• Program Outcomes and Measures
• The desired outcomes of the work and the measures
  that you will use to indicate that you are
  progressing toward those outcomes
• Methodology and Approach
• Methodology or techniques (e.g., surveys, use of
  agency records) that you will be using to collect
  the measurement data

5
Lets Construct

• Data Collection Management and Work-plan
• The data sources (e.g. administrative data
  sources, respondent groups) that will be used,
  how data will be managed, and who will be
  responsible
• Proposed Products
• - A report, executive summary, a presentation,
  handouts, flyers, posters, charts, press
  releases? Who will receive them? How will they be
  used?

6
Research Questions

• 1. Implementation Was it done? How well? Process
  barriers?
• 2. Opportunities What services or information
  exists? Were services, information, resources
  provided as planned?
• 3. Participation and Utilization How many
  participate/d? Barriers to participation?
• Source We Did it Ourselves An Evaluation
  Guidebook, Sierra Health Foundation, 2000.

7
Research Questions

• 4. Satisfaction Are/Were participants satisfied?
  Why? Why not?
• 5. Awareness What is the level of awareness of
  the subject in the target community? Has
  awareness increased?
• 6. Attitudes, norms What is the perception of
  an activity or service (ex child care quality)?
  Have perceptions changed?
• Source We Did it Ourselves An Evaluation
  Guidebook, Sierra Health Foundation, 2000.

8
Research Questions

• 7. Knowledge What does the target population
  know about an issue or service (ex the parent
  center)? Do they now know more about the issue or
  service?
• 8. Behavior What do people do (ex read to their
  children)? Has the target population altered its
  behavior?
• Source We Did it Ourselves An Evaluation
  Guidebook, Sierra Health Foundation, 2000.

9
Outcomes and Measures

• What are the goals?
• How do they connect to the strategies (link the
  two)
• Examine assumptions as you link the two
• How will progress along the goals be assessed
  what indicators or measures will tell you how you
  are doing
• Short term and Long term and the time-frames for
  each

10
Methodology and Approach

• Specify the data collection methods for each
  measure
• Qualitative and quantitative measures
• 1) Because some things can be counted and some
  things cant be and
• 2) Because what we count never tells us the whole
  story -- qualitative information helps us
  interpret the quantitative data that we collect.

11
Data Collection Management and Work-plan

• What specific types of data will we collect, from
  which sources? Who are the respondent groups to
  reach?
• What will be our timeline for collecting the
  data?
• What systems (computerized or paper) do we have
  in place to manage and store data? If none,
  whats our plan for addressing this gap?

12
Data Collection Management and Work-plan

• What are the tasks (e.g., designing tools,
  building necessary relationships to obtain data)?
• Who is responsible for design, collection,
  analysis, and presentation?
• How long will it take to collect, analyze and
  prepare to present the information?
• How much will it cost?

13
Data Collection Management and Work-plan

• Projecting the Time Involved
• Account for preparation time for focus groups,
  interviews, site visits.
• Logistics of arranging space, refreshments,
  babysitting, transportation, etc.
• Participant recruitment, invitations, follow-up
• Instrument development and training/practice
• Obtaining data from programs or
  administrative/public sources can be time
  consuming
• Plan for pursuit time, follow-up time,
  clarification after receipt, etc.

14
Data Collection Management and Work-plan

• Projecting the Time Involved
• A day of data collection often requires a day of
  analysis
• Plan on two hours of analysis for a two hour
  focus group
• Build time for review and feedback from other
  stakeholders into the preparation of products
  phase
• Allow for flex time in your projection of time
  involved for general project management, meetings
  to discuss the data collection analysis and
  unintended events

15
Data Collection Management and Work-plan

• Projecting the Cost Involved
• Labor costs, consultants
• Direct costs (mailing, copying, telephone use)
• Incentives for participation
• Costs for conducting focus groups or forums
  (food, space, babysitting, transportation)
• Travel costs
• Presentation materials

16
Proposed Products

• Reports and Other Products
• Audience and Purpose
• What would I want to know as a community member?
  What would I want to read and see to be able to
  understand the issue? What would I need to know
  in order to take action?

17
Proposed Products

• Reports (Short and Long)
• Use of Color
• Use of Charts and Graphs
• Use of Text-Boxes and Side Bar Stories
• Use of Pull-out Quotes
• Findings as Headings
• Recommendations as Headings
• Executive Summary (3-5 pages) with all findings
  and conclusions as bullets

18
Proposed Products

• Other Vehicles for the Information
• Share Feedback (executive summary, flyers,
  conversations) with other stakeholders (families
  and other community members collaborative
  partners referral sources)
• Invite Reponses Hold a community forum or
  smaller focus group meeting to discuss findings,
  recommendations and next steps

19
Proposed Products

• Other Vehicles for the Information
• Use poster displays to share results post in
  schools, community centers, health center waiting
  rooms, etc.
• Use post-cards or flyers that community members
  can pick up at the local grocery store,
  laundromat, etc.
• Include options for getting involved action
  steps and who to contact