Monitoring and Evaluating the CARE Planning Process

1
Monitoring and Evaluating the CARE Planning
Process

• C. Scott Thompson, MPH
• HRSA ALL Titles Meeting
• Washington, DC
• August 28-31, 2006

2
Introduction

• Name
• Where you are from
• On word describing your thoughts about evaluation

3
Agenda Overview

• Introductions
• Designing an Evaluation
• Gathering Credible Evidence
• Managing and Analyzing Evaluation Results
• Using the Evaluation Results
• Workshop Evaluation/Wrap-up

4
Training Objectives

• By the end of this workshop,
• you will be able to
• Understand and list the six steps of evaluation
• Describe possible goals, objectives, and
  indicators of each planning activity
• Conduct an evaluation of planning activities

5
(No Transcript)
6
Benefits of Evaluating Planning Activities

• Evaluating Planning Activities has several
  benefits
• Helps ensure your planning activities are in line
  with the requirements and recommendations
  outlined by Congress and HRSA/HAB
• Fosters buy-in from the Council
• Helps ensure that all information and planning
  processes are open to all stakeholders involved
  with the planning activity
• Helps you outline possible areas for improvement
  in the overall planning process

7
Importance of Engaging Key Stakeholders

• Ensures that diverse perspectives are included,
  understood, and addressed
• Facilitates support of the evaluation process and
  evaluation results
• Builds trust and buy-in

8
Identifying Key Stakeholders

• In general, the following principal groups of
  stakeholders is critical
• Those involved with the planning council and
  carrying out the planning activities
• Those affected by the outcomes of the planning
  process
• The primary users of the evaluation and
  evaluation results

9
Engaging Key Stakeholders

• Key stakeholders can be engaged by implementing
  one or a combination of the following strategies
• Develop an Advisory Group/Ad Hoc evaluation
  committee
• Develop periodic reports at select intervals or
  milestones in the evaluation
• Provide communication/updates at planning council
  meetings

10
What are Goals?

• General, big picture statements of what a program
  intends to accomplish
• For planning activities, goals are the broad
  statements about a long-term outcome or ultimate
  purpose of that activity

11
What are Objectives?

• Objectives describe the more specific processes
  or outcomes that must be reached in order to
  accomplish the broader goal
• For planning activities, objectives are the big
  steps planning councils must take in order to
  attain the planning activity goal
• Objectives answer the questions, What steps must
  be completed in order to accomplish our goal?

12
What are Indicators?

• Qualitative or quantitative measures that help
  the Planning Council define and measure progress
  toward the goals and objectives of a planning
  activity
• Helps you know if you are achieving the intended
  objectives of each planning activity and
  ultimately the overall goal
• Help you develop the evaluation questions for
  each objective

13
Goals, Objectives, and Indicators
Goal
Objectives
Indicators
14
Sections of the Evaluation Worksheets (1 of 2)

• For each planning activity, the following
  sections are included
• A cover sheet includes the planning activity
  being evaluated and instructions on how to use
  the worksheets.
• A summary page of the goals and objectives of the
  planning activity.

15
Sections of the Evaluation Worksheets (2 of 2)

• The objectives are divided into three core
  evaluation tasks
• Design the tasks that need to happen or
  resources that need to be secured prior to
  conducting the activity
• Implementation activities that take place to
  complete the planning activity
• Output the end result of the planning activity

16
Sections of the Individual Evaluation Worksheets

• Each individual worksheet has four sections
• Objectives
• Indicators
• Evaluation Questions
• A section for additional comments

17
Steps to use the Evaluation Worksheets

• Planning Councils should conduct the following
  steps to use the worksheets
• Select the planning activity that you would like
  to evaluate
• Decide which objectives you would like to include
  in the evaluation
• If desired, tailor the objectives following the
  SMART guidelines.
• Tailor the corresponding evaluation questions.
• Determine the format or data collection process
  for asking the questions.

18
Gathering Credible Evidence

• Consider what sources of evidence you will use,
  including
• Documents by-laws, meeting minutes, the
  comprehensive plan, and public comment
• People Planning council members, providers, and
  consumers
• Direct observation observing certain planning
  activities that were implemented

19
Data Collection Instruments and Methods(1 of 2)

• The most frequently used data collection methods
  include
• Paper and pencil questionnaires A self-report
  method useful for collecting information that you
  want to quantify. (Counts, Frequencies,
  Averages). Also good for establishing baselines.
  Can be completed by planning members to allow
  them to evaluate the scope and quality of the
  planning process.
• Interviews useful for collecting in-depth
  information about a topic or issue either as a
  one-on-one interview or a focus group.

20
Data Collection Instruments and Methods(2 of 2)

• The most frequently used data collection methods
  include
• In-depth personal interview useful for
  conducting interviews with the chair(s) of the
  planning council to understand how planning
  activities were planned and completed.
• Focus group interview useful for obtaining
  feedback from a group of similarly situated
  stakeholders (I.e., PLWHA, service providers,
  community activists, etc.)
• Record or document review an efficient way to
  obtain evaluation information as you have limited
  to no reliability on individuals to gather the
  information

21
Adapting Data Collection Instruments

• Some issues to consider in adapting existing
  instruments
• Each instrument and the items in it should be
  clearly linked with the specific evaluation
  questions and indicators you are trying to
  measure.
• Any instrument should be culturally appropriate
  with the wording of questions, language, and
  reading level appropriate to the intended
  respondents.

22
Adapting Data Collection Instruments

• Some issues to consider in adapting existing
  instruments
• Eliminate wording bias to avoid leading
  respondents to the desired answer.
• The response format should match the question
  format
• Instruments should be sensitive to potential
  response burden so respondents are not
  overburdened with the length of instrument and
  instrument completion

23
Why Develop an Evaluation Plan

• A well-organized evaluation plan gives you a
  road-map for implementing your evaluation and
  keeps you on track
• It helps give you a comprehensive picture of
• What you are doing?
• Why you are doing it?
• When you are doing it?
• Who is doing what to implement the evaluation?

24
Developing an Implementation Strategy(1 of 2)

• Developing an implementation strategy means
  outlining the following questions
• How will the evaluation be implemented?
• What is our process for completing the
  evaluation?

25
Developing an Implementation Strategy(2 of 2)

• The evaluation implementation strategy is an
  important step as it will outline
• The activities or tasks that need to be
  completed.
• Key milestones or deliverables in the evaluation
  process
• The activity/task completion dates
• When the evaluation will end

26
Managing the Evaluation Process

• Track issues such as
• Time spent on the process
• Response rates from planning council members or
  other respondents
• Success of data collection methods (interviews,
  questionnaires)
• Data collection inconsistencies - If the same
  questions tend to be missed, it may indicate that
  the questions are confusing, misunderstood, or
  not applicable
• Data needed for analysis but unavailable
• Printing, supplies, and other materials beyond
  the cost of committee of individual time spent on
  the evaluation process

27
Managing the Evaluation Process

• If you find that a data collection method is not
  working
• Avoid mid-stream changes as it can weaken the
  evaluation findings and frustrate planning body
  members
• If the problem is so substantial that you are
  likely to get little useful data, then a
  mid-stream correction may be needed

28
Data Management Strategies

• Assign an individual or committee to monitor and
  track the data collection process
• Develop a data collection log for monitoring the
  flow of data from its source to the data
  collection monitor
• Develop procedures for monitoring the collection
  of data.
• Develop a quality monitoring plan for reviewing
  the quality of the data received

29
Analyzing Evaluation Data

• Analyzing the results means figuring out the
  meaning of the findings and linking the data to
  specific evaluation questions that you are trying
  to answer.

30
Summarizing Evaluation Data

• The Summary of Evaluation Responses Form
• was designed to help you gather and review all
  findings in one place.
• No matter which questions you choose for each
  planning activity, the results can be recorded
  and tallied in one easy-to-use document.

31
Summary of Evaluation Responses Priority Setting
Comments _______________________________________
__________________________________________________
__________________________________________________
__________________________________________________
__________________________________________________
__________________________________________________
___________________________________
32
Summarizing Evaluation Data

• Use all survey and evaluation results you receive
  related to the corresponding planning activity
  objective.
• Place the total number of yes responses in
  column A.
• Place the total number of no responses in
  column B.
• Place the total number of not sure responses in
  column C.
• In column D, place the total number of responses
  (yes, no, and not sure). You get this number by
  adding columns A, B, and C.
• In Column E, divide the total number of yes
  responses (column A) by the total number of
  responses (column D) to get a percentage of
  positive responses for each objective.

33
Summarizing Evaluation Data

• As an optional step, you may also want to
  determine the percentage of Not Sure responses
  in Columns F.
• These responses can demonstrate uninformed or
  uninvolved planning council members or may
  represent a need for training and orientation to
  the process.
• You can also use this form to add additional
  comments from the surveys or from other methods
  of data collection.

34
Summary of Evaluation Responses Priority Setting
Comments I was out when they talked about the
priority setting process, so I was not
involved. We all were involved in deciding the
process, reviewing the priorities, and the
outcome. What are our conflict of interest
guidelines?
35
Interpreting Evaluation Findings

• When interpreting the evaluation findings, you
  are answering the questions
• How should we interpret the data to answer our
  evaluation questions?
• What do our evaluation results tell us about how
  we are completing our planning activities?
• What is the significance of the findings?
• Why did the findings turn out this way?
• What were the limitations of our evaluation
  process?

36
Interpreting Evaluation Data

• Use the following steps to interpret evaluation
  data
• Step 1 Bring all evaluation committee and CPG
  members together to review.
• Step 2 Look for common themes in both
  qualitative and quantitative data.
• Step 3 Explain themes in a narrative format.
• Step 4 Make note of unexpected outcomes,
  especially if the evaluation findings do not meet
  your expectations.
• Step 5 Include supporting information that
  suggests reasons that the unexpected results
  occurred and what you could do to address the
  unexpected outcomes.

37
The Evaluation Report

• The purpose of the report is to identify what
  worked well and explain areas that could use
  improvement.
• The audience for the evaluation report include
  all of the stakeholders identified in Module 1.
• It is important to tailor the report based on the
  specific stakeholder audience.
• For example, HRSA/HAB staff may expect a full
  written report on the findings, but the needs
  assessment (NA) committee may only be interested
  in the findings that have an impact on the NA
  process.

38
Evaluation Report Outline

• Executive Summary
• Introduction
• Explanation of the purpose
• A section on methods and evaluation design
• Presentation of the results
• Recommendations
• Listing of the strengths and weaknesses of the
  evaluation

39
Using the Evaluation Results

• Prior to disseminating your evaluation results to
  your key stakeholders, ask other planning council
  members for their first impressions and
  suggestions. These questions might include
• Do the findings seem reasonable?
• Are they presented clearly?
• What questions do they raise that are not
  answered in the report?
• Are explanations of problem areas and proposed
  remedies satisfactory and feasible?
• What other tables or charts would be helpful?
• Does anything seem to be missing?
• Once suggestions are made, revise the evaluation
  report
• As needed.

40
Evaluation Activity Instructions

• Work in small groups to interpret evaluation data
  for one of four planning activities
• Read H 4-5, Sample Evaluation Data for the
  groups assigned planning activity
• Use H4-4, Evaluation Analysis and Reporting Notes
  to develop recommendations based on the sample
  evaluation data presented.

41
Follow-up and TA

• To request Technical Assistance contact your HRSA
  Project Officer.
• For questions regarding the training or to
  request training directly from AED, please call
  Scott Thompson at 202-884-8895 (sthompso_at_aed.org)