Evaluating Community Coalitions and Partnerships

1
Evaluating Community Coalitions and Partnerships

• Frances D. Butterfoss, PhD
• Center for Pediatric Research
• Norfolk, Virginia

2
Coalition...

• Group of groups with a shared goal and
• some awareness that united we stand
• and divided we fall.
• Ron Labonte, 1993

3
Coalitions

• Each coalition is unique developed within
  context
• of its community
• Coalition often serves as umbrella organization
  to
• help facilitate variety of issues projects for
  member organizations
• For many grant-funded projects, coalition is
  foundation underlines what we do

4
What Coalition Partnerships Do

• Network share information
• Plan coordinate
• Implement specific projects
• Provide technical assistance training
• Advocate

5
Characteristics of Effective Coalitions

• Well defined, specific issue
• Agreed upon vision goal
• Solidarity among members
• Unambiguous adversary or health problem
• Strong leadership

6
Benefits of Coalition Approaches

• Span boundaries between populations
  organizations in various settings
• Minimize duplication of effort services
• Build trust respect among organizations who
  previously experienced mistrust or conflict
• Effectively pool innovative talents resources
• Reach previously untapped community assets
• Implement some interventions (e.g., policy
  media advocacy) better than traditional public
  health social service agencies

7
Challenges of Coalition Approaches

• Coalitions take time to build in order to be
  able to
• react to public health issues
• Ownership of issues power ultimately rests
  with
• coalition members
• Health issue (e.g., asthma) has to resonate with
  
• stakeholders take priority over other issues

8
Why Evaluate Coalitions?

• Builds capacity within coalition community
• Determines whether objectives are achieved
• Improves coalition interventions
• Provides accountability to community, funding
  agencies stakeholders
• Increases community awareness support
• Contributes to scientific base
• Informs policy decisions

9
Barriers to Effective Coalition Evaluation

• Evaluation not built into planning process
• Costly of time resources
• Some changes occur slowly or do not last
• Difficult to distinguish between cause effect
• Evaluation not based on design do it yourself
  evaluation
• Coalition coordinators program managers
  motivated to make programs look effective
• Adapted from Solomon, 1987

10
Evaluation Standards

• Useful? Will amount/type of information collected
  meet needs of those who will use findings?
• Feasible? Will evaluation be practical
  realistic?
• Accurate? Will findings be correct?
• Fair? Will evaluation be conducted with awareness
  of rights of people involved in coalition or
  program?
• The Program Evaluation Standards How to Assess
  Evaluations of Educational Programs. 2nd Ed.
  Sage Publications, 1994

11
Six Steps to SuccessfulCoalition Evaluation

• Engage stakeholders
• Describe or plan coalition programs
• Focus the evaluation
• Gather credible evidence
• Justify conclusions
• Ensure use and share lessons learned
• Adapted from CDCs 6 steps to program
  evaluation

12
Step 1 Engage Coalition Stakeholders

• Include 4 groups of professionals lay persons
• Implementers - involved in program operation
• Partners - actively support program
• Participants - served/affected by program
• Decision-makers in position to make decisions
  about program
• Establish evaluation plan at onset
• Obtain buy-in commitment to evaluation

13
Step 2 Describe or Plan Coalition Program

• Stage of coalition/program development
• Planning or formation
• Implementation
• Maintenance
• Statement of problem
• What is nature and magnitude of problem?
• What are consequences of problem?
• What are causes and trends of problem?
• Create logic model

14
Program/Coalition Logic Models

• Describes core components of program or coalition
• Shows connection between program or coalition
  components expected outcomes
• Includes key information about context (i.e.,
  influencing factors)

15
Community Coalition Action Theory (Logic Model)
Community Context
Synergy
Operations Processes
Implementation of Strategies
Pooled Resources
Leadership Staffing
Lead Agency/ Convener Group
Community Change Outcomes
Member Engagement
Coalition Membership
Assessment Planning
Community Capacity
Health/ Social Outcomes
Structures
Institutionalization
Formation
Maintenance
16
Community Coalition Action Theory (Logic
Model) Propositions
Community Context

• Stages of Development
• Coalitions develop in specific stages and recycle
  through these stages as new members are
  recruited, plans are renewed, and/or new issues
  are added.
• At each stage, specific factors enhance coalition
  function and progression to the next stage.
• Community Context
• Coalitions are heavily influenced by contextual
  factors in the community throughout all stages of
  coalition development.

Institutionalization
Formation
Maintenance
17
Community Coalition Action Theory (Logic Model)
Propositions

• Pooled Member and External Resources
• The synergistic pooling of member and community
  resources prompts effective assessment, planning
  and implementation of strategies.
• Member Engagement
• Satisfied and committed members will participate
  more fully in the work of the coalition.
• Assessment and Planning
• Successful implementation of strategies is more
  likely when comprehensive assessment and planning
  occur.

18
Components of a Basic Logic Model
INPUTS Investments or resources (e.g., time,
staff, volunteers, money, materials)
INFLUENTIAL FACTORS Surrounding environments in
which the program exists (e.g., politics, other
initiatives, socioeconomic factors, staff
turnover, social norms and conditions, program
history, stage of development) that can affect
its success either positively or negatively
ACTIVITIES Events or actions (e.g., workshops,
curriculum development, training, social
marketing, special events, advocacy)
INTERMEDIATE OUTCOMES Medium-term results (e.g.,
behavior, normative, or policy changes)
INITIAL OUTCOMES Short-term effects of program
(e.g., knowledge, attitude, skill, and awareness
changes)
OUTPUTS Direct products of program (e.g., number
of people reached or sessions held)
LONG-TERM OUTCOMES Ultimate impact (e.g., social
or environmental change)
GOAL Mission or purpose of program
19
Step 3 Focus the Evaluation

• Ask stakeholders what they need to know about
• The coalition
• Its programs activities
• Short, intermediate long-term outcomes
• Coalition members program participants
• Impact of coalition programs on institutions
  community-at-large
• External factors that influence coalition or
  programs

20
Coalition Evaluation Questions

• Are we meeting our members needs?
• Do our work groups function well?
• Which strategies are effective?
• Are we on tract to accomplish goals objectives?
• Are we reaching our priority audiences?
• Are we influencing policies practices?
• Are we building community capacity?
• Could we improve? Are we making a difference?
• Are there external factors that affect our work?
• What unintended outcomes are occurring?

21
Step 4 Gather Credible Evidence

• Collect quality indicators that are well-defined,
  measurable, acceptable
• Dont try to measure everything - Specify use for
  indicators you measure!
• Choose indicators and sources of data
  (qualitative/quantitative) to assess different
  aspects of evaluation question
• Collect enough data to be reliable, but be
  realistic consider burden on respondents

22
Levels of Coalition Outcomes

• Process measures of coalition effectiveness
• Impacts from specific projects implemented by
  coalition
• Outcomes related to changes in community
  policies, practices environment (e.g.,
  community buy-in, linkages, coordination of
  services, empowerment, capacity-building,
  institutionalization of programs)
• Outcomes related to health status indicators
  (e.g., ED hospitalization rates related to
  asthma)

23
Short-term (process) 0utcome Measures

• of coalition meetings in past year
• members from priority population attending
  training
• of staff members certified in conflict
  resolutionmedia training
• Level of satisfaction of coalition members
• hours contributed by volunteers

24
Intermediate Outcome Measures

• Increased knowledge about chronic disease
• Improved cultural awareness of priority
  population
• Increased self-efficacy
• Improved skill in administering insulin
• Decreased absenteeism

25
Long-term Outcome Measures

• New indoor air policies, new enrollment
  procedures for health insurance
• New clinic referral practice, adoption of
  coalition program by community group, increase in
  community green spaces
• Decreased emergency department visits for
  unintentional injury, increased incidence of
  hospital admissions for diabetes, decreased rates
  of STDs, decreased cholesterol levels

26
Methods to Collect Data

• Written or telephone surveys
• Structured interviews
• Activity logs
• Document review minutes, records, rosters
• Focus groups
• Observation
• Existing (extant) data
• Physical or psychological measures

27
Coalition Evaluation Tools

• Coalition Effectiveness Inventory (CEI)
• Meeting Effectiveness Inventory (MEI)
• Plan Quality Index (PQI) State Plan Index (SPI)
• Member Satisfaction Surveys

28
Evaluation Designs

• Cross-sectional design
• Pre and post-test design (comparison groups when
  possible)
• Time-series design

29
Step 5 Justify Conclusions

• Analyze data
• Enter data check for errors
• Tabulate stratify data
• Make comparisons put in clear format
• Interpret results
• Are results as expected? Why or why not?
• How do they compare to other programs?
• Same results from multiple indicators?
• Limitations of evaluation?

30
Ensure Use Share Lessons Learned

• Recommendations Action-oriented, relevant
  useful
• Summarize Evaluation plan procedures,
  strengths/weaknesses of evaluation, pros/cons of
  recommendations, clear tables/graphs,
  stakeholders roles in evaluation/follow-up
• Channels Mailings, web sites, community forums,
  media, personal contacts, listservs, newsletters

31
Evaluation Resources

• Berkowitz, B. (2001). Studying the outcomes of
  community-based coalitions. American Journal of
  Community Psychology, 29(2), 213-227.
• Butterfoss, FD Francisco, VT. Evaluating
  community partnerships coalitions with
  practitioners in mind. Evaluation to Practice
  Department, Health Promotion Practice, in press,
  April 2004.
• The Community Toolbox - http//www.ctb.edu
• Empowerment Evaluation Website
  http//www.stanford.edu/empowermentevaluation
• Physical Activity Evaluation Handbook. (2002). US
  Dept of Health Human Services, CDC, 2002.