'We all have different ideas of what success will look like

1
CFI II Evaluation Design
Prepared for

• Irvine Foundation

February 27, 2007
2
Advisory Team Meetings Completed

• Morning of October 10th (In Person) Evaluation
  Advisory Team reviewed overarching priorities,
  roles, and timeline and provided input on several
  design questions
• October 10th (Convening) Discussion of
  evaluation priorities, timeline, and roles with
  entire CFI II at convening. Convening attendees
  worked together to answer three questions
• Looking ahead three years, what would success
  look like for this Initiative?
• How will we know that weve achieved success
  (what evidence will we want to have in hand)?
• What advice do you have for FSG in designing and
  carrying out the evaluation of this project
• November 13th (In Person) Evaluation Advisory
  Team reviewed the preliminary evaluation design -
  Indicators of progress toward goals for community
  foundations and CFI II as a whole, data
  collection plan, and tools for data collection
• December 1st (Conference Call) Goals,
  indicators, and data collection plan finalized
  with Evaluation Advisory Team

3
Agenda

• Evaluation Purpose and Design Priorities
• Questions and Definitions of Success
• Process and Workplan

4

The Irvine Foundation Focuses on Using Evaluation
to Refine Program Strategy, Strengthen Grantee
Capacity, and Share Knowledge

• Evaluation should inform program decisions and be
  incorporated into programs for internal
  decision-making and planning processes
• Evaluation should measure progress toward program
  outcomes, as well as create better understanding
  about the organizational and program processes
  necessary to achieve those outcomes

1. Refine Program Strategy

• Evaluation should be participatory and
  collaborative whenever appropriate, involving
  grantees and other stakeholders in planning and
  implementation
• Evaluation should be pragmatic, flexible, and
  dynamic, reflecting the different needs and
  priorities of a wide range of programs and
  audiences

2. Strengthen Grantees Capacity for Success
3. Generate New Knowledge and Learning for the
Field

• Evaluation should include dissemination of both
  achievements and shortcomings, with the goal of
  improving the understanding of practitioners

Source The James Irvine Foundation
5
Evaluation Begins with the Irvine Foundations
Defined Goals for CFI II
Develop philanthropic resources in regions of the
state that have been traditionally underserved by
organized philanthropy
Accelerate the growth and development of a subset
of Californias community foundations, helping
them to function as more robust local
philanthropic vehicles and create lasting,
permanent resources for each community
Support a visible and strategic community project
which will raise awareness of the community
foundations work and increase credibility among
stakeholders
Provide resources that effectively develop
internal capacity, particularly in the areas of
governance, sustainability, and communications
visibility

• For each participating community foundation, CFI
  II expects that it will
• Build its asset base and cultivate new donors.
• Improve its capability to engage in high-quality,
  community-based grantmaking.
• Enhance its standing in the community, and its
  ability to mobilize community resources and help
  diverse groups come together to define community
  issues and search for solutions.

Source The James Irvine Foundation
6
Irvine Staff Provided Input on Several Design
Questions Which Guide FSGs Evaluation Design

• Emphasis on internal community foundation growth
  and development

• Emphasis on external changes resulting from
  community foundation efforts

• Use organizational development benchmarks tied to
  National Standards

• Define a narrower range of organizational
  development benchmarks specifically for CFI II

• Provide support to Community Foundation grantees
  in developing their own independent evaluation
  capacity

• Concentrate efforts to support each community
  foundation on providing site reports and
  reflection opportunities

• Share all evaluation findings and materials with
  Irvine staff - FOR POSITIVE EXAMPLES AND SUCCESS
  STORIES

• Share only aggregate findings and descriptive
  profile data with Irvine, not each individual CF
  site report

• Ask each community foundation to participate in
  data gathering

• Limit new data gathering as much as possible
  outside of existing reporting requirements and
  what FSG can do on its own

7
The Evaluation Advisory Team Provided Key Input
on Several Additional Design Questions
Including Board Involvement

• Engage each community foundations Board in the
  evaluation process and annual review of findings

• Provide the option of Board involvement, but
  limit time and involvement requirements as much
  as possible

• Concentrate data gathering efforts during each
  year of the initiative in a single timeframe, to
  minimize the number of requests for information

• Collect data at several different intervals
  during the year, to minimize the effort needed at
  any given time

• Share site reports through formal, in-person
  meetings that involve Board members

• Share site reports more informally, with a
  conference call to discuss conclusions and
  reactions

• Design site reports to be as simple and efficient
  as possible

• Design site reports to be as rich and detailed as
  possible

As a result of Advisory Team guidance, the site
reports will actually be facilitated annual
Board discussions of progress, learnings, and
future priorities
8
Agenda

• Evaluation Purpose and Design Priorities
• Questions and Definitions of Success
• Process and Workplan

9
Several Questions Have Been Defined to Focus
Learning from the Evaluation Process
Questions About Initiative Implementation
Questions Relevant for the Field

• What types of changes take place within each of
  the nine participating CFs and in CFI II overall?
  What types of success are achieved?
• Does the mix of resources and types of assistance
  provided help the community foundations achieve
  the objectives of the initiative and what the
  community foundations define as success? What
  are the barriers?

• What are the different paths of development,
  drivers of growth, and organizational design
  choices that lead to success for todays emerging
  community foundations?
• How do emerging CFs balance the development of
  core capabilities, recognizing that each cannot
  fully mature at the same time
• Cultivation and engagement of current donors
• Development of new funds
• Grantmaking and community leadership
• How does where each CF places emphasis shape the
  kind of organization it becomes?
• How are Board roles different for emerging
  community foundations, particularly in defining
  priorities and opportunities for the foundation?

10
CFI II Participants Have Defined Success in Many
Important Ways Each Should Be Monitored to
Understand Progress and Learnings
A Deeply Involved Board
Clarity of Purpose and Role (Internal)

• Energizing and developing the leadership of the
  Board to act in full partnership with the ED/CEO
• Ensuring the ability to of the Board to
  confidently articulate the CF identity,
  responsibilities, and contribution
• This is our community foundation This is the
  Board role This is what we should be doing to
  have impact.

• Developing a vision of how the community
  foundation plays a unique role in its region
• We will have a vision of how the county or
  region will change because the community
  foundation exists and is successful
• Creating a common understanding of CF products
  and processes and the ability to define who the
  CF serves and why

Visibility in the Community (External)
Community-Based Grantmaking

• Creating a basic presence and identity
• Establishing brand (e.g., clear and consistent
  identity, core messages, stories, donor database,
  etc)
• Spreading the word and using proactive
  communications techniques (e.g., leverage
  contacts, communicate impact)
• Being sought out by potential donors, Board
  members, and community partners

• Increasing expertise in community-based
  grantmaking, including processes that involve
  community members and/or donors in setting
  priorities and making decisions
• Originating a robust set of grants that help the
  community foundation make new connections and
  raise its profile among different stakeholders

Within each area, CFs will prioritize aspects of
success differently We all have different
ideas of what success will look like
11
CFI II Participants Have Defined Success in Many
Important Ways Each Should Be Monitored to
Understand Progress and Learnings
Proactive Approach to Sustainability Needs
Growth and Donor Development within Priority
Areas

• Recognizing that decisions made today will
  dramatically change the foundations future asset
  composition and economics
• Setting policies and processes to improve the
  organizations economics and operate more
  cost-effectively (e.g., pricing, policies)
• Focusing on the ability to maintain staffing
  beyond Irvine grants

• Prioritizing the types of growth the foundation
  wants to support in terms of kinds of donors
  and fund or product types
• For each foundation, the pure dollar value of
  assets may not be the primary measure of growth
• Growing the foundations revenue base to support
  donors, make grants, and provide community
  leadership

Expanded Networks and Relationships
Learning Organization

• Forming new types of relationships
• In the community, to define issues and work
  toward solutions
• In CFI II, to work together as a cohort, create a
  productive learning community
• In the broader field philanthropy, to learn from
  peers, form partnerships for increased impact,
  and/or find economies of scale in collaboration

• Institutionalizing core values and building on
  strategic thinking in an ongoing way, even as the
  organization and its leadership change
• Defining new possibilities for community impact
  as each foundation learns more about its
  organizational priorities and its community

12
Each Type of Success Will Be Assessed by
Collecting a Set of Indicators and Other Data
that Together Document the CFs Development
13
Each Type of Success Will Be Assessed by
Collecting a Set of Indicators and Other Data
that Together Document the CFs Development
14
Each Type of Success Will Be Assessed by
Collecting a Set of Indicators and Other Data
that Together Document the CFs Development
15
Each Type of Success Will Be Assessed by
Collecting a Set of Indicators and Other Data
that Together Document the CFs Development
16
Agenda

• Evaluation Purpose and Design Priorities
• Questions and Definitions of Success
• Process and Workplan

17

Everyone Will Have a Role in the CFI II
Evaluation Effort
IRV CFI II Evaluation Participants
FSG as Evaluator

• Incorporate perspectives from inside and outside
  the CFI II effort to guide evaluation design
• Design evaluation process
• Develop and execute data collection including
  interviews, brief Board surveys, and other basic
  facts
• Facilitate annual Board discussion of progress,
  learnings, and plans for the next year
• Develop annual interim cohort-wide reports to
  share with Irvine and all CFs at convenings
• Prepare a final synthesis document at the end of
  CFI II highlighting its results and learnings and
  support dissemination

Irvine Foundation Staff
Technical Assistance Consultant Teams
Participating Community Foundations
Evaluation Advisory Team

• Guide overall evaluation design
• Review results
• Respond to learnings

• Guide evaluation design, choice of metrics, and
  collection process
• Provide feedback on annual and final reports

• Share information, as appropriate, to avoid
  duplication of effort
• Participate in data collection and offer candid
  assessments in annual interviews

• Participate in data collection and offer candid
  assessments in annual interviews
• Participate in annual Board discussion and
  convening discussions

Source FSG
18

The Data Collection Process Will Gather
Information Annually and Provide Reflection
Opportunities for All CFI II Participants
Source The James Irvine Foundation
19
Executive Directors and a Subset of Board Members
Will Be Asked to Provide Insights into Each
Community Foundations Experiences
1 Hour Phone Interview

• What kinds of changes have taken place within
  your community foundation over the last year?
• What opportunities and challenges do you hope to
  take on in the coming year? In what critical
  areas do you need to make progress to achieve
  your goals?
• What are your observations about your
  foundations participation in the Irvine
  Initiative (CFI II)?

DETAILED INTERVIEW QUESTIONS IN ATTACHED DOCUMENT
20
Annual Reporting Data Requested Will Cover Basic
Facts Focused on Growth and Organizational
Development
EACH WILL BE COLLECTED AS AVAILABLE
2 Page Summary Major Events, Decisions, and
Reflections
Grant Guidelines, Application Materials, RFPs By
type (e.g., DAF, Regranting, Co-Funding)
Operating Budget (Costs and Revenues by Source)
Including CFI II Grant Expenditures
Summary of Grants Made
File with Fund Information as of End of Fiscal
Year
Packet of Any New Marketing Materials
VisibilityMeasures
Note Many CFs will not have new marketing
materials or visibility measures until after work
with Williams Group
21
Each Board and Staff Member Will Have a Voice in
Sharing Thoughts about Organizational
Development, Achievements, and Future Priorities
10-20 Minute Online Survey (With Paper-Based
Alternative)
Q1 Respond Yes or No to whether each
area has been a organizational development
priority Q2 Rank each area in terms of
organizational development priority for the
past year
Q3 Rate progress in each area on a scale of
1-7 Q4 Why? (optional open-ended question)
Q5 Rate theimportance of progress in each
area going forward on a scale of 1-7
THE PRIMARY PURPOSE OF THIS SURVEY IS TO INFORM
ANNUAL BOARD DISCUSSIONS WHICH SERVE AS A DATA
SOURCE FOR THE EVALUATION
22

FSG Will Do an Annual Site Visit to Each CF and
Facilitate a Discussion Focused on Learning and
Planning for the Future
1
3

• FSG will share synthesis of evaluation
  information specific to each CF

• Board members and staff will be asked to identify
  annual organizational development priorities

2

• FSG will facilitate discussion among Board
  members and staff to reflect on examples of
  success, challenges, and learnings

Discussions at each CF will be a combination of
sharing evaluation insights and interactive
conversation with Board and staff
23

FSG Will Also Work With the Consultant Teams on
an Ongoing Basis to Understand the Types of
Assistance Received, Progress, and Challenges
24
Timeline Annual Data Collection Process
Annual Evaluation Cycle
Repeated Annually 2007-2010

• Data Collection Process
• Collect annual data through Irvine reporting
  process
• Track technical assistance activities through
  monthly consultant calls
• Interview consultant teams
• Conduct annual phone interviews with board and
  staff members at each CF
• Administer survey
• Synthesize info for each CF
• Visit each CF and facilitate a discussion of
  progress, learnings, and next steps
• Facilitate a cohort-wide discussion of progress,
  learnings, and next steps
• Write a report synthesizing results and learnings

Apr 1st
Irvine Staff Meetings
Advisory Group Meetings
25
The FSG Team Will Be Expanded to Address the Full
Range of Evaluation Activities

• Becca and Eva will continue to be involved in the
  process in managing the overall evaluation,
  designing data collection methods, meeting with
  individual foundations and the cohort, and
  synthesizing evaluation information
• An additional team member who has not been
  involved with the sustainability planning work
  will join the team to conduct the interviews that
  inform the evaluation. Our hope is that you will
  be extremely candid in the evaluation process
  about FSGs work as a technical assistance
  provider
• Fay Hanleybrown, Mark Kramer, and John Kania
  Managing Directors at FSG will oversee the
  evaluation and participate in the process of
  identifying the most important leanings for the
  field

26
Products from the Evaluation Will Be Designed for
Different Audiences Some Will Be Shared Only
with Individual Foundations and Some Only with
Irvine
Ultimately, the evaluation will also produce
reports to share learning with the field
27
Evaluation Will Be a Part of the Initiative
Through 2010
Evaluation Timeline and Milestones

• Evaluation Design
• Determine overall approach
• Choose meaningful indicators of success
• Develop data collection methods and timing

• Final Synthesis
• Collect 2010 information
• Synthesize 2007-2010 information
• Prepare a final CFI II report
• Disseminate among cohort and to external audiences

• Evaluation Implementation
• Collect information and learnings relevant to
  evaluation effort
• Prepare individual reports and facilitate
  discussions about progress at each community
  foundation
• Prepare and present annual cohort-wide reports to
  Irvine and at a CFI II convening