Jennifer Nahas, Brigham Nahas Research Associates jenniferbnra'net

1
Jennifer Nahas, Brigham Nahas Research
Associatesjennifer_at_bnra.net
Sustainability Tips and Tools for
Maintaining/Growing Your Programs
2
Agenda

• Why us? Who is BNRA?
• Why program evaluation is intimately linked to
  long term sustainability
• Developing an Evaluation Master Plan
•  

3
Brigham Nahas Research Associates

• Is a small Boston-based research and evaluation
  firm
• Has worked across a variety of programs
• Uses a variety of evaluation approaches, all to
  support action-oriented and collaborative
  research
• Most of our work has been with college access and
  retention programs.

•  

4
How Sustainability and Evaluation Intersect

• Long term sustainability requires stable and
  diverse funding to support the work.
• To attract funding, programs must make a strong
  case for investment.
• Programs must articulate precisely how what they
  do results in a positive impact on youth,
  families, schools, and the larger community.
• Understanding and articulating impact is the
  basis of a programs evaluation master plan.
•  

5
Evaluation Master Planning

• Starts with a philosophical commitment from
  program leaders and an authentic process that
  involves program staff.
• Requires backward mappingstarting with what you
  hope to achieve.
• Results in an evaluation strategic plan which is
  phased in over time. The plan
• Utilizes multiple evaluation strategies
• Is managed by skilled staff/ consultants
• Generates data that is central in decision making
  and tells you story to the external community

6
Evaluation Master PlanningStart by articulating
why you do what you do

• Articulate succinctly, why it is you do what you
  do. Its the theory behind your program model.
• This requires hanging what you do on sound
  research and principles, and answering why you
  believe this model is best to address challenges.
• Results in a statement or headline about what you
  do that is easy to understand, guides decision
  making and informs the external world.

7
Program Statements/Headlines are tricky!

• Our program prepares students what kind of
  students to academically and socially succeed
  succeed? or graduate? in school high quality
  schools? selective schools?.

8
Phase One ElementsCreating the master plan

• Developing/fine-tuning the logic model (process)
• Conducting a data audit at the program
  (process)
• Conducting satisfaction surveys/focus groups of
  primary stakeholders (both)
• Students and families
• Host institutions
• Analyzing existing data from a secondary source
  (outcome)
• Analysis of academic growth through existing, off
  the shelf pre/post tests that schools administer.
  
• Engaging in an Implementation Analysis (process)

9
Phase Two ElementsImplementing the Master Plan

• Instituting internal systems for ongoing data
  gathering and analysis
• In-depth case study to document the model
• Retrospective follow up study with students that
  have graduated from program services
• Evidencing change through secondary sources such
  as high school teachers and family
• Benchmarking student change in grades, retention,
  safety, and drug use against local, state,
  national indicators
• Developing instruments that capture student
  change in perspective, attitude and behavior.
• Exploring a comparison evaluation

10
Phase Three ElementsMoving to Large Scale
Evaluation Work

• Rigorous evaluation models such as comparison
  designs
• Economic modeling for value-added and
  cost-effectiveness analysis
• Community impact

11
Closing comments

• Evaluation as a means towards sustainability is a
  philosophical approach.
• Well worth the investment of having a clear,
  aligned strategy that can evidence your programs
  effectiveness and client outcomes.