TAH Project Evaluation

1
TAH Project Evaluation

• Great - Yet Realistic - Expectations
• Jeff Sun, Sun Associates

2
Our Background

• Initiative-wide Qualitative Evaluation
• District-level initiatives
• State-level initiatives
• Strategic Planning
• STEM, Science, Instructional Technology, Social
  Studies, New Media
• Evaluation of Teacher Professional Development
• 10 TAH projects (7 of these are current)

3
Qualitative is the Emphasis

• Makes the connection between what projects say
  they want to do and what participants say that
  projects have done.
• This data is invariably qualitative and accounts
  for a wide range of effects
• Quantitative data helps paint the picture in
  broad outlines...
• ...but qualitative data helps you understand what
  the quantitative data means and to use the
  evaluation strategically

4
This Presentation

• We want to talk about how to
• Create a good evaluation
• Use evaluation strategically
• Work effectively with evaluators

5
Four Functions for Evaluation

• Evaluation performs four meaningful functions in
  a projects life...
• As a part of the proposal
• To clarifying a projects work
• To provide formative assessment of Progress and
  Implementation
• Summatively, to account for the projects
  success (or lack thereof)

6
First Function - The Proposal

• As expressed in the logic map, evaluation helps
  explain
• Need for the project
• The projects actions in response to those needs
• Who the project will serve
• What the project expects to produce
• All in one tidy package in the proposal text!

7
Issues Related to Function 1

• We need time to develop the evaluation plan
• Ideally, we come in when the basic work is set,
  but the details are not fully fixed
• We need space to fully describe the evaluation
  process
• Its hard to create this plan in 2 paragraphs or
  a half page
• We need connect with those who will be actually
  implementing the project if funded
• Proposal-writers will help you get funded...but
  its often no coincidence that theyre nowhere to
  be found once you get the grant.

8
Second Function - Clarification

• Once youre funded, the real work begins
• Use the evaluation plan as an organizer for
  talking with partners about
• what the project will be doing
• expectations for partners and participants
• This discussion clarifies goals and develops a
  shared vision for project success
• Out of this discussion come the projects
  performance indicators...which we organize into
  rubrics
• It would be nice to have all of your indicators
  before funding, as part of the proposal
• But realistically, theres never time for that
• Projects need their own indicators
• EDs indicators are not the same and cannot
  replace your own indicators

9
Issues Related to Function 2

• We cant help if we arent at the table
• Often, we have to really bug projects to bring us
  on board early
• Evaluation is too often seen as the thing that
  happens after the work is done...not as the thing
  that can get work done.
• We read the proposal and base the actual
  evaluation on what is written
• This can be a problem when the proposal was
  written by the now-absent proposal-writer
• The clarification discussion - assuming it
  happens early enough - is where this problem is
  resolved

10
Third Function - Formative

• Collecting the data and sharing it as the project
  progresses
• Measuring what the project is doing against what
  it said that it would do
• Using indicator rubrics to account for data that
  is highly qualitative

11
Issues Related to Function 3

• Our challenge is to identify data collection
  points that get at the essence of the project and
  its impact.
• This can be one of the most important parts of
  our work.
• Questions continue the process of clarifying the
  project work through participant feedback on
  impact
• Next most difficult is connecting directly with
  participants

12
Fourth Function - Summative

• Annual comparison of progress to goals
• Sometimes more reporting than annual is
  important
• More reporting might be useful in the beginning,
  less reporting might be necessary as the project
  matures
• Summative data should play a role in the
  development of new proposals

13
Issues Related to Function 4

• Qualitative data is valuable for projects, but
  does not seem to be something that ED is
  interested in.
• This is a problem because projects cannot really
  produce meaningful data that ED is interested in.
• So, do we spend extra time (money) producing two
  sets of data?
• Or do we sacrifice the meaningful data to
  concentrate on producing what were told to
  create?
• This has been the on-going debate among TAH
  evaluators on a national level

14
Questions? Comments?

• What challenges have you faced when considering
  project evaluation?
• How have you used evaluation in different stages
  of your project? (from planning to reporting)
• jsun_at_sun-associates.com or 978-251-1600 x 204

15
(No Transcript)