Using this Resource

1
Using this Resource

• Click the left mouse button or your space bar to
  progress to the next slide
• A printer-friendly version of this presentation
  will be available at the conclusion
• Please note The content of this presentation
  most closely applies to K-12 education in
  Pennsylvania. Other organizations and
  disciplines may have different explanations for
  some concepts.

2
Evaluation 101

• Yolanda Yugar
• Leslie Kirby
• Allegheny Intermediate Unit 3

3
Objectives

• Participants will
• Gain a basic understanding of evaluation.
• Understand the differences between evaluation,
  monitoring, research.
• Learn about evaluation roles (local, state,
  national).

4
What is Evaluation?

• Evaluation is the process by which organizations
  examine the implementation and outcomes of an
  intervention to determine whether or not (and to
  what extent) the intervention contributed to a
  change.

5
Understanding the Differences

• Monitoring
• Checklist, Review of Documentation
• Required by Funding Agency
• Separate from the Evaluation
• Research
• Uses research methods such as control group,
  well-matched comparison group, quasi-experimental
  design to determine effectiveness

6
Caution The Notion of Causation

• It is common to think about a program as causing
  a change. However, it is important to realize
  that programs do not occur in a vacuum. A single
  intervention usually cannot be the sole cause of
  an effect when dealing with people. Other
  programs, interventions, personal situations, and
  even chance contribute to the outcomes discovered
  during evaluation.

7
Evaluations are used to

• Demonstrate accountability
• Fulfill reporting requirements
• Assess needs
• Improve programs
• Determine relationship between cost and benefit
• Determine feasibility and replicability

8
Types of Evaluation

• Program implementation To what extent was the
  program implemented as designed?
• Program outcomes To what extent has the
  initiative influenced behaviors or practices?
• Program impact To what extent has the
  initiative influenced student achievement?

9
Formative Evaluation

• occurs while the intervention is in progress
  (usually during the developmental phase)
• allows management to improve program for current
  term
• concentrates on implementation and feedback from
  participants and/or program staff

10
Summative Evaluation

• compilation or summary of what occurred during
  the intervention or a particular term, the
  results of the intervention, and whether goals
  were achieved
• assesses overall impact

11
Mixed Method Approach

• Most evaluations use a combination of these forms
  of evaluation to gain a clear understanding of
  the program in order to give possible outcomes a
  context.
• Using more methods also allows the evaluator to
  examine a single idea or portion using more
  sources of data to strengthen interpretation
  (triangulation).

12
Types of Data

• Participant demographics
• Participant achievement
• Processes
• Perceptions
• (adapted from Victoria Bernhardt)

13
Quantitative Data

• Quantitative data is data that can be counted or
  measured.
• Includes
• Assessment/test (achievement) data
• Participation and attendance rates
• Forced-choice survey questions (i.e. select
  answer from a list)

14
Qualitative Data

• Qualitative data is data that cannot be easily
  counted. Instead, it is used to identify themes
  or trends.
• Includes
• Focus group responses and Interviews
• Open-ended survey questions
• Portfolios or compilations of materials
• Observations

15
Common Evaluator Tasks

• Help staff understand evaluation and how it can
  help a program plan and implement more effective
  improvement activities and make corrections in
  those improvement efforts when they are needed.
• Develop or help to develop the overall plan for
  evaluating the effort.
• Involve staff in the implementation of the plan
  and, if necessary, train them to carry out
  specific evaluation tasks.
• Help staff use specific data gathering
  methods/instruments in a reliable way.

16
Common Evaluator Tasks

• Build capacity to collect, store, analyze, and
  use the data to inform decision-making.
• Help staff interpret data collected and make
  decisions consistent with the results of specific
  analyses.
• Write reports that facilitate local use of the
  data collected in decision-making and that
  provide information.
• Serve as liaison.
• Help the local effort to achieve its purposes
  through the design and conduct of the evaluation.

17
Selecting an Evaluator

• You should select an evaluator who
• Reflects the needed characteristics or
  requirements
• Understands the organization he/she is evaluating
• Is willing to work collaboratively to identify or
  develop an evaluation that meets the requirements
  of the initiative and the needs of the
  organization

18
Characteristics of an Evaluator

• Understands confidentiality of student data.
• Ability to develop or find appropriate assessment
  instruments.
• Capacity to collect, manage, analyze, and
  interpret data and produce written reports based
  on that data.
• Capacity to guide districts in using data for
  decision-making.
• Understands of state and local needs.

19
Where would I find an evaluator?

• University or college
• Intermediate Unit
• Internal staff with specialized skills
• Independent contractor
• For-profit evaluation company

Your local evaluator may be one individual, or a
group of individuals, depending on the
organizations and projects needs.
20
How do I get started?

• The program and evaluator need to negotiate an
  agreement that sets forth the actual role and
  specific tasks for which the local evaluator will
  be responsible. The agreement would delineate
  how the evaluator will be compensated for his/her
  services.
• Rule of thumb for services is 8-10 of the total
  project budget depending on the scope of services.

21
Establishing an Evaluation Team

• The team would include representation from
• Organization staff
• Program implementers
• Evaluator(s)
• Other stakeholders (community partners, funding
  source, district support services, etc.)

22
Resources

• Bernhardt, V. (1998) Data Analysis for
  Comprehensive School Improvement
• Brainard, E. (1996) A Hands-on Guide to School
  Program Evaluation
• Wahlstrom, D. (2002) Using Data to Improve
  Student Achievement

23
Contact

• Allegheny Intermediate Unit
• 475 East Waterfront Drive
• Homestead, PA 15120
• [phone]
• www.aiu3.net/GEDT.aspx
• www.aiu3.net/evaluations.aspx