Comprehensive Centers Evaluation Meeting: Using Evaluation for Project Improvement

1
Comprehensive Centers Evaluation MeetingUsing
Evaluation for Project Improvement

• Nancy Loy
• U.S. Department of Education
• December 5, 2006

2
We heard you

• Hold a meeting with Comp Center Directors
  Evaluators sooner rather than later
• No more meetings on Mondays or Fridays
• Focus the Evaluation Meeting
• Clarify the Departments expectations concerning
  the Centers evaluations
• Give an update on the National Evaluation
• Work toward a shared understanding of Quality,
  Relevance, and Usefulness

3
ED requirements

• The U.S. Department of Education is not
  requiring Centers
• To use the same definitions, methods, or
  instruments, or
• To come to consensus on how to measure Quality,
  Relevance, and Usefulness.

4
Working toward a shared understanding

• Centers expressed interest in learning from each
  other and working toward a shared understanding.
• The Centers are in charge of the Quality,
  Relevance, and Usefulness sessions.
• As chair of the Performance Indicator Committee
  (PIC), Kit Peixotto will take the lead on the
  Quality, Relevance, and Usefulness sessions.

5
Summary of the sessions

• Kit Peixotto and PIC will prepare a summary
  that captures key points from the sessions on
  Quality, Relevance, and Usefulness to share with
  the Comprehensive Centers, the Department, and
  the National Evaluation team.

6
GPRA reporting

• The U.S. Department of Education will use the
  findings from the National Evaluation for the
  Comprehensive Centers Program as a whole for the
  GPRA report each year.

7
How will ED use Centers reports on Quality,
Relevance, and Usefulness?

• To assess whether Centers made substantial
  progress toward their goals and objectives for
  continuation grants
• To understand Centers work and manage and
  improve the Program
• To answer questions and explain the work and
  progress of the Centers and the Program to the
  Department, OMB, and Congress
• To help provide contextual information for GPRA
  and OMBs PART process

8
Centers evaluation reports similarities and
differences

• Ways in which evaluation reports addressed
  Quality, Relevance, and Usefulness
• Similarities e.g. using client surveys
• Differences e.g. setting targets
• ( of clients giving a high rating)

9
Some evaluators asked clients what criteria they
considered Quality

• Clients assessed Quality in terms of
• Reliability
• Organization and efficiency
• Adaptability to the priorities and schedules of
  the SEA staff

10
Some evaluators asked clients what criteria they
considered Relevance

• Clients assessed Relevance in terms of
• Whether technical assistance took into
  consideration State needs and SEA capacity to
  meet identified needs, especially concerning
  Federal requirements

11
Some evaluators asked clients what criteria they
considered Usefulness

• Clients assessed Usefulness in terms of
• Progress in achieving goals
• Capacity to expand networks of information and
  contacts, and to sustain change
• Consideration by the technical assistance
  providers of the priorities and limited time of
  the SEA staff

12
Quality, Relevance, and Usefulness context

• Quality, Relevance, and Usefulness of technical
  assistance services toward
• Achieving the Centers goals and objectives
• Building SEA capacity to provide sustained
  support to districts and schools as they
  implement NCLB reforms (Notice Inviting
  Applications)

13
Purpose of Centers evaluations

• Ongoing assessment and reflection for continuous
  project improvement
• What is working well?
• Where are there opportunities for improvement?
• Need for changes in how to carry out the work?
  Need for changes in objectives?
• Negotiate changes with SEAs and discuss with ED.

14
First Year evaluation reports

• Technical assistance services were limited and in
  an early stage.
• Some clients said that it was too early to judge
  whether the technical assistance services were
  useful.
• Comprehensive Center focus on building State
  capacity was new to clients.
• Some States with tight resources wanted technical
  assistance system to relieve them of some of the
  new functions under NCLB.

15
Issues from Centers evaluation reports staff
location

• How best to use Center staff to leverage and
  improve State capacity?
• Locate Center staff outside the SEAs?
• Locate Center staff inside the SEAs full-time?
• Keep the same model in Year 2 or change it?

16
Issues from Centers evaluation reports limited
resources

• How will Centers handle growing demands on their
  limited resources?
• Focus Center resources by targeting fewer
  priority needs to provide intensive and sustained
  technical assistance?
• Other solutions?

17
Issues from Centers evaluation reports limited
SEA capacity

• Some implementation lagged, because the SEA
  lacked the capacity to move forward and needed
  more basic technical assistance, including
  pre-requisite tools, policies, and procedures.
• How to deal with uneven capacity among States in
  the region?

18
Issues from Centers evaluation reports many TA
organizations

• How can a Center successfully gain recognition of
  its expertise, strengthen its identity as
  knowledge broker, and coordinate with other
  technical assistance organizations?
• How to reduce confusion and clarify the roles of
  the different technical assistance providers?