Evaluation

1
I N T E R N A T I O N A L E D U C A T I O N P R
O G R A M S S E R V I C E
Evaluation
CIBE, FLAS, IIPP, LRC, NRC Technical Assistance
Workshop February 2-3, 2009
Access to the World and Its Languages
www.ed.gov/ope/iegps
2
Title VI 50th Anniversary Update

• Madeline Albright is the Keynote speaker
• Please make separate reservations
• Conference registration
• Hotel registration
• Rooms are going quickly
• Contacts Rob Glew and Tomas Hult, Michigan State
  University
• http//titlevi50th.msu.edu

3
Evaluation

• Love

4
Panelists

• Eleanor Briscoe, Senior Budget Analyst
• John Clement, Director, Strategic Planning
  Services
• Susan Lehmann, Evaluation Analyst, FIPSE
• Karla Ver Bryck Block, Senior Advisor, IEPS

5
Why???

• ED believes in and supports high quality program
  evaluations to help inform budgetary and program
  decisions
• ED Strategic Plan
• Goal 3, Objective 1 Increase success in and
  completion of quality postsecondary education.
• Strategy 3. Prepare more graduates for employment
  in areas of vital interest to the United States,
  especially in critical-need languages,
  mathematics, and the sciences.

6
Administration Update

• Accountability and Transparency
• GPRA continues
• PART may or may not continue but the need to
  measure program effectiveness remains

7
Congress

• Clear interest in program evaluations
• Directed a 1.5 million review of Title VI and
  Fulbright-Hays programs by the National Research
  Council of the National Academies
• International Education and Foreign Language
  Keys to Securing Americas Future (March 2007)

8
Keys to Securing Americas Future

• National Academies recommendation
• The Department of Education should commission
  independent outcome and impact evaluations of all
  programs every 4 to 5 years

9
IEPS Program Level Initiatives

• Evaluations are underway with AIR
• Language Resource Centers
• Fulbright-Hays Group Projects Abroad Overseas
  Intensive Language Projects
• Analysis of the International Research and
  Studies Program is concluding
• National Resource Centers are next
• Future plans include Centers for International
  Business Education

10
IRIS

• IEPS gathers information via IRIS that is used
  for program evaluations
• Breakouts for 4 programs today (CIBE, LRC) and
  tomorrow (NRC, FLAS) to discuss
• Current and future program performance and
  efficiency measures
• Changes to IRIS for the next clearance
• Evaluation efforts
• Trivia, prizes, chocolate!!!

11
Keys to Securing Americas Future

• National Academies Recommendation
• The Department of Education should work with
  universities to create a system of continuous
  improvement for Title VI and Fulbright-Hays
  programs. The system would help develop
  performance indicators and other improvement
  tools and should include networks of similar
  centers and university officials with overall
  responsibilities in language, area, and
  international studies.

12
Your Project

• Think
• Goal-oriented
• Data driven
• ED does not have specific models, timelines,
  budgets, etc for your evaluation plans. Develop
  a plan that serves your needs.

13
Performance Objectives Measures

• A measurable indication used to determine how
  well goals are being met
• How will progress be measured?
• How much progress will constitute success?
• How will it be known if an objective or part of
  an objective has been achieved?
• Are we measuring process or outcome?

14
Objective Measure Types

• A specific achievement/accomplishment
• Change in a level of achievement/accomplishment
• Achievement/accomplishment (or change) compared
  with other students or institutions

15
Components of Objectives Measures

• What will change
• How much change is expected
• Who will achieve the change
• When the change will take place

16
Performance Objective Example

• Help local businesses to internationalize

• 20 (how much) of local businesses with more
  than 200 employees (who) will open an office in a
  foreign country or enter into a joint venture
  with a foreign company (what) within two years
  (when).

17
Performance Objective Example

• Teacher workshop
• Developing Materials for LCTLs

50 (how much) LCTL teaching professionals (who)
will complete LCTL workshop (what) annually
(when). 90 (how much) of LCTL workshop
participants (who) respond with 4 or higher on
survey (what) one year after completing program
(when).
18
Performance Measure Example

• Revise public health curriculum to enhance
  international knowledge

The required MPH courses (who) will be revised
(what) to cover the 10 important international
health areas (how much) by next fall (when).
19
Performance Measure Example

• Improve enrollment in summer language institute

Undergraduate student enrollment in the summer
language institute (who) will increase (what) by
five percent (how much) next summer (when).
20
Improving Performance Measures

• Original Performance Measure Provide overseas
  internships.
• What/Who
• Provide dynamic student internships with partners
  in a foreign country.
• When
• Provide dynamic student internships with partners
  in a foreign country each semester.
• How Much
• 20 students will engage in dynamic student
  internships with partners in a foreign country
  each semester.

21
Improving Performance Measures

• Original Performance Measure Enhance Business
  Resource Center.
• What/Who
• Enhance Business Resource Center to advise
  businesses on international trade regulations.
• When/How Much
• Business Resource Center documents and staff will
  be updated on new international trade regulations
  within one week of passage.

22
Remember!!!

• Activities are not performance measures!

23
Resolving the Activity-as-Measures Dilemma

• Create opportunities for local faculty to
  strengthen area skills. (activity)
• Create visiting faculty training programs and
  train 2 per year. (outcome)
• Enhance study abroad. (activity)
• Increase number of students studying abroad by X
  each year. (outcome)

24
Summary Good Objectives Measures

• Each objective should have one or a few specific
  performance measures to demonstrate how progress
  will be measured toward meeting the objective
• More objectives and measures ? better objectives
  and measures!
• Ensure data is collectible
• Quantify and define success

25
Summary Strong Evaluation

• Partner with a competent evaluator
• Develop quantifiable objectives and measures
• Measure what is important
• Measure what can be changed
• Get frequent progress reports
• Keep key stakeholders informed of results
• Use results to adjust project regularly
• Keep analyzing