Evaluating the Undergraduate Research Experience

1
Evaluating the Undergraduate Research Experience

• A Primer Introduction
• Connie Kubo Della-Piana, Ph.D.
• National Science Foundation
• Views expressed during the presentation are those
  of the author and does not represent the National
  Science Foundation

2
Before You Start

• Why should the evaluation be carried out?
• What are the issues that need to be addressed?
• What resources are available for conducting the
  evaluation?
• What is the anticipated cost to conduct the
  evaluation?
• What expertise/experience is required to complete
  the evaluation?
• Who is responsible to whom for what?
• What is the time frame for completion?

3
Evaluation is

• the systematic collection of information about
  program activities, characteristics, and outcomes
  for
• Making judgments about the program
• Improving program effectiveness
• Informing decisions about future programs

4
A good project evaluation should

• Examine how the project functions within the
  economic, social, and political environment of
  its setting and larger community.
• Help with the planning, setting up, improving,
  and carrying out the project, as well as
  documenting of its evolution.
• Assess the effectiveness, short-comings, and
  short- and long-term results of the project.

5
Focus of the Evaluation

• New Program?
• Established Program?
• Piloting Future Programs?

6
Purpose

• Improvement
• Understanding/Knowledge
• Accountability/Worth

7
Key Questions

• What do I want to know about my program?
• What would others want to know about my program?

8
Essential Elements

• Description
• Plan
• Implementation
• Use

9
Iterative Process

• Describing the program
• Target population, need, goals, activities,
  expectations
• Planning the evaluation
• Questions, indicators, measures, methods
• Implementing the evaluation
• Data collection, analyses, findings
• Utilizing the information
• Worth, improvement, knowledge

10
Project Description Template

• Context
• What are the needs of the target population(s)?
• What are the political, economic, and social
  realities? Their potential for assets and
  obstacles?
• What are the resources available to support the
  project?
• Goals/Objectives
• What is the purpose of the project?
• What are the expected accomplishments?
• Activities
• What are the projects core activities?
• What are the essential elements and processes of
  the core activities?
• How will activities make possible project
  accomplishments (expectations)?
• Expectations
• How will the target population(s) change?
• How will other participants and/or institutions
  change?

11
Evaluation Planning Matrix
12
Implementation Template
13
Tools and Techniques
14
Utilization Guidelines

• Who will use the findings? Who will see the
  findings?
• How will they be used?
• Are the findings credible?
• What is the level of commitment to implement
  recommendations?

15
Selecting an Evaluator

• Finding an Evaluator
• Departments of education, educational psychology,
  psychology, administration or sociology
• Campus teaching and learning centers
• Colleagues and researchers
• Professional organizations
• Independent consultants.

• Selecting an Evaluator
• Specialized training
• Previous evaluation experience (relevance, match)
• Professional orientation (philosophical/methodolog
  ical)
• Previous performance
• Personal style and characteristics

16
Elements of an Agreement

• Description of the parties to the agreement
• Description of the project/program
• Description of the evaluation
• Reporting requirements
• Description of mechanisms to protect participants
• Ownership of the data
• Provisions for the unexpected
• Budget and timelines
• Signature and dates

17
Contact Information

• Connie Kubo Della-Piana, PhD
• Program Evaluation Manager
• Office of Integrative Activities
• cdellapi_at_nsf.gov