Agenda

1
Agenda
Introduction Benchmarks Benchmarking Survey
data and benchmarking
2
Benchmarks

• Benchmark
• Used to establish industry standards based on
  external and internal comparisons
• Comparisons to similar institutions establish
  benchmarks
• CCSSE measures what students are doing

3
Benchmarks
An example from the corporate world An interview
with Jeff Immelt, CEO of General Electric,
identified Toyota, Dell, and Procter Gamble as
the three companies that GE has benchmarked the
most GE looks at Toyota and Dell to learn from
their process excellence GE looks at Procter
Gamble to learn from their marketing and
commercial excellence
4
Benchmarks
Opportunities abound in educational
research Corporate benchmarking tends to be
sharing between partners Corporate benchmarking
limited to a few competitors in contrast to
education that has a much larger universe with
which to compare Are these approaches really that
different? Should institutions attempt to find
similar or different partners for benchmarking?
5
Benchmarks

• What are the most important characteristics of a
  benchmark?

6
Benchmarks
Elements Credibilityreliability and
validity Comparativeresults examined relative to
peers Comprehensivemeasures key elements
according to the experts Performance
Importancewhat has the greatest
impact Confidentialityit takes courage to assess
oneself Continuousit takes time for
improvement -Joseph A. Pica, CEO of Educational
Benchmarking Inc. in About Campus
7
Benchmarks

• The CCSSE survey
• is administered directly to community college
  students at CCSSE member colleges in randomly
  selected classes.
• is based on research, asking questions about
  institutional practices and student behaviors
  demonstrated to promote student learning and
  retention.
• uses a sampling methodology that is consistent
  across all participating colleges.

8
Benchmarks

• The five CCSSE benchmarks
• Active and Collaborative Learning
• Student-Faculty Interaction
• Academic Challenge
• Support for Learners
• Student Effort

9
Benchmarks

• Active and Collaborative Learning
• Worked with other students on projects during
  class
• Worked with classmates outside of class to
  prepare class assignments
• Tutored or taught other students (paid or
  voluntary)
• Participated in a community-based project as a
  part of a regular course
• Made a class presentation
• Asked questions in class or contributed to class
  discussions.
• Discussed ideas from your readings or classes
  with others outside of class (students, family
  members, co-workers, etc.)

10
Criterion Benchmarking
How do you determine which measure you use to
compare yourself with other institutions?
11
Benchmarking

• Normative
• Compare your college with the mean
• Criterion
• Compare your college with a predetermined value

12
Benchmarking

• Normative Benchmarks Provide context
• Determine what the mean you would like to be
  compared with is
• Normative Benchmarks Situate Your Results
• What does it mean to have 80 of your students
  satisfied?
• A good place to start, but not necessarily the
  end point

13
Benchmarking

• Normative Benchmarking with CCSSE
• Look for Differences of 5 points (a standardized
  effect size of .2)
• Is .2 a noteworthy difference?

14
Criterion Benchmarking

• Criterion Benchmarking with CCSSE
• What is the college mission?
• What are the colleges accreditation goals?
• Are all students equally engaged?

15
Benchmarking

• Five ways to colleges can reach for excellence
  using CCSSE Benchmarks
• Compare themselves to national average
• Compare themselves to high-performing colleges
• Measure their performance against their
  least-engaged group
• Gauge work in areas most strongly valued
• Compare now to where they want to be

16
Benchmarking
Comparisons yourself with high-performers
17
Benchmarking

• Measure performance against least-engaged group
• Breakout by race, gender, enrollment status,
  parental education, traditional vs.
  non-traditional age
• At risk students vs. other students
• Define at-risk at your college

18
Benchmarking

• Density Curves of Student v. Institutions
  benchmarks (within institution v. between
  institution variation)

19
Benchmarking

• Gauge work in areas most strongly valued
• Focus On Current College Initiatives
• Sharing thoughts about how to use CCSSE data to
  evaluate different programs
• Examine institutional mission, vision, and values

20
Benchmarking
Understand values by sharing results Share
results with others to determine what is most
strongly valued Faculty, students, and
administrators will likely have different
opinions on what it is that accounts for
particular phenomena Hopefully, the questions
that are created from benchmarks are more focused
questions than the original question
21
Benchmarking
Compare now to where you want to be
22
Survey Data as Benchmarks

• Should benchmarks derived from surveys be used to
  rank colleges?
• Are there fundamental differences in how census
  benchmarks (e.g., graduation rates) and
  benchmarks derived from surveys should be used?

23
Survey Data as Benchmarks

• Ewells distinguishes between hard statistics
  and second-order statistics
• Hard statistics are clearly enumerated and
  based on census-type data, such as numbers of
  students, graduates, and degrees awarded
• Second-order statistics measure phenomena that
  cannot be directly counted, such as student
  satisfaction and students self-assessments of
  their behavior, and as such contain some
  statistical instability
• Hard statistics are preferable for performance
  funding because they are more statistically
  stable than second-order statistics
• Source Ewell, P. T. (1999). Linking performance
  measures to resource allocation Exploring
  unmapped terrain. Quality in Higher Education, 5
  (3), 191-209.

24
Survey Data as Benchmarks
Texas Community Colleges Performance
Measures 1.The rate at which students completed
courses attempted. 2. The number and types of
degrees and certificates awarded. 3. The
percentage of graduates who passed licensing
exams related to the degree or certificate
awarded, to the extent the information can be
determined. 4. The number of students or
graduates who transfer to or are admitted to a
public university. 5. The passing rates for
students required to be tested under the Section
51.306. 6. The percentage of students enrolled
who are academically disadvantaged. 7. The
percentage of students enrolled who are
economically disadvantaged. 8. The racial and
ethnic composition of the districts student
body. 9. The percentage of students contact hours
taught by full-time faculty. Source
http//www.thecb.state.tx.us/reports/DOC/1197.DOC
25
Survey Data as Benchmarks
Texas State-Level Benchmarks for Higher
Education Percent of recent high school
graduates enrolled in a Texas public college or
university Percent of first-time, full-time
freshmen returning after one academic year
Percent of first-time, full-time freshmen who
graduate within four years Percent of
first-time, full-time freshmen who graduate
within six years Percent of two-year college
students who transfer to four-year institutions
Percent of two-year transfer students who
graduate from four-year institutions Percent of
population age 24 and older with
vocational/technical certificate as highest level
of educational attainment Percent of population
age 24 and older with two-year college degree as
highest level of educational attainment Source
http//www.thecb.state.tx.us/reports/DOC/1197.DOC
26
Survey Data as Benchmarks

• If performance funding is based on verifiable
  hard statistics, what role does survey data have
  in benchmarking?
• Or, why should we concern ourselves with the
  student experience?
• Input and outcome versus process measures
• To achieve outcomes, we need to understand the
  process by which they are obtained

27
Survey Data as Benchmarks
We can tell people almost anything about
education except how well students are learning.
Patrick M. Callan, President, National Center
for Public Policy and Higher Education
28
Survey Data as Benchmarks

• Input -gt Process -gt Outcome Model
• Inputs include costs, numbers admitted, etc.
• Outputs include graduation rates, retention,
  graduate satisfaction
• How do we measure the Process component?

29
Survey Data as Benchmarks

• Input and Ranking
• Inputs are heavily emphasized in media rankings
  and potentially serve to maintain an
  establishment
• Inputs and outputs are naturally correlated
• The challenge for institutions is to maximize
  process to improve on the ability of inputs to
  predict outputs

30
Survey Data as Benchmarks

• Input -gt Process -gt Outcome Model
• Inputs include costs, numbers admitted, etc.
• Outputs include graduation rates, retention,
  graduate satisfaction
• How do we measure the Process component?

31
Survey Data as Benchmarks

• Is there a danger of impacting results by raising
  the stakes?
• Increasing the outcome without increasing the
  process
• To achieve outcomes, we need to understand the
  process by which they are obtained
• Increasing the outcome may not reflect
  improvement, but increasing the process wont
  hurt

32
Survey Data as Benchmarks

• CCSSE does not rank
• There is not a single criteria or set of criteria
  that can be used universally
• Institutional characteristics matter
• Institutional missions differ
• Benchmarking with CCSSE data is best when
  presented in a non-threatening manner
• Improvement requires an understanding of the
  process
• Understanding the process in an institution will
  require hearing different voices and different
  perspectives on the same issues

33
Survey Data as Benchmarks

• Stability of CCSSE Benchmarks
• Correlate 2005 and 2006 benchmarks for colleges
  that participated both years
• 45 institutions
• 55,903 students

34
Survey Data as Benchmarks
35
Summary

• Summary
• Survey results are second order data not ideal
  for performance funding and
• Understanding hard statistics naturally leads
  to a discussion of processes.
• Survey data present an opportunity to understand
  processes and impact hard, outcome measures
• As such, survey data presents opportunities for
  non-threatening discussions of