Assessing Student Attainment in the Undergrad. Major:

1
Assessing Student Attainment in the Undergrad.
Major

• Examples of Discipline-Specific Strategies
• Senior Thesis or Writing Project
• Senior Research Project
• Capstone Course with Assessment Embedded in the
  Course
• Comprehensive Examination or Proficiency Test
• Student Portfolio, Performance, or Exhibit
• -- with expert or peer assessment
• Senior Essay followed by Focus Group Discussions
  with Faculty
• Student/Faculty Retreat for Collective Assessment
• Internship or Field Work
• Analysis of Historical, Archival, or Transcript
  Data
• Alumni and Employer Surveys and Interviews

2
Should I purchase an instrument or create my own??

• If you develop your own instrument
• can be less expensive (at least in dollars)
• can address institutionally-specific issues
• can be very time consuming
• must attend to formatting, ease of completion
  for respondents, printing, on-site scoring
• may require pilot testing
• can require extensive commitments for
  reliability, validity checks or can
  jeopardize credibility
• Has less external credibility, can increase
  chances for criticism

3
Development of Local Instruments
Literature Review and Collection of Survey Items
Item Reduction Revisions by Research Team
Pilot Test to Determine Psychometric Properties
Revise
Final Survey Items
Seniors
Vet with Faculty, Employers, Alumni, and
Administrators
Alumni
Faculty
Program Chairs
Employers
4
Tips for Improving Response Rates

• Explain the Importance of the Survey
• Explain How the Results will be Used
• Assure Confidentiality
• Thank Participants
• Make the Instrument Interesting
• Offer Incentives, if possible
• Offer to Share the Results
• Be sure the Survey is easy to understand and
  complete
• Offer multiple modes of response (web, paper,
  telephone)

5
Should I purchase an instrument or create my own??

• If you purchase an instrument
• Requires less time for design, printing, etc.
• Likely has established reliability and validity
• Often includes norms groups for
  inter- institutional comparisons
• May include scoring and reporting services
• Likely to have external credibility, Increases
  face validity
• May not fit your curriculum goals
• Can be expensive in dollars

6
Advantages of Standardized Tests

• Validity Established through
• content specification
• expert panel reviews for fairness
• pilot testing
• Minimal Staff Time Involved
• Comparative Data and User Norms Available
• Short Turn-around Time for Results
• Access to Professional Expertise

7

• Why not use the GRE for assessment?
• GRE Exams have the following deficiencies.
• GRE scores are relational and only answer the
  comparison question. You dont know what a 600
  means in terms of student achievement because you
  dont know how many questions were answered
  correctly and incorrectly.
• The GRE uses the wrong comparison group. Instead
  of comparing each GRE score against all college
  graduates, student GRE scores are in relation to
  a graduate school-bound population.
• There are no GRE subfield scores within each
  major field, so you cannot tell if student
  performance is congruent with the curriculum.
• The ETS Major Field Exam was constructed in
  response to these weaknesses in the GRE. The
  Major Field Exam scores not only are normed on
  populations of graduating seniors, and not only
  indicate the number answered correct (non
  relational), but also report scores by subfield,
  thus providing useful information for analyzing
  the curriculum. I recommend that the faculty
  send off for a specimen set, take the exam,
  discuss and decide.

8
Validity of Self-Reported Data

• Self-reports vs. actual scores/tests in various
  studies shown to correlate (r .50 to .90)
  with
• SAT GRE Verbal and Quantitative scores
• National Teacher Examination scores
• Criterion-referenced achievement tests in
  English, math, science, and social studies
• College and High School GPA
• Course material tests
• Behaviors/activities consistent with reported
  gains

9
When Self-Reports are Reasonable Proxies for
Objective Measures

• The information requested is known to the
  respondent
• Questions are phrased clearly and unambiguously
• A moderate-to-high degree of overlap in content
  exists between the measures
• The instruments measure the same constructs, and
  questions refer to recent activities
• Respondent believes questions merit a serious,
  thoughtful response
• Answering wont threaten, embarrass, or violate
  the respondents privacy, nor encourage socially
  desirable responses
• Self-reported gains/proficiencies are for groups
  (averaged), rather than for individuals
• (Based on Kuh, 2005 and Pike, 1995)

10
Strengths of Surveys

• Great Flexibility and Universal Applicability
• Ease of Construction
• Application to all Students
• Results do not Require Expert Interpretation
• Relatively Inexpensive

11
Weaknesses of Surveys

• Better for Measuring Group Responses than for
  Individual Responses
• Best if Supplemented by other information from
  Interviews Focus Groups.
• To Construct Good Surveys
• avoid vague items
• Construct multi-item scales
• Link to constructs in your model
• or to educational goals
• Effective visual layout

12
Value of Alumni Studies(Volkwein, 1990)

• Alumni provide a Janusian perspective -
  Internal and external experiences.
• Alumni and employers have legitimacy with both
  internal and external stakeholders.
• Internally, Alumni Studies can assess important
  outcomes and provide info for enhancing curriculu
  m, programs, and policy.
• Externally, Alumni studies can support
  accreditation, accountability, recruitment, and
  fund raising.
• Such studies provide an opportunity for faculty
  and administrative collaboration.
• Centralized collection, decentralized uses

13
Outcomes Approach to Alumni Assessment

• Assessment of alumni rests on the premise that
  institutional quality and effectiveness can be
  measured by examining
• Educational Job satisfaction
• Income Socio-economic Status
• Occupational Attainment
• Engagement in civic and political activities
• Values like Tolerance for diversity
• Collegiate experiences
• General Satisfaction with Institution
• Quality of Instruction received
• Preparedness for employment Grad School
• Willingness to enroll again Donate
• (Volkwein, 1990, 1998 Dellow Romano, 2002)

14
Selected Alumni Surveys

• ACT Alumni Survey (2-year 4-year)
• SUNY Alumni Outcomes Survey
• Appalachian Region Alumni Outcomes
• SUNY-Albany Alumni Survey
• HEDS Alumni Survey
• HERI Alumni Survey
• NCHEMS Alumni Assessment Survey
• Clemson University Alumni Survey
• Georgia Tech Alumni Survey
• Penn State Engineering Alumni Survey

15
ACT Alumni Outcomes Survey

• Demographics/Background/Career (15)
• Employment History and Experiences (18)
• Educational Outcomes (22)
• Critical Thinking Problem Solving (4)
• Life-long Learning (2)
• Social/Moral Values/Ethics (3)
• Multi-Cultural/Diversity (4)
• Citizenship/Global Issues (2)
• Team Work (2)
• Communication (2)
• Overall (3)
• Educational Experiences/Satisfaction (31)
• Activities and Organizations (11)
• Additional Questions (30 spaces)
• Space for Comments/Suggestions

16
Penn State - CSHE Engineering Alumni Survey

• Demographics/Background/Career (13)
• Undergraduate Engineering Outcomes/Experiences
  (72)
• Technical Skills and Abilities (15)
• Professional Skills (23)
• Analytical/Thinking Skills (10)
• Nature of Engineering Courses (15)
• Diversity Tolerance (9)
• Additional Information (9)
• Satisfaction (1)
• Test performance (1)
• Degrees, field, and minor (4)
• Planned and Actual employment (3)

Surveys available at http//www.ed.psu.edu/cshe/ab
et/instruments.html
17
Percent of Alumni That Would "Attend All Over
Again" and Select Same Major and Same Career
(N1743)
Mean Response on a 5 point Scale
12
66
3.9
Attend All Over Again
11
67
3.9
63
14
3.8
26
54
3.5
20
Select Same Major
61
3.7
These mean responses are significantly higher
than those in careers not related to the
major.(plt.01)
34
41
3.1
19
58
3.6
16
62
Select Same Career
3.7
21
52
3.5
0
20
40
60
80
100
0
20
40
60
80
100
Probably Definitely No
Probably Definitely Yes
All Alumni
Career Related to Major
Career Not Related to Major
18
Perceived contribution of the College Experience
To Alumni Development as Undergraduates
Mean Response
(Scale of 1 to 5)
68
3.8
4
Intellectual Growth
4.2
70
4
3.7
63
9
Means are significantly different from pre-2000
(plt.01).
Personal Growth
71
3.9
12
3.6
58
11
Pre-2000
Social Growth
70
3.9
10
Post-2000
3.7
62
Effective Preparation
9
51
for Graduate School
3.5
13
3.6
Effective Preparation
56
12
for Employment
42
3.3
19
0
20
40
60
80
100
0
20
40
60
80
Percent responding 1 or 2 on a 5 point scale
Percent responding 4 or 5 on a 5 point scale
19
Abilities and Skills Rated by Engineering
Employers as Highly Important for Success

• 99
• 99
• 97
• 97
• 95
• 94
• 92
• 90
• 85
• 73
• 70

• Communicate effectively
• Problem solving
• Apply math, science, and engineering
• Use modern engineering tools
• Teamwork
• Understand professional and ethical
  responsibilities
• Design a system to meet needs
• Life-long learning
• Design and conduct experiments
• Knowledge of contemporary issues
• Engineering In global and social contexts

20
Abilities and Skills Rated by Non-Engineering
Alumni as Highly Important for Success

• Function independently, without supervision
• Exercise personal responsibility
• Listen effectively
• Exercise self-discipline
• Exercise problem-solving skills
• Maintain openness to new ideas
• Speak effectively
• Evaluate and choose between alternative actions
• Think analytically and logically
• Acquire new skill and knowledge on your own
• Possess clear goals
• Cope with conflict
• Understand myself
• Write effectively
• Lead and supervise tasks and groups of people
• Function effectively as a member of a team
• Learn how to learn

• 96
• 96
• 93
• 91
• 91
• 90
• 90
• 89
• 89
• 87
• 84
• 82
• 80
• 80
• 78
• 75
• 75

21
Consensus Liberal Education Outcomes

• Over the past ten years, the AACU appears to
  have forged a national consensus among business,
  government, and accreditation leaders about the
  liberal education outcomes that all
  undergraduates should possess, summarized in Our
  Students Best Work A Framework for
  Accountability Worthy of Our Mission (2004)
• Strong analytical, communication, quantitative,
  and information skills.
• Deep understanding or hands-on experience with
  the inquiry practices of disciplines that
  explore the natural, social, and cultural
  realms.
• intercultural knowledge and collaborative
  problem-solving skills.
• A proactive sense of responsibility for
  individual, civic, and social choices.
• Habits of mind that foster integrative thinking
  and the ability to transfer skills and knowledge
  from one setting to another.

22
The Value of a Model

• Encourages Clarity of Purpose
• Stands as a Road Map or Guide for Developing
• Research Questions
• Assessment Design
• Data Collection
• Hypothesis Testing
• Statistical Analysis
• Serves as a Logic Chart (Otherwise Causality is
  not Clear)
• Assists Workload Conservation and Fog Dissipation
  (concentrates energy and attention, streamlines
  research design, reduces the amount of data
  collection, data storage, analysis, and reporting)

23
 
Volkwein Effectiveness Model
1. Janusian Duality
2. Five Questions 1. Are you meeting
your goals?2. Are you improving? 3. Do you meet
the standard?4. How do you
compare?5. Are your efforts
cost-effective?
Inspirational Purposes-for Internal Improvement

• 5.
• Communicate Results
• Take Academic
• Administrative Action
• Improve Strengthen
• Programs

4. Collect Analyze Evaluation Evidence
Pragmatic Purposes- for External Accountability
3. Select Methods Measures
3. Select Model, Methods, Measures
For Academic Effectiveness Student Learning,
Research Scholarship
For Administrative EffectivenessPlanning
Resource Management
For Administrative EffectivenessStrategic
Planning Resource Mgmt
For Academic Effectiveness Student Learning,
Research Scholarship
For Each Level --Institution --Program
--Individual
24
Todays Effectiveness Approach

• Undergraduate Focus,
• Goal driven,
• Evidence based,
• Improvement oriented

25
Assessment Resources

• Patrick Terenzinis article, Assessment with
  Open Eyes (JHE, 1989).
• Middle States 2003 Handbook, Student learning
  assessment
• Options and resources.
• Linda Suskies 2004 book, Assessing Student
  Learning A Common Sense Guide
• Several books by Trudy Banta and her associates
  constitute helpful
• resources for campus assessment efforts,
  Assessment in practice
• putting principles to work on college campuses
  (Banta, et al. 1996),
• Assessment essentials planning, implementing,
  and improving
• assessment in higher education (Palomba Banta
  1999), and
• Building a scholarship of assessment (Banta et
  al., 2002).
• Additionally, there have been several
  constructive national attempts to
• develop guidelines and standards for good
  assessment practices.
• The most significant of these are the AAHE Nine
  Principles of Good
• Practice for Assessing Student Learning, and the
  recommendations of
• the AACU (from Our Students Best Work, 2004).