Exploring High Quality Student Affairs Learning Outcomes Assessment Practices at Three American Rese

1
Exploring High Quality Student Affairs Learning
Outcomes Assessment Practices at Three American
Research Universities

• Adam S. Green, Ed.D.
• West Virginia Higher Education Policy Commission
• Research Planning Analyst
• 8 June 2007
• International Assessment Retention Conference

2
Problem

• Shift in the role of student affairs
  professionals
• From service providers to educators
• From satisfaction measures to learning outcomes
• No research on how those who have responded to
  calls to assess have successfully and fully
  implemented their student learning outcomes
  assessment plans

3
What is Assessment?

• Assessment defined
• The process of gathering and discovering
  information from multiple and diverse sources in
  order to develop a deep understanding of what
  students know, understand and can do with their
  knowledge as a result of their educational
  experiences the process culminates when
  assessment results are used to improve subsequent
  learning (Huba Freed, 2000, p. 8).
• Knowledge, skills, attitudes, and habits of mind
  that students take with them from a learning
  experience (Suskie, 2004, p. 75).

4
Purpose Significance of Study

• To explore high quality student affairs learning
  outcomes assessment practices at three American
  Research Universities in order to
• Produce a rich descriptive explanation of model
  student affairs assessment practices.
• Provide insight into how these student affairs
  divisions are enhancing student learning.
• Provide information from which to better
  understand what these divisions learned from
  their assessment processes, thus allowing others
  to develop high quality assessment strategies.

5
Research Questions

• What are the student learning outcomes
  articulated in the student affairs assessment
  plans?
• How are these learning outcomes assessed?
• Who is involved in the assessment planning
  process, and what are the major responsibilities
  of these individuals?

6
Research Questions Continued

• Are student affairs educators collaborating with
  others, such as academic affairs, in the creation
  and administration of assessments? How?
• How is student affairs assessment data used to
  enhance student learning experiences?
• What successes and challenges do student affairs
  educators face as they implement their assessment
  plans?

7
Conceptual Framework Assessment Process
8
Review of Literature

• Calls for reform
• Articulating student learning outcomes
• Using results to enhance student learning
  experiences
• Sustaining student affairs assessment

9
Research Design

• Qualitative research design
• Case study approach
• Multiple case designthree Research Universities
• Case institutions identified as those with high
  quality student affairs learning outcomes
  assessment models will guide case selection.
• Strengths and limitations of research design
• Generalizability
• Triangulation
• Reliability and validity

10
Pilot Study

• Pilot Study Data collection
• On-site interviews with
• Student Affairs Vice President
• Two Assistant Student Affairs Vice Presidents
• One Director of Multicultural Services and
  Disability Support Services
• Modifications made based upon pilot study
  feedback
• Additional scriptexplanation of learning
  outcomes
• Move question 1 to question 3 in interview
  protocol
• Add a final question Should I need to contact
  you
• Make sure all participant invitation letters are
  photocopied in color
• Follow-up emails to participants

11
Research Methods

• Site Selection
• Selection of Sites Initially 15 possible sites
• Three sites participated in this study
• 2 Doctoral Research Extensive Institutions
• 1 Doctoral Research Intensive Institution
• Initiating contact and gaining approval
• Assessment Directors/Coordinator served as
  liaisons for studythey identified appropriate
  participants based upon researchers criteria

12
Research Methods

• Purposeful SamplingResearchers Selection
  Criteria
• One Senior Student Affairs Officer (if not
  available, then an Assistant Senior Student
  Affairs Officer)
• One Director/Coordinator of Student Affairs
  Assessment
• Three Student Affairs Assessment
  Committee/Council Members
• Three Other Student Affairs Professional Staff
  Members (with at least one serving in the
  capacity of unit director and one non-unit
  director)
• Purposeful SamplingNumber of participants
• 8 participants at Western
• 8 participants at Southern
• 9 participants at Eastern
• 25 total

13
Research Methods

• Data collectionInterviews and Document Analysis
• Points of data aligned with research questions

14
Research Methods

• Data organization and analysis
• Concurrent process
• Analysis within case institution before comparing
• Coding procedures

15
Overview of Case Institutions
16
Overview and Rank of Participants
17
Case Institutions Student Affairs Assessment
History
18
RQ 1 What are the student learning outcomes
articulated in the student affairs assessment
plans?
19
RQ 1 What are the student learning outcomes
articulated in the student affairs assessment
plans?
20
RQ 1 What are the student learning outcomes
articulated in the student affairs assessment
plans?
21
RQ 2 How are these learning outcomes assessed?

• Most common methods identified at Southern and
  Western

22
RQ 2 How are these learning outcomes assessed?

• Eastern Universitys Approach
• Primarily centralized methodSurvey (1strongly
  disagree 5strongly agree)
• 2002 Administered at 273 programs
• 2003 Administered at 575 programs
• Limited evidence of unit level assessment

23
RQ3 Who is involved in the assessment planning
process, and what are the major responsibilities
of these individuals?

• Four Vital Levels of Participation
• Vice Presidents
• Leadership Financial support
• Assessment Directors/Coordinator
• Facilitator Change agent
• Assessment Committees
• Create reporting timelines and guidelines
• Initiate assessment dialogue
• Review unit level assessment plans and provide
  feedback
• Unit Level
• Worker bees

24
RQ 4 Are Student Affairs Educators Collaborating
With Others, Such as Academic Affairs in the
Creation and Administration of Assessments? How?

• Very little evidence of collaboration across
  cases
• Few Examples Identified in Study Included
• Southern University Multicultural Services and
  Greek Life
• Western University Assessment Committees
  creation of six broad learning goals
• Eastern University Office of Diversity Programs
  and Provosts Office

25
RQ 5 How are Student Affairs Assessment Data
Used to Enhance Student Learning Experiences?

• Southern and Western Universities

26
RQ 5 How are Student Affairs Assessment Data
Used to Enhance Student Learning Experiences?
Eastern University
27
RQ 6 What Successes and Challenges do Student
Affairs Educators Face as They Implement Their
Assessment Plans?
Southern University
28
RQ 6 What Successes and Challenges do Student
Affairs Educators Face as They Implement Their
Assessment Plans?
Western University
29
RQ 6 What Successes and Challenges do Student
Affairs Educators Face as They Implement Their
Assessment Plans?
Eastern University
30
Conclusions

• Assessment activities across cases depended
  highly upon four levels of support
• Vice President, Coordinator/Director of
  Assessment, Assessment Committees, and Unit Level
  Staff
• Decentralized model of assessment, facilitated by
  a coordinator or director, is most appropriate in
  student affairs
• Southern and Westerns articulated learning
  outcomes were primarily cognitive (65) and less
  were affective (35)
• Surveys were most common method used to assess
  student affairs learning outcomes

31
Conclusions

• Assessment planning and implementation is
  primarily a single-unit task with very little
  evidence of collaboration between student affairs
  units or outside of the division
• Less than half (45) of decisions made at
  Southern and Western based upon assessment
  results were to modify educational program or
  service. Other decisions included modify
  assessment method, continue same practices,
  enhance assessment administration training
• Successes and challenges varied across
  institutions

32
Recommendations for Practice

• Engaging in student affairs learning outcomes
  assessment efforts requires adequate level of
  support from divisions leaders.
• Student affairs divisions should identify broad
  learning goals or objectives based upon the
  institutions mission to provide guidance to unit
  leaders when they are developing their own
  learning outcomes.
• Student affairs educators must consider carefully
  what they want students to learn as a result of
  their program or service.
• Continuous professional development opportunities
  must be provided to help unit level staff become
  assessment experts.

33
Recommendations for Future Research

• This study could be replicated at the unit level
  
• (e.g., Compare five career service units
  assessment practices)
• Comparisons between student affairs assessment
  efforts that are located within the various
  accrediting bodies across the United States
• Examination of assessment recommendations
  provided from national professional organizations
  and how units are implementing those
  recommendations
• Because evidence of collaboration was limited in
  this study, it would be useful to examine
  successful collaborative assessment efforts
  within the field

34
Questions?

• Comments?

Adam S. Green, Ed.D. green_at_hepc.wvnet.edu
304.558.1112