Educating Ourselves through Program Assessment Continuously Improving our Service to Students

1
Educating Ourselves through Program Assessment
Continuously Improving our Service to Students

• Dr. Ann Groves Lloyd
• Associate Dean
• Student Academic Affairs
• College of Letters Science
• University of Wisconsin-Madison

2
Snapshot of Student Academic Affairs
3
Letters Science Student Academic Affairs
Demographics

• Approximately 18 units
• Size varies from 15 staff members to one
• Some units have assigned populations
• Some units have a campus-wide mission
• College encompasses nearly 17,000 students
• Familiarity and use of tracking and evaluations
  tools varied widely

4
SAA Vision for Assessment

• Each unit would assess program quality,
  productivity, need, and demand
• Continuously improve quality of programs offered
  by capturing voices of students and other
  constituents
• Enhance program planning and budgeting
• Establish baseline data to explore longitudinal
  trends and patterns
• Satisfy institutional/state/national reporting
  and review requirements

5
Charge to Assessment Committee

• Develop instruments to measure student
  perceptions and overall quality of student
  experience
• Develop standardized expectations for unit
  directors/coordinators
• Provide staff with a practical, helpful tool
• Begin systematizing data collection to acquire
  retention and longitudinal data
• Pilot project in place for 2002-03 academic year

6
LS SAA Assessment Committee

• Made up of a cross-section of staff in SAA
• Determined that a handbook would be the best
  route to go
• Developed timeline and content areas
• Split into teams to work on each section

7
Factors for Success

• Involved those who will implement and use the
  Handbook
• Process involved on-going negotiation and
  compromise
• Wanted to take the assessment picture down to
  very small and practical increments
• Took existing systems into account
• Developed flexible, adaptable plan
• Encourage risk-taking

8
Foundation

• Differentiates between tracking for annual
  reporting and collecting data for program
  improvements (Service Quality versus Student
  Learning Outcomes)
• Does not ask unit directors/coordinators to get
  into determining retention rates
• Next steps will be defining and measuring
  learning outcomes

9
Handbook Introduction

• Two-hour session with unit coordinators/directors
  in May 2002
• Associate Dean discussed her vision for
  assessment for our division of student academic
  affairs
• Assessment Team walked everyone through the
  contents of the Handbook

10
Handbook Contents

• Associate Deans vision for Assessment in Student
  Academic Affairs
• Ethics in assessment
• Expectations for unit directors regarding
  collecting usage statistics and program
  improvement data, developing annual reports, and
  conducting self-studies
• Primer on the basic tools of assessment
• A resource section

11
Ethics in Evaluation

• Ensure ethical treatment of research participants
• Avoid harm
• Informed consent
• Freedom from coercion
• Ensure ethical collection of data
• Ensure ethical interpretation and presentation of
  data

12
Primer on Statistical Concepts

• Includes brief discussion and cautions regarding
  interpretation of data
• Also addresses managing data, including issues of
  confidentiality and security
• Has changed significantly since evolution of
  on-line survey tools such as Zoomerang.com and
  Survey Monkey

13
On-Going Assessments

• Capturing student names/IDs in ISIS and/or
  stand-alone databases (discouraged)
• Collect demographic information about program
  participants
• Capture information about advising appointments
• Establish systems to regularly assess student
  satisfaction with services
• Obtain feedback for program improvement
• Systematically gather data to assess retention
  and persistence (done at the divisional and
  college level)

14
Ways to Collect On-Going Data

• Phone calls
• Emails
• Advising contacts
• Resource use
• Workshop attendance
• Events in general
• Website hits
• Number of students turned away from services

15
Annual Reports

• Each SAA unit will submit an assessment plan in
  September each year.
• Each unit will submit an annual report due in
  January each year.
• Responsibility of unit director/coordinator
• Snapshot of previous years accomplishments and a
  look toward future initiatives and challenges

16
Contents of Annual Report

• Table of Contents
• Executive Summary
• Mission Statement and Core Values
• Strategic Plan if applicable
• Overview of past year
• Future goals and initiatives

17
Self Studies

• Ten-year cycle for self studies
• Deans office has determined expectations for
  self studies
• Additional details are outlined in the handbook

18
Assessment 101

• Selecting Assessment Tools
• Statistical Concepts
• Interpretation
• Managing Data
• Surveys
• Interviews

19
Selecting Assessment Tools

• Quantitative
• Quantitative
• Advantages and disadvantages of each
• Encourages multiple measures and methods

20
Surveys

• Gathering demographic information
• Types of questions
• Ways to gather survey data
• Tips
• Survey Kit by Arlene Fink (Sage Publications)

21
Interviews

• Role of human subjects committee
• FAQ prior to interviews
• Focus groups versus in-depth individual
  interviews
• Analyzing interviews

22
Resource Section

• List of campus resources
• List of web-based resources
• Books on assessment
• Article by Upcraft Schuh from About Campus
  (1998)
• AAHE 9 Principles of Good Practice for
  Assessing Student Learning
• Handbook can be found at
• http//wiscinfo.doit.wisc.edu/obpa/Assessment/SAA
  _Assessment.htm

23
Where are we now?

• Got the tracking and evaluation systems
  established in all units
• Have had two sets of assessment plans and annual
  reports turned in
• Obtained feedback from unit directors/coordinators
  and began updating handbook
• SAA has spend the past nine months going through
  its own assessment and evaluation process

24
Final Thoughts

• Assessment can be intimidating, but knowing where
  you are is essential to determining where you
  want to go!
• Just pick someplace to begin dont get
  overwhelmed by the big picture
• Develop a culture that embraces continuous
  improvement and inclusion of student voices in
  the planning

25
Stretch and Take a Break!

• Discuss with your colleagues your biggest
  challenge regarding assessment

26
Self-Evaluation of Assessment Practices

• Tool for evaluating where youre at regarding
  assessment in your unit/department
• Identify strengths upon which to build
• Identify aspects to improve or enhance
• Catalyst for important conversation and action

27
(No Transcript)