Learning Outcomes Assessment in Student Services Methods, Strategies, Resources

1
Learning Outcomes Assessment in Student
ServicesMethods, Strategies, Resources
2
Presenters

• John Baker
• VP Student Services
• Gavilan College
• 408- 848-4738
• EMAIL Jbaker_at_gavilan.cc.ca.us
• Robert S. Gabriner, Dean
• Office of Research, Planning
• and Grants
• City College of San Francisco
• 415-239-3014
• EMAIL rgabrine_at_ccsf.cc.ca.us
•  
• Margery Regalado
• Enrollment Management
• Gavilan College
• EMAIL mregalado_at_mail1.gavilan.edu

• Jerry Rudmann
• Research
• Coastline College
• 714-241-6338
• EMAIL jrudmann_at_mail.ccc.cccd.edu
• Terrence Willett
• Research
• Gavilin College
• EMAIL twillett_at_mail1.gavilan.edu

3
Electronic ResourcesCalifornia Assessment
Institutes Website http//www.ca-assessment-inst
.org/
4
Electronic ResourcesLearning Outcomes Listserv

• To subscribe to the Listserv
• Send an e-mail message to Listserv_at_Listserv.cccnex
  t.net
• Do not put any words in the subject area.
• In the body of the email message, Mary Smith
  would write the following subscribe
  LearningAssessment Mary Smith
• When sending the message, substitute your first
  and last names in place of "Mary Smith"

5
Resources to help you at your campus
Table of Contents

• I. Standard IIB available from the AACJC
• II. In this PowerPoint
• A. Discussion Questions for getting people
  thinking about learning outcomes in Student
  Services
• B. Examples of assessment tools
• C. Criteria for evaluating assessment tools
• D. Sources of SLOs in Student Services
• E. Considerations related to assessment
• F. Hypothetic assessment plan for diversity
• G. Steps for developing an assessment plan
• III. Worksheets for individual and group work

6
Content review of standard IIB

• Learning support needs of students
• Identify, support, assure, evidence
• Learning outcomes specified in IIB
• Personal and civic responsibility, intellectual,
  aesthetic, and personal development
• Understanding and appreciation of diversity
• Contribution to student learning outcomes
• Evidence

7
Discussion and Thought Questions Regarding
Outcome Assessment in Student Services

• We wish to acknowledge and thank
• Kathe German
• of City College San Francisco for contributing
  these excellent questions.

8
Q. What are some major outcomes we expect
students to achieve as a result of a college
education?

• Develop self-aware and interpersonally sensitive
  individuals
• Developing democratic citizens who participate in
  the political and social workings of the society
• Becoming persons having a master of a body of
  knowledge and insight into oneself and the world
• Development of skills that would qualify one as a
  skilled worker
• Development of lifelong learning skills

9
Q. What is the purpose of student learning
outcomes and why are they important (other than
to meet the new accreditation standards)?

• To provide a focus, both for staff and students
  a way of articulating the primary purpose for the
  institution.
• To provide a framework against which to assess,
  document, and improve the achievement of learning
  outcome.
• Help focus our efforts on student growth, and
  thereby influencing the structure and delivery of
  programs and services.
• Help our constituencies, from students to the
  public, understand our intentions, affirm the
  values of our contributions, and document the
  impact Student Services is/are having on students.

10
Q. What do student learning outcomes have to do
you with, as a faculty member, staff person, or
administrator working in Student Services?

• We you and I are the purveyors of these
  experiences, and we you and I are the only
  ones who can really identify and foster the
  development of student learning outcomes within
  Student Services at our own college.

11
Q. What Student Service functions/units are
represented by those in the room today?
12
Q. Can any of you provide some possible learning
outcomes that your Student Services function
impacts? That is what are ways in which you
already address learning outcomes with your
students?
13
Q. What are some challenges and opportunities
that we face as we try to address identification
and assessment of learning outcomes?

• Commuter college
• Bureaucratic inertia
• Shrinking resources and tight budgets
• More

14
Q. What impact will the implementation of student
development and learning outcomes have on your
role and recognition within the institution?
15
B. Examples of some useful assessment
tools in student services
16
Instruments

• Home grown survey instruments
• CCSEQ (Community College Student Experiences
  Questionnaire)
• The LASSI (Learning Study Strategies
• Inventory)

17
Home-grown surveys

• Coastline developed a student service survey. For
  each service, students are asked the following
• Have you heard of the service?
• If you have heard of the service, have you used
  it?
• If you have used the service, please express your
  level of satisfaction with the service.
• Please comment on the service.
• This survey is administered to
• selected web classes (an online survey)
• selected classroom courses (paper survey)
• students who file a petition to graduate

18
CCSEQ (Community College Student Experiences
Questionnaire) College Activity Areas measured
by the CCSEQ (83 items)

• Course Activities
• Library Activities
• Faculty
• Student Acquaintances
• Art, Music, Theater Activities
• Writing Activities

• Science Activities
• Vocational Skills
• Clubs and Organizations
• Athletic Activities
• Counseling and Career Planning
• Learning and Study Skills

19
Self-assessed gains

• CCSEQ asks students to rate their progress on 23
  key educational outcomes.
• Some of the outcomes are highly relevant to
  Student Service programs and activities
• Gaining information about career opportunities
• Developing clearer career goals
• Understanding other people and the ability to get
  along with different kinds of people
• Developing good health habits and physical fitness

20
Selected activity items from the CCSEQ
(Community College Student Experiences
Questionnaire) relevant to the diversity example.

• One questions asksWhile in attendance at
  this college DURING THE CURRENT SCHOOL YEAR,
  about how often have you had serious discussions
  with students
• who were much older or much younger than you
• whose ethnic or cultural background was different
  from yours
• whose philosophy of life or personal values were
  very different from yours
• whose political opinions were very different than
  yours
• Whose religious beliefs were very different than
  yours
• From a country different than yours.
• Response options available to the student
• __ Never
• __ Occasionally
• __ Often
• __ Very Often

21
Self-assessed gains on CCSEQ

• Students are asked to rate their progress on 23
  key educational outcomes.
• Gain items relevant to our diversity example
  include
• Understanding other people and the ability to get
  along with different kinds of people.
• Becoming aware of different philosophies,
  cultures, and ways of life.
• Learning more about other parts of the world and
  other people (Asia, Africa, South American,
  etc.).
• Developing the ability to speak and understand
  anther language.
• There are creative ways to increase positive
  student to student contact.
• ESL students meet with students in Spanish
  classes.

22
Some CCSEQ Findings

• the more effort students put into their college
  experiences, the more likely they are to make
  personal and academic gains and what students
  get of our college will depend, to a considerable
  degree, on the extent to which they take
  advantage of opportunities that are provided in
  the college setting (Friedlander and
  MacDougall).
• We should encourage students to take advantage of
  enriching events offered by the college.
• We (administrators, faculty, support staff)
  should be role models by attending these events.
• Faculty should make sure we are offering quality
  opportunities/assignments in our classes.
• Quality of effort includes in-class and
  out-of-class experiences.

23
CCSEQ
Center for the Study of Higher Education 308
Browning Hall The University of Memphis
Memphis, TN 38152 Phone [phone] Fax
[phone]  E-mail CCSEQLIB_at_cc.memphis.edu
24
LASSI

• Learning and Study Strategies Inventory

• The instrument can used as a diagnostic tool with
  students.
• Good before and after measure for evaluating the
  
• effectiveness of courses or workshops on
  improving study
• practices and attitudes.
• The instrument includes 77 items related to
  studying and learning.
• There is a paper version and a computerized
  version.
• Results can be customized to fit resources on
  your campus.

25
LASSI Learning and Study Strategies Inventory

• Information Processing Ability to create
  imaginal and verbal elaborations and
  organizations to foster understanding and recall.
  
• Selecting Main Ideas Skill at selecting
  important information from the unimportant
  information that does not need to be remembered.
• Study Aids/Strategies - Ability to use or create
  study aids that support and increase learning and
  retention.
• Self-Testing Testing ones understanding.
• Test Strategies Use of effective strategies to
  prepare for a test and other effective strategies
  useful while taking a test.

• Attitude General attitude and motivation for
  succeeding in school and performing the tasks
  related to school success.
• Motivation The degree to which one accepts
  responsibility for doing the specific tasks
  necessary for school success.
• Time Management This scale measures the degree
  to which one creates and uses schedules.
• Anxiety This scale measures how tense or
  anxious one is when approaching academic tasks.
• Concentration Ability to concentrate and direct
  ones attention to school and school-related
  tasks, including studying.

26
(No Transcript)
27
LASSI
HH Publishing Co., Inc.1231 Kapp Drive,
Clearwater, FL 33765-2116(800) 366-4079, (727)
442-7760 http//www.hhpubco.com/LASSI/
28
Student Exposure to Learning Study Skills
InstructionRudmann (1997) (N 634
students)This shows that many students have had
little, if any, instruction on strategies needed
to succeed in college.
Percentages of students exposed to three types
of learning strategies
29
Other data-gathering strategies

• Surveys
• Scannable approaches
• Online surveys
• Automatic telephone surveys
• Focus Groups

30
C. Criteria for selecting assessment instruments

31
Important criteria for selecting or creating a
quantitative instrument (test) and extremely
valuable resource documents

• Student service staff experts should do a content
  review of the items to assure they appear to
  measure the outcomes of interest. (Test names can
  mislead.)
• Test manual containing information on the tests
• reliability
• validity
• norming studies
• test bias review
• Scoring efficiency
• Cost
• Key references
• Matriculation Local Research Options - Written
  in 1991 and 1992, this series of three manuals
  remains an invaluable tool for assessment test
  validation and evaluation http//www.rpgroup.org
  /
• Standards, Policies and Procedures for the
  Evaluation of Assessment Instruments Used in the
  California Community Colleges Available from
  the CCC Chancellors Office (Student Services
  unit) at
• http//www.cccco.edu/divisions/ss/matriculation/at
  tachments/stdpoprevalass.pdf

32
D. Assessment considerations
33
Categories of assessment

• Direct versus indirect assessment of learning
  outcomes
• Direct assessment of learning
• Usually means assessment using quantitative
  assessment tools e.g., tests, rubrics.
• Indirect assessment of learning
• Some feel all qualitative assessments (e.g.,
  surveys, focus groups, exit interviews) are
  indirect measures of learning.
• Assessment of achievement
• Graduation rates, transfer rates, course success
  rates, etc.

34
Quantitative and Qualitative Assessment Options

• Quantitative and qualitative
• Quantitative examples
• Scores on standardized or homegrown tests
• Scores derived from using a rubric to assess
  performance of some ability (e.g., a speech)
• Scores derived from an observation checklist
• Qualitative examples
• Narrative drawn from a focus group
• Narrative and trends summarized from exit
  interviews

35
Considerations for qualitative approaches to
assessment

• Equipment
• People power
• Training

36
Decisions to make, factors to consider regarding
your assessment plan

• Voluntary or mandatory participation of students?
• Cross-sectional and/or longitudinal data
  gathering?
• Small or large random samples?
• Convenience samples? (Be careful here.)

37
Triangulation

• Triangulation refers to the use of more than one
  measure for assessing progress on a given
  learning outcome.
• Triangulation is necessary because relying on a
  single measure (e.g., survey, interview, or
  standardized test) will always miss important
  aspects of what you are trying to measure.

38
Exampleusing triangulation to assess why
students drop out of college

• Interview students when they submit withdrawal
  forms at AR.
• Send a questionnaire to parents.
• Give dropout students a personality or attitude
  scale.
• Interview teachers of advisors of dropout
  students.
• Compare institutional records of dropout students
  with those of persisting students.

39
Wrong assumptions about assessment

• Whoever made the survey knew exactly what was
  important to ask the students.
• Everything important can be obtained through a
  single survey, standardized instrument, or
  interview.
• Students who participate in the data collection
  are representative of all students on campus.
• The information sought isnt available elsewhere.

40
Analyze and report the data

• The 1 page report (Jon Kangas)
• Title outcome(s) assessed
• Method brief summary of how and when data were
  gathered
• Results Narrative, followed by table, followed
  by a graph
• Conclusion(s)

Place analyzed data into an easily understood
format
41

• It may take up to 3 semesters to
• Identify the outcomes to assess
• Select instruments
• Create in-house instruments
• Plan out and pilot test any strategies using
  qualitative methods
• Create an efficient student tracking system
• Develop and pilot-test a documentation system
• Think of outcomes assessment as an on-going,
  term-by-term activity, not something to worry
  about every 6 years for the self-study.

42
After one assessment cycle, evaluate the methods
used and the results obtained

• Are your assessment tools appropriate or do they
  need modification? Efficient, cost-effective?
• Regarding the learning outcomes retain, modify,
  delete, add?
• Are your sampling strategies appropriate or need
  adjustment?
• It may take several cycles to perfect the entire
  process.

How effective was the effort to assess learning
outcomes in Student Services?
43
E. Some sources of learning outcomes in Student
Services
44
SLO Sources in Student Services Clubs

• Biology Weekend field trips
• Criminal Justice National competitions
• Geology Weekend field trips
• Political Science Model United Nations
• Psychology Attend professional conferences
• Miscellaneous clubs Ethnic groups, hobbyists,
  recreational, MTE (Making Transfer Easy)

45
At many 2-year colleges Psi Beta (Psychology)
students attend professional conferences and
present their behavioral research studies.

• Students receive recognition from the National
  Psi Beta Office

46
SLO Sources in Student Services Matriculation
Orientations

• Students receive information about
• Campus Resources
• Adding and dropping classes
• Transfer information
• Interpretation of assessment results
• Etc.

47
SLO Sources in Student Services Peer Tutoring

• Tutors must complete a course on tutoring
• How to help peers prepare for tests, manage time,
  etc.
• Tutors themselves acquire deeper understanding of
  material

48
SLO Sources in Student Services Serving on
College Governance Committees

• Examples
• Academic Senate
• Curriculum Committee
• DSPS Advisory Committee
• Research Advisory Committee

49
SLO Sources in Student Services Interactions
with Staff

• Student affairs professionals often serve as
  role models. They also
• explain procedures,
• serve as advocates,
• provide advice and assistance.

50
SLO Sources in Student Services Student
Government

• Some colleges offer leadership classes that are
  transferable as electives to CSU. Content
• Improve leadership and management skills
• (Students earn a leadership certificate 12
  units)
• Students attend leadership conferences and
• serve on college governance committees
• SLO Resource Miller, T.K. (ed), (1997). The Book
  of Professional Standards for Higher Education.
  Washington, DC Council for the Advancement of
  Standards in Higher Education. www.cas.edu This
  resource provides a list of student leadership
  SLOs organized under 1) foundations of
  leadership, 2) personal development, and 3)
  organizational development.

51
SLO Sources in Student Services Workshops

• Health Office presentations
• Eating disorder screening and workshop
• Depression screening workshop
• Safe sex, sexually transmitted diseases
• Stress management
• College Transfer Workshops

52
SLO Sources in Student Services Work Study

• Students work in various jobs at the college
  (funded by EOPS and Financial Aid Office).
• Some of these may be sources of learning outcomes
  for students.

53

• It is important to acknowledge that Student
  Service units often play a key role by supporting
  and enabling many educationally enriching
  opportunities for our students.
• These events and activities often are rich
  sources of learning.

54
F. Hypothetical outcomes, educational strategies,
and and assessment plan for campus diversity

• Understanding and Appreciation of Diversity

55
Students will exhibit tolerance and acceptance
of persons from different cultural, racial, and
religious heritages.

• Students will be able to
• Evaluate their current levels of acceptance for
  culturally different people.
• Exhibit knowledge of diverse human cultures.
• Interact with people from different cultural
  heritages.
• Accept feedback from different cultural
  heritages.
• Provide feedback to others from different
  cultural heritages.
• Identify and distinguish between examples of
  artistic expression from diverse cultures.
• Describe different religious beliefs throughout
  the world.
• Evaluate currently held biases, prejudices, and
  stereotypical thought.

56
Hypothetical strategies for developing an
understanding and appreciation of diversity on
the campus

• All GE courses include at least one tolerance
  development lesson.
• Students encouraged to participate in
  multi-cultural programs and activities.
• Student orientation includes a section on
  institutional expectations concerning relations
  among students from differing ethnic backgrounds.
• Leaders of tolerance development activities
  purposely includes staff and students from
  differing cultural and racial backgrounds.
• Community service activities are designed to
  bring together students of differing backgrounds.

57
A sample plan for the assessment of tolerance of
racial, ethnic diversity on the campus
58
G. Assessing Student Learning Outcomes
in Student Services

• Steps for getting started

59
Systematic Assessment Cycle
Interpret Evidence
Mission/Purposes
Objectives/Goals
Gather Evidence
Outcomes
Make decisions to improve programs enhance
student learning make informed institutional
decisions public accountability
Adapted from Peggy Maki PhD, Marilee J.
Bresciani, PhD
60
Typical Components of An Assessment Plan
(adapted from M. J. Bresciani, PhD)

• Mission
• Objectives or Goals
• Evaluation Methods
• With criteria and by outcomes
• Implementation of Assessment
• Who is responsible for what?
• Timeline
• Results
• Decisions and Recommendations
• For each outcome
• For the assessment process itself

61
Mission Statement (adapted from M. J.
Bresciani, PhD)

• Derived from colleges mission statement,
  divisions mission statement, strategic plan,
  focus groups, etc.
• Can also come from your professional organization.

62
Please Write Down a Mission Statement for Your
Program/Service (adapted from M. J. Bresciani,
PhD)

• Examples
• Student Government promote leadership skills
• Student Activities Office enhance students
  understanding and appreciation of diversity
  promote personal and social development
• Health Office promote healthy lifestyles
• DSPS support students so they can achieve
  their education goals

63
Program Objectives or Goals (adapted from M.
J. Bresciani, PhD NC State University)

• Are broad, general statements of (1) what the
  program wants students to be able to do and know,
  or (2) what the program will do to ensure what
  students will be able to do and know.
• Are related to the colleges and programs
  mission and objectives.
• Are by evaluated by measuring specific outcomes
  related to the objective.

64
Program Objectives or Goals - examples

• Student government will provide opportunities to
  develop and improve leadership skills.
• DSPS will provide students with counseling and
  guidance on how to effectively advocate for
  themselves.
• Student activities will provide events and
  speakers to promote a better understanding and
  appreciation of cultural and ethnic diversity.

65
Evaluate Each Objective or Goal adapted from M.
J. Bresciani

• Is it meaningful?
• Is it important?
• Is it a broad, general statement of what the
  program will do to help students be able to do
  and know OR what the program wants students to be
  able to do and know?
• Does it relate to the program mission?
• Can an outcome be specified to measure
  achievement of this objective?

66
Another Example

• Career Course
• To provide students with information and
  activities to help them make informed major and
  vocational choices.

67

• Please write at least one objective / goal for
  your student services program.

68
Outcomes adapted from M. J. Bresciani NC
State University

• Are more detailed and specific statements derived
  from your goals.
• Are the end results of your efforts what you
  want the student to know or do.
• Are the deliverables.
• Use active verbs such as explain, conduct,
  describe, present.
• Career course example The student will be able
  to prepare a good resume.

69
Online Resources for Writing Learning
Outcomes Blooms Taxonomy

• http//www.kent.wednet.edu/KSD/MA/resources/blooms
  /teachers_blooms.html
• http//www.nwlink.com/donclark/hrd/bloom.html

70
Questions to Ask About Each Outcome adapted
from M. J. Bresciani NC State University

• Can it be measured?
• Is it meaningful?
• Is it manageable?
• Target audience of your outcome?
• Will I know if it has been met? How?

71
Examples of outcomes for understanding and
appreciating diversity

• Students will organize a multicultural event
  that attendees deem relevant to their
  understanding of other cultural perspectives.
• Students will engage in conversations with
  students from different cultural backgrounds and
  reflect that they feel the experience was helpful
  for understanding a different culture from their
  own.
• Students from the Study Abroad program will be
  able to recall the major cities, foods,
  historical sites, and local customs of the cities
  they visit.

72
Other Example Outcomes

• Student government officers will demonstrate a
  working knowledge of Roberts Rules of Order.
• Students will be able to demonstrate an
  understanding of the IGETC transfer option.
• Students needing special accommodations for note
  taking, testing, etc. will demonstrate
  self-advocacy skills when dealing with
  instructors and support staff.

73
Please write or refine at least one of your
outcomes. If you have time, write additional
outcomes.
74
Meaningful Data adapted from Peggy
Maki, Ph.D.

• Collect data from different sources (e.g.,
  program samples, other samples of student work).
• Collect data you believe will be useful for
  answering the important questions you have
  raised.
• Collect data that will help you make decisions
  for continuous improvement.
• Organize reports around issues, not solely data.

75
Measurement Methods (adapted from Palomba and
Banta, 1999)

• Two basic types of learning evidence
• Direct methods information collected when
  students are required to display their knowledge
  and skills
• Indirect methods students or others reflect or
  estimate how much a student has learned (student
  doesnt actually demonstrate the learning)

76
Evidence

• Examples of direct evidence
• Student work samples
• Portfolios of student work
• Embedded assessment
• Observations of student behavior
• Juried review of student projects
• Locally developed or standardized test
• Pre and post tests
• Essay tests

• Examples of indirect evidence
• Surveys of students, alumni, employers
• Focus groups
• Exit interview
• Transfer student follow-up studies
• Transfer rate
• Academic success after transfer
• Job placement statistics
• Diversity of student body
• Percentage of students who study abroad

77
Some Questions When Selecting a Measurement
Tool

• Is your budget sufficient?
• Is the timeline do-able?
• What are your data analysis capabilities?
• Can this measurement method be fit into annual
  responsibilities?
• Will this kind of evidence lead to decisions?
• How will you document the evidence and the
  decisions derived from that evidence?

78
Example Outcome and Possible Assessment Tools

• Outcome The student will be able to prepare a
  good resume.
• Assessment
• Career instructors develop a rubric that assesses
  the content of a good resume. The rubric is used
  to assess resumes.
• Instructor develops exam questions on content of
  a good resume.
• End-of-course survey asks students to self-assess
  their ability to prepare a good resume.

79
Developing an Assessment Rubric for a Resume
80
A Rubric is Good!

• Facilitates staff dialogue regarding satisfactory
  performance.
• Creates a more objective assessment.
• Makes expectation explicit to the student.
• Encourages metacognitive skill of self-monitoring
  own learning.
• Facilitates scoring and reporting of data.

81
Rubric Resource

• Dr. Mary Allens (CSU Bakersfield) rubric
  resource site
• http//www.calstate.edu/acadaff/sloa/

82
 
83

• Please write down three different assessment
  tools and/or strategies that you will use to
  measure your top priority learning outcome.

84
One Minute Evaluation

• What is the most valuable thing you learned from
  this workshop?
• What is a question you still have?
• What is the next step that you need to take to
  implement systematic of your student services
  program?