Understanding Student Learning Outcomes and Assessment

1
Understanding Student Learning Outcomes and
Assessment

• Bob Barr
• Andrew LaManque
• Office of Institutional Research and Planning

Breakout Session, District Opening Day, September
21, 2006
Adapted from material prepared by Fred Trapp,
Long Beach CC, the RP Group, and James O.Nichols,
University of Mississippi
2
Session Outcomes

• Describe a multi-step process for conducting
  learning outcomes assessment and understand the
  purpose of each step.
• Distinguish among levels and types of outcomes
  and assessment work.

3
What Is Student Learning Outcomes Assessment?

• An ongoing process of systematically, gathering,
  analyzing, and interpreting evidence aimed at
  enabling the institution to improve student
  learning and success.
• Requires making learning expectations explicit
  and public.
• Requires setting appropriate criteria and high
  standards for learning quality.

4
Why Do SLO Assessment?

• Improve student learning and institutional
  support for student learning and success.
• Provide for continuous reflection on improving
  teaching-learning and support environments.
• Document and display our work for students, the
  public for Regional or program accreditation.

5
How Does SLO Assessment Differ From What We
Already Do?

• Its a matter of unit of analysis and purposes.
• It is assessment of our work rather than
  students work.
• Its purpose is to enable us to improve the work
  we do (of enabling students to learn).
• Data is assembled in the aggregate rather than
  for an individual student

6
How Does SLO Assessment Differ From What We
Already Do?
Purpose
Summative
Formative
Assessment for Evaluation
Assessment for Improvement
Institutional
Level
Assessment for Grades
Assessment for Learning
Individual
7
The Assessment Process
8
Key Questions for Each Assessment Process Step

• Identify institutional mission institutional
  student learning outcomes link levels of
  outcomes
• Where and how does this program fit into the
  mission and learning outcomes of the college?
• Specify intended learning outcomes
• What learning comes of completing the course,
  program, or degree?

9
Key Questions for Each Assessment Process Step

• Identify means of assessment and criteria for
  success
• How shall we determine what students have
  learned? How good is good enough?
• Assemble the results and make meaning from them
• How well did students do relative to our
  expectations?

10
Key Questions for Each Assessment Process Step

• Use the results to redesign learning environments
• What shall we do with these results? How shall
  we redesign our learning and support environments
  to make the more effective for student learning
  and success?

11
Levels of SLO Assessment Work

• Institutional Outcomes GE outcomes or core
  competencies
• Instructional or Student Services Program
  Outcomes
• Course Outcomes

12
Step 1. Identify Mission, Institutional Outcomes
and Link Levels

• Mission of the college
• Broad statements of defining purposes and
  outcomes
• GE, program, course purpose and outcome
  statements
• Link the GE package and programs to the college
  mission and institutional outcomes
• Link course outcomes to GE and program outcomes
• Purpose of assessment ( self-study)
• How well are we accomplishing our purposes and
  how can we improve?

13
Step 1. Example of Linking the Three Levels

• College or GE Outcome
• Students will be able to speak and write
  effectively.
• Academic Program Outcome (Psychology)
• Students can speak and write effectively using
  the principles and concepts of the discipline.
• Course Outcome (Psych Measurement)
• Students can prepare a written summary and
  interpretation of standardized test results.

14
Two Types of Educational Performance Outcomes

• Student Achievement Outcomes
• Success rates
• Retention or persistence rates
• Number and rates of graduates or transfers
• Student Learning Outcomes
• What the college intends students to know, value,
  or be able to do when they have completed a
  degree, program, or course.

15
Step 2. Specify Intended Learning Outcomes

• Ideally, the curriculum and courses are designed
  by working backwards from the ultimate outcomes.
• What roles will program graduates hold and what
  must they be able to do out there in the real
  world or in upper division work?
• How will the outcomes be demonstrated or assessed
  in here?
• What skills and concepts need to be learned?

16
Step 2. Specify Intended Learning Outcomes

• Helpful hints
• Use concrete, action verbs
• See Blooms Taxonomy of Educational Objectives
• Anticipate the deliverable
• What is it that students must do to convince you
  that they have learned what you want them to
  learn?
• Focus on terminal rather than enabling outcomes
• Target key concepts or change points in the
  program.
• Focus on major aspects of learning central to the
  discipline, program, or course.

17
Example Specify Intended GE Learning Outcomes

• Be exposed to good writing examples. (in need of
  repair)
• Upon completion of a degree, the student will be
  able to
• Write effectively
• Compose a range of written products designed for
  different audiences and purposes.

18
Example Specify Intended Psychology Program
Learning Outcomes

• Hear and read about the scientific method of
  inquiry as used in the social sciences. (in need
  of repair)
• Upon completion of the program, the student will
  be able to
• Use critical thinking, skeptical inquiry, and,
  when possible, the scientific approach to solve
  problems related to behavior and mental processes.

19
Example Auto Tech Program Intended Student
Learning Outcomes

• Graduates of the Automotive Technology Program
  will be successfully employed in the field.
  (achievement outcome, not a SLO)
• Graduates of the Automotive Technology Program
  will be technically proficient.
• Employers of Automotive Technology Program
  graduates will rate them competent based on the
  education received in the program and would
  employ others.

20
Example Specify Intended Student Services
Program Learning Outcomes

• Know how to be a more prepared student. (in need
  of repair)
• Upon completion of the EOPS summer transition
  program, the student will be able to
• Develop and use a personal schedule that
  incorporates regular study time, regular study
  location and study groups to match their class
  schedule.

21
Characteristics of Good SLO Statements

• Student-focused, not teacher focused
• Focused on learning resulting from an activity,
  not the activity itself
• Focused on important, non-trivial aspects of
  learning
• Focused on skills abilities central to the
  discipline, based on professional standards of
  excellence

22
Characteristics of Good SLO Statements

• General enough to capture important learning but
  clear and specific enough to be measurable
• Focused on aspects of learning that will develop
  and endure but that can be assessed in some form
  now

23
Aligning Outcomes,Teaching-Learning and
Assessment
Outcome statements Methods of teaching/ learning Assumptions about teaching/ learning Methods of assessing Assumptions about methods
What do you expect students to know, understand, be able to do as a result of your teaching? What methods develop/ foster your primary outcomes? What assumptions underlie your methods? What is your theory of learning? What methods do you believe provide evidence of desired student outcomes? What assumptions underlie your belief that each method will elicit students demonstration of your desired outcomes?
24
Step 3. Identify Assessment Means

• Levels of assessment
• Institutional (GE, Core Abilities)
• Program (Deg, cert, or course sequence)
• Course
• Types of evidence
• Direct
• Indirect

25
Step 3. Identify Assessment Means

• Direct evidence of student learning outcomes
  (think in the aggregate)
• Passage of license, certification or subject area
  exams
• Results of general education exams
• Capstone projects or performances
• Employer or internship supervisor ratings of
  student performance
• Class assignments, projects, etc.
• Departmental exam results
• Grades (sometimes) in the aggregate

26
Step 3. Identify Assessment Means

• Indirect evidence of student learning outcomes
  (think in the aggregate)
• Student self-reflective reports
• Surveys of perception
• Alumni, graduating or current students
• Focus group interviews

27
Step 3. Identify Assessment Means

• General Types of Learning Outcomes
• Cognitive (knowledge, thinking disciplines)
• Attitudinal (values, beliefs)
• Behavioral/performance (use of knowledge, values,
  skills, or abilities)

28
Example Institutional, Program, Course
Assessment vs. Individual Student Assessment
Total down the column for individual grades.
Analyze across the row for assessment of intended
outcomes from the group.
James Nichols
29
Step 3. Identify Assessment Means

• Embedded Assignments
• Case study
• Portfolio (paper or electronic)
• Capstone project
• Senior exit essay
• Performances
• Critical situation scenario
• Debate
• Demonstration
• Oral presentation
• Online monitoring
• Tests
• Standardized test
• Homegrown test
• In class objective test
• In class essay

• Student Centered
• Exit interview
• Focus group
• Self-rating of progress
• Archival
• Library use, on-line activity
• Transcript analysis
• Transfer records
• Stakeholders Perspective
• Alumni, employee, employer surveys
• Experience of faculty who inherit our students

Weve always done this, but now used
systematically to improve student learning rather
than only to assign grades.
30
Step 3. Identify Assessment Means Cognitive
Outcomes

• Common course exams or questions
• Sampled embedded class assignments
• Portfolios

31
Step 3. Identify Assessment Means
Attitudinal/Values Outcomes

• Surveys of students, graduates
• Exam or task to explore attitudes and/or to apply
  values

32
Step 3. Identify Assessment Means Behavioral
and Performance

• Case studies
• Internships
• Simulations
• Projects
• Capstone course work
• Public performance

33
Step 3. Criteria for Success

• Identify the criteria before assessment is done
• How well ought the completers (on average)
  perform?
• Criteria are a result of dialogue, professional
  judgment and are a collective responsibility

34
Step 3. Criteria for Success -Using Rubrics

• Key Questions
• What essential elements must be present in
  student work to ensure high quality?
• What does student work look like at each level of
  mastery?

35
Step 3. Criteria for Success - Elements of a
Rubric

• Dimensions/criteria of rating (the rows)
• Levels of mastery (the columns) e.g.,
• Beginner
• Developed
• Accomplished
• Commentary describing features of work at each
  level of mastery (the cells)
• Build from strengths and weaknesses teachers see
  in student work over the years

36
Step 3. Means of Assessment - Course Exams

• Agreed upon course exam questions
• Monitor student responses to identify topics of
  weakness
• Item analysis to discern what groups are having
  difficulty with selected items
• Revisit teaching approach to the topic

37
Step 3. Means of Assessment - Embedded
Assignments

• Embedded narrative class assignments
• Evaluated for the grade mark
• Later a sample is re-inspected to
• Assess program accomplishment against expected
  outcomes
• Explore the learning process
• Agreed upon performance criteria
• The fruits of campus dialogue
• Expert judgment of the faculty
• Inter rater reliability

38
Step 3. Means of Assessment - Portfolios

• A collection of student work
• Selective or all inclusive
• Displays what students can do
• Best if student is required to reflect upon and
  evaluate their own work and learning
• Caution in program use
• Require the same work product from all students
  at pre-determined points in time.

39
Example Criteria for Success - Automotive
Technology Program

Means of Program Assessment and Criteria for
Success 2a. At the close of their final term,
90 of the graduates will be able to identify
and correct within a given period of time all of
the mechanical problems in five test cars that
have been "prepared" for the students by
Automotive Technology Program faculty. No single
automotive malfunction will fail to be identified
and corrected by more than 20 of students. 2b.
80 of Automotive Technology Program graduates
will pass the National Automotive Test. On no
subscale will students average missing 30 or
more of the items. 3. 80 of the respondents to
an Employer Survey conducted every 3 years by the
college will respond that they find graduates
competent and would employ others.
Program Intended Educational Outcomes 2.
Graduates of the Automotive Technology Program
will be technically proficient. 3.
Employers of the Automotive Technology Program
graduates will be rate them competent based on
the education received in the program.
40
Step 4. Assembling and Interpreting Assessment
Results

• Assembling Results
• Logistics of doing assessment
• Recording results
• Making meaning
• The creative tension between ought and is

41
Example Recording ResultsGraduates of the
Automotive Technology Program will be
technically proficient.

Summary of Data Collected 2a. 79 overall
success rate. Electrical system malfunction
undetected by 34 of students. 2b. Pass
rate on National Automotive Test was 83
however, on hydraulic theory subscale students
missed an average of 34 of questions.
Means of Program Assessment and Criteria for
Success 2a. At the close of their final term,
90 of the graduates will be able to
identify and correct within a given period of
time all of the mechanical problems in five test
cars that have been "prepared" for the students
by Automotive Technology Program faculty. No
single automotive malfunction will fail to be
identified and corrected by more than 20 of
students. 2b. 80 of Automotive Technology
Program graduates will pass the National
Automotive Test. On no subscale will students
average missing 30 or more of the items.
42
Step 5. Using Results to Redesign Learning
Environments

• Adjust what we teach
• Adjust how we teach
• Adjust the means of assessment
• Adjust where we put the bar of success

43
Example Auto Tech Program(Using Results
Documenting Those Actions)

Use of Results 2a. Expanded electrical
trouble-shooting component of AT 202 to include
automotive electrical systems. 2b. Modified
means of teaching hydraulic theory during AT 102
(Basic Auto Systems). 3. Added body shop
representative to Advisory Committee and are
reviewing curriculum to determine if separate
program is needed.
Summary of Data Collected 2a. 79 overall
success rate. Electrical system malfunction
undetected by 34 of students. 2b. Pass rate
on National Automotive Test was 83 however, on
hydraulic theory subscale students missed an
average of 34 of questions. 3. 90 reported
willingness to employ graduates, but only 50 of
body shops.
44
The Bottom Line on Assessment

• SLO assessment is about ourselves and our work
  because
• A Learning College is a learning organization
  composed of learners intent improving its
  achievement of its mission i.e., intent on
  improving student learning and success.