Department Chairpersons Guide to Learning Outcomes in the Major

1
Department ChairpersonsGuide to Learning
Outcomes in the Major

• Dr. Judy Shoemaker
• Director, Office of Research Evaluation
• Division of Undergraduate Education
• University of California, Irvine
• May 21, 2008

2
Todays presentation

• Review the WASC requirements and UCIs approach
• Review the reporting form and when its due
• Explore the 5 questions on the form, with lots of
  examples and practice
• Write 3 learning outcomes
• Identify 2 assessment methods
• Answer your questions

3
WASC Re-Accreditation

• Longer time frame
• 3-stage review process
• Two core commitments
• Capacity
• Educational Effectiveness
• Increased emphasis on learning
• Use of evidence

• Western Association of Schools and Colleges, our
  regional accreditation agency
• accredits the entire University, not individual
  programs.

4
WASC Expectations for Undergraduate Majors

• Formal learning outcomes at the program level
  (major)
• Published
• Aligned with instruction
• Evidence that graduates achieves those outcomes
  (assessment of outcomes)
• A review process and use of findings
• ?UCIs 5 questions

5
The Process of Assessing Student Learning
Outcomes at UC Irvine

• Focus on undergraduate majors
• Student learning in the major as a major theme
  for the WASC review
• Assessment at the program level
• Start with status reports, Fall 2008
• Due Sept 15, 2008 judy_at_uci.edu
• Training and technical support available
• Office of Research Evaluation
• Teaching, Learning Technology Center
• Office of Institutional Research

6
Purpose To help program faculty answer the
following questions

• Are our students learning what we think is
  important?
• Are they learning what they need to succeed in
  this field or profession?
• Should our curriculum or teaching strategies be
  modified?
• Are there other techniques or additional
  resources that would help our students learn more
  effectively?

7
Important points

• The primary purpose of identifying and assessing
  learning outcomes at the program level is for
  program improvement.
• Assessment at the program level is based on
  aggregated student performance data it is not
  intended to give feedback to individual students.
• Assessment is a faculty-owned and managed
  process.
• Purpose is to provide feedback to faculty about
  how well they are meeting their program goals
  not an administrative tool for making decisions
  regarding program cuts.
• Documenting how you used assessment results to
  make program improvements is as important to WASC
  as the actual results of the assessment.

8
Answer these 5 questions

• What do you expect your majors to know and be
  able to do by the time they graduate?
• What is the relationship between the curriculum
  and the learning outcomes?
• How will you communicate information about your
  learning outcomes to your majors and potential
  majors?
• How will you determine whether or not your majors
  have achieved the learning outcomes?
• How will your results be used for the improvement
  of undergraduate education?

9
NOTE!!!

• It is NOT expected that each major will have
  outcomes or assessments in place at this time.
• Describe your plans.
• If you do not have any outcomes, describe your
  plan for identifying them.
• If you have outcomes but have not assessed them,
  describe how you will develop or select
  assessment methods and when you plan to do a
  pilot test.
• If you have an assessment grant, use that
  information to complete the form.

10
1. What do you expect your majors to know or be
able to do by the time they graduate?

• What do you want all of your majors to learn by
  the time they graduate?
• Think about the knowledge, skills, and attitudes
  that you want your majors to acquire.
• Focus on the 3 to 5 most important or essential
  outcomes.
• Use concrete action verbs to describe what
  students will learn or be able to do.

11
Types of learning outcomes

• Knowledge (cognitive)
• Discipline-specific knowledge
• Blooms Taxonomy (factual knowledge,
  comprehension, application, analysis, synthesis,
  evaluation)
• Skills (performance)
• Communication skills
• Research methods
• Information literacy
• Personal
• Attitudes (affective)
• Appreciate, respect, value

12
Examples of learning outcomes for Knowledge

• Demonstrate an understanding of culture and
  society. (anthropology)
• Describe contemporary theories concerning
  economic growth, inflation, unemployment,
  distribution of income, and international
  environment. (economics)
• Critically analyze a literary text. (English)
• Distinguish among a variety of genres of primary
  and secondary historical texts and use them
  appropriately and effectively in academic work.
  (history)

13
More examples

• Know the values, attitudes and norms that shape
  the cultural differences of peoples who live in
  countries outside the United States.
  (international studies)
• Apply physical principles to real-world problems.
  (physics)
• Use creative and critical thinking to solve
  problems related to the behavior and mental
  processes. (psychology)
• Describe the institutions and processes of the
  American political system and analyze its
  strengths and weaknesses in the 21st century.
  (political science)

14
Blooms Taxonomy of Educational Objectives
15
Take a few minutes...

• Practice writing student learning outcomes for
  your major
• Use one or more of Blooms categories
• Factual knowledge
• Comprehension
• Application
• Analysis
• Evaluation
• Synthesis

16
Examples of learning outcomes for Skills

• Communication skills (oral and in writing)
• Write persuasively, using a variety of rhetorical
  strategies (e.g., expository, argumentative,
  descriptive).
• Communicate orally in clear, coherent, and
  persuasive language appropriate to purpose and
  audience.
• Communicate results of scientific inquiries
  verbally and in writing.
• Produce historical essays that are coherent,
  cogent, and grammatically correct.

17
More examples

• Research skills
• Assemble experimental chemical apparatus, design
  experiments, and use appropriate apparatus to
  measure chemical composition and properties.
  (chemistry)
• Perform statistical analysis such as multiple
  regression and understand similar analyses
  performed by others. (economics)
• Research and conduct an investigation, consulting
  appropriate historical works for developing a
  bibliography. (history)

18
More examples

• Information literacy
• Locate relevant information and critically
  analyze its sources and validity.
• Interpersonal skills
• Be an effective member of a team, demonstrating
  appropriate professional and interpersonal
  skills.
• Develop leadership skills.

19
Examples of learning outcomes for Attitudes

• Value the scientific approach to understanding
  the natural world. (biology)
• Respect the professional code of ethics for
  pharmacy practice. (pharmaceutical science)
• Appreciate the perspective of people from
  different backgrounds and cultures.
  (anthropology)
• Gain an appreciation for the importance of
  writing, past and present. (English)

20
Take a few minutes...

• Practice writing student learning outcomes for
  your major
• Write one outcome for skills
• Communication skills
• Research skills
• Write one outcome for attitudes

21
Suggestions for engaging faculty in a discussion
about outcomes.

• The ideal graduate
• Discipline-specific organizations
• Majors at other colleges and universities
• Internal documents (catalogue, recruitment
  materials)
• Course-level goals and objectives, especially
  capstone or similar senior-level courses
• Teaching Goals Inventory
• Feedback from students, alumni, employers,
  graduate schools (UCUES)
• Graduate school admissions requirements and
  examinations

22
Take a few minutes...

• Review catalogue descriptions of selected UCI
  majors
• Write one or more learning outcome for one of the
  majors
• Identify it as knowledge/skills/attitudes
• Identify your action verbs

23
2. What is the relationship between the majors
core curriculum and your learning outcomes?

• Look at the relationship between what students
  are required to do in their courses and what
  faculty expect them to learn.
• Where are important student learning outcomes
  taught and practiced?
• Important outcomes should be introduced early and
  reinforced and further developed throughout the
  curriculum.

24
Core Curriculum for the UCI English Major
25
Core Curriculum for UCI Biological Sciences Major
26
Curriculum Map (Hypothetical Example)
T taught, P practiced, A could be used for
program assessment activities
27
Analysis of the Curriculum Map

• How well is the curriculum aligned with the
  outcomes?
• Are all outcomes taught in these courses?
• Are all courses aligned with at least one
  outcome?
• Why do you think assessment is taking place in
  only the upper division courses?
• What type of course might 195W be?

28
3. How will you communicate your learning
outcomes to your majors and prospective majors?

• Where will they be published and/or distributed?
• Suggestions
• UCI Catalogue
• Web site for dept/major
• Recruitment materials
• Distribute during orientation
• Other locations/materials
• Why is this important?

29
4. How will you determine whether or not your
majors have achieved your learning outcomes?

• Describe the assessment methods you use, or plan
  to use, to evaluate whether specific outcomes are
  being achieved.
• Consider embedding assessment in existing
  courses.
• Consider a variety of assessment methods for each
  outcome.
• Include at least one direct examination of
  student work (not just surveys, for example).
• Start with pilot projects, one outcome at a time.
• Focus on senior-level courses

30
Two types of assessment methods

• Direct measures
• Students demonstrate their learning by doing
  something, such as responding to a test question
  or writing an essay.
• Indirect measures
• Ask for perceptions or opinions about learning
  from students or other
• Self-assessment or assessment by others

31
Direct methods of assessment

• Capstone course (or similar senior courses).
• Faculty review student performance.
• Course-embedded assessment in senior-level
  courses
• Shared questions on final exams, shared paper
  assignments, evaluated by two or more faculty.
• External tests and exams.
• GRE scores, ETS Major Field Achievement Tests,
  national board certification tests, etc.
• Portfolio evaluation
• Faculty review a sample of portfolios using a
  scoring rubric
• Pre/post test evaluation
• Students take exam upon entry and again as
  seniors
• Observations of oral presentations, performances

32
Indirect methods of assessment

• Student course evaluations, particularly if they
  include course or program learning goals.
• Student surveys
• Entering students, seniors, alumni
• UC Undergraduate Experience Survey
• Exit interviews
• Seniors
• Employer surveys
• Students written self-reflections or
  self-assessment on their learning (individual
  courses, end of major, before/after completing
  major)

33
Examples of assessment methodsUniversity of
Colorado at Boulder
34
Suggestions

• Use what you have!
• Assessment instruments should be
• valid and reliable.
• interesting and meaningful to faculty.
• Include at least one direct method of assessment.
• To improve reliability, use more than one
  assessment method per outcome.
• Okay to sample students and to sample students
  work.
• As needed, develop your own scoring rubrics for
  qualitative data
• student essays and reports, team projects, oral
  presentations
• Consider pre/post designs, cross-sectional
  studies, comparing subgroups
• useful for studying the impact of new curricula

35
Take a few minutes...

• For your learning outcome, identify two different
  methods of assessment
• one direct measure of student learning
• one indirect measure of student learning
• When and where will the assessment take place?
• Will you need to develop a scoring rubric?
• Will you use any sampling (of students, of
  student work)?
• How will you motivate your students to do their
  best work on this assessment?

36
5. How will assessment results be used for the
improvement of undergraduate education?

• What is your plan for sharing the results with
  faculty?
• When will results be shared, and with whom?
• Consider annual faculty retreats, meetings of the
  undergraduate curriculum committee, etc.
• Include time for faculty to reflect on the
  results.
• Keep track of recommendations for improvements
  and how they were implemented.

37
Reported departmental changes due to assessment
activitiesUniversity of Illinois
38
Next Steps

• Next year
• Workshops, individual consultation
• Campuswide Assessment Workshop Spring 2009
• UC Undergraduate Experience Survey 2008 data
• Office of Institutional Research data
• Contact us
• Judy Shoemaker, judy_at_uci.edu
• Natalie Schonfeld, nschonfe_at_uci.edu