Developing and Assessing General Education Learning Outcomes: A Collaborative Commitment across The

1
Developing and Assessing General Education
Learning Outcomes A Collaborative Commitment
across The Institution

• Workshop at Miami Dade College
• November 21, 2005
• Peggy Maki
• PeggyMaki_at_aol.com

2
Workshop Foci

• Building a Culture of Evidence across The
  Institution
• Grounding Assessment of GE in Teaching and
  Learning
• Collaboratively Developing Learning Outcome
  Statements-- Claims about Student Learning

3

• Validating Learning Outcome Statements through
  Maps and Inventories of Educational Practice
• Designing or Selecting Valid Assessment Methods
  that Align with Students Educational Experiences
• Developing Standards and Criteria of
• Judgment

4

• Analyzing and Interpreting Results of Student
  Work
• Closing the Inquiry Loop

5
Gather Evidence
Interpret Evidence
Mission/Purposes Learning Outcome Statements
How well do students achieve our outcomes?
Enhance teaching/ learning inform institutional
decision- making, planning, budgeting
6
Your Learning Outcomes

• Articulate some GE learning outcome statements
  that align with what and how students learn in
  your programs and services
• Map GE outcome statements to assure students have
  diverse and multiple opportunities to learn
• Identify some direct and indirect assessment
  methods to capture student learning

7

• Develop some standards and criteria of judgment
  to score student work
• Identify when and where to assess and how to
  collect evidence of student learning
• Identify when and who will assess evidence of
  student learning

8

• Identify possible times across the institution
  when colleagues can come together to interpret
  results and reach consensus about ways to improve
  student learning
• After implementing changes, identify when you
  will reassess the efficacy of changes.

9
Building a Culture of Evidence
10
R.W. Emerson, Intellect, Essays (1841)

• How can we speak of the action of the mind
  under any divisions, as of its knowledge, of its
  ethics, of its works, and so forth, since it
  melts will into perception, knowledge into act?
  Each becomes the other. Itself alone is. Its
  vision is not like the vision of the eye, but is
  union with the things known.

11
How do you learn?

• List several strategies you use to learn
• ________________________________________________
  __________________________________________________
  __________________________________________________
  _____________________

12
Grounding Assessment of GE in Teaching and
Learning

• Learning is a complex process of
  interpretation-not a linear process
• Learners create meaning as opposed to receive
  meaning
• Knowledge is socially constructed (importance of
  peer-to-peer interaction)
• National Research Council. Knowing What Students
  Know, 2001.

13

• Learning involves creating relationships between
  short-term and long-term memory
• Transfer of new knowledge into different contexts
  is important to deepen understanding
• Practice in various contexts creates expertise

14

• People learn differentlyprefer certain ways
  of learning
• Deep learning occurs over timetransference
• Meta-cognitive processes are a significant means
  of reinforcing learning (thinking about ones
  thinking and ways of knowing)

15
Integration of learning and development over
time.
16
Specific Questions that Guide Assessment

• What do you expect your students to know and be
  able to do by the end of their education at your
  institution?
• What do the curricula and other educational
  experiences add up to?
• What do you do in your classes or in your
  programs or services to promote the kinds of
  learning or development that the institution
  seeks?

17
Questions (cond)

• Which students benefit from various classroom
  teaching strategies or educational experiences?
• What educational processes are responsible for
  the intended student outcomes the institution
  seeks?
• How can you help students make connections
  between classroom learning and experiences
  outside of the classroom?

18
Questions, cond

• What pedagogies/educational experiences develop
  knowledge, abilities, habits of mind, ways of
  knowing/problem solving, and dispositions?
• How are the curriculum and co-curriculum
  designed to develop knowledge, abilities, habits
  of mind, ways of knowing, and dispositions?

19

• How do you intentionally build upon what each of
  you teaches or fosters to achieve programmatic
  and institutional objectivescontexts for
  learning?
• What methods of assessment capture desired
  student learning--methods that align with
  pedagogy, content, curricular and instructional
  design?

20
Common Categories of GE Learning

• Writing
• Speaking
• Quantitative Reasoning
• Problem solving, critical thinking

21

• Leadership
• Lifelong learning
• Ethical awareness social responsibility
• Team work
• Global perspectives multiple perspectives

22
Mesa Community College Categories (AZ)

• Written and oral communication
• Critical thinking/problem solving
• Numeracy
• Arts and humanities
• Scientific inquiry
• Information literacy
• Cultural diversity

23
Inventory from MDCs Student Services Last Friday

• Writing
• Speaking
• Reading Comprehension
• Critical thinking/problem solving
• Quantitative reasoning/problem solving
• Technology
• Application of knowledge
• Proficiency in a chosen field

24

• Cultural literacy
• Globalism
• Teamwork/solo work
• Self-initiative/independence
• Social responsibility
• Ethical awareness
• Leadership
• Ability to adapt to environments/changes

25
Categories under which students learn and develop

• List several categories under which you believe
  students learn or develop as a result of MDCs GE
  program? _________________________________________
  __________________________________________________
  __________________________________________________
  __________________________________________________
  _

26
Inventory Based on Nov. 21 Cross-Campus Group Work

• Writing
• Speaking
• Listening
• Quantitative reasoning, including ability to
  assess and evaluate
• Critical thinking
• Ethical awareness
• personal/social responsibility, including
  cultural dimensions
  

27

• Environmental ethics
• Computer/information literacy
• Cultural Literacies
• Problem solving
• Problem posing
• Financial responsibility
• Workforce skills
• Knowledge about self, others, community, world

28

• Leadership
• Active learning (self)
• Teamwork
• Ability to link across the curriculum and
  experiences
• Time management
• Global perspectives/diversity
• Cultural sensitivity
• Interpersonal skills
• Adaptability

29

• Scientific thinking/methods
• Appreciation of the arts, including a global
  perspective
• Life skills

30
Collaboratively Developing Learning
Outcome Statements

• Learning outcome statements describe what
  students should be able to demonstrate,
  represent, or produce based on how and what they
  learn at the institution through multiple,
  varied, and intentional learning opportunities.

31

• Rely on active verbs, such as create, compose,
  calculate, develop, build, evaluate, translate,
  etc., that target what we expect students to be
  able to demonstrate
• Emerge from what we value and how we teach or
  students learn that is, they emerge from our
  educational practices and are developed through
  consensus
• Are written for a course, program, service, or
  the institution

32

• Can be mapped to the curriculum and co-curriculum
• Can be assessed quantitatively or qualitatively

33
Levels of Learning Outcome Statements
34
Distinguishing between Objectives and Outcomes

• Objectives state overarching expectations such
  as
• Students will develop effective oral
• communication skills.
• OR
• Students will understand different
• economic principles.

35
Mesa Outcomes under Arts and Humanities

• Demonstrate knowledge of human creations
• Demonstrate an awareness that different contexts
  or world views produce different human creations
• Demonstrate an understanding and awareness of the
  impact that a piece has on the relationship and
  perspective of the audience
• Demonstrate an ability to evaluate human creations

36
Capital Community College (CT)

• Communicate effectively
• Reason scientifically and or quantitatively
• Think critically
• Develop a global perspective
• (See handout)

37
EthicsStudents should be able to

• Identify and analyze real world ethical problems
  or dilemmas, and identify those affected by the
  dilemma.
• Describe and analyze the complexity and
  importance of choices that are available to the
  decision-makers concerned with this dilemma

38

• Articulate and acknowledge their own deeply
  held beliefs and assumptions as part
  of a conscious value system
• Describe and analyze their own and others
  perceptions and ethical frameworks for
  decision-making
• Consider and use multiple choices, beliefs, and
  diverse ethical frameworks when making decisions
  to respond to ethical dilemmas or problems.
• California State University Monterey Bay
  University Learning Requirements, 2002

39
Example from ACRL

• Literate student evaluates information and its
  sources critically and incorporates selected
  information into his or her knowledge and value
  system.
• ONE OUTCOME Student examines and compares
  information from various sources in order to
  evaluate reliability, validity,accuracy,
  timeliness, and point of view or bias.

40
Quantitative Literate Graduates according to MAA
Should be Able to

• 1. Interpret mathematical models such as
  formulas, graphs, tables, and schematics, and
  draw inferences from them.
• 2. Represent mathematical information
  symbolically, visually, numerically, and
  verbally.
• 3. Use arithmetical, algebraic, geometric, and
  statistical methods to solve problems.

41

• Estimate and check answers to mathematical
  problems in order to determine reasonableness,
  identify alternatives, and select optimal
  results.
• Recognize that mathematical and statistical
  methods have limits.
• (http//www.ma.org/pubs/books/qrs.html)
• The Mathematics Association of America
  (Quantitative Reasoning for College Graduates A
  Complement to the Standards, 1996). See also
  AMATYC draft, 2006.

42
Writing

• See NCTA Guidelines
• See WPA Outcomes in attachments for outcomes at
  the end of the first year of writing

43
Ways to Articulate Outcomes

• Adapt from professional organizations
• Derive from mission of institution/program/departm
  ent/service
• Derive from students work

44

• Derive from ethnographic process
• Derive from exercise focused on listing one or
  two outcomes you attend to
• Consult taxonomies

45
Taxonomies That May Help You Develop Outcome
Statements

• Blooms Taxonomycognitive, psychomotor,
  affective
• Webbs Taxonomydepth of knowledge
• Shulmans Taxonomytable of learning

46
Depth of Knowledge (Webb)

• Recall and recognition
• Processing skills and concepts
• Strategic thinking
• Extended thinking (complex reasoning, planning,
  design)

47
Dimensions of Knowledge

• Facts
• Proceduresseries of step-by-step actions and
  decisions that result in the achievement of a
  task
• Processesflow of events or activities that
  describe the big picture

48

• Conceptsclass of items, words, or ideas
• known by a common name
• Principlesguidelines, rules, parameters
• Metacognitiveknowledge of ones own cognition

49
Shulmans Taxonomy

• Engagement (active learning)
• Knowledge and understanding
• Performance, practice, or action (act in and on
  the world)
• Reflection and critique (cease action to discover
  or make progress)

50

• Judgment and designconsider contexteven
  restraints
• Commitment and Identitymove inward and connect
  outward
• http///www.carnegiefoundation.org/elibrary/docs/p
  rintable/
• making_differences.htm

51
Exercise Write one or two GE learning outcome
statements under a category of learning

• __________________________________
• __________________________________
• ___________________________________

52
Exercise

• How well do your learning outcome statements
  meet the criteria for well-written outcome
  statements (see handout)?

53
Validating Learning Outcome Statements
through Maps and Inventories of Practice

• Reveal how we translate outcomes into educational
  practices offering students multiple and diverse
  opportunities to learn
• Help us to identify appropriate times to assess
  those outcomes
• Identify gaps in learning or opportunities to
  practice

54

• Help students understand our expectations of them
• Place ownership of learning on students
• Enable them to develop their own maps or learning
  chronologies

55
Collaborative Development of A
Curricular-Co-Curricular Map
56
Inventories of Educational Practice

• Provide in-depth information about how students
  learn along the continuum of their studies
• Identify the range of educational practices and
  assessment experiences that contribute to
  learning outcomes (See handouts)

57
Exercise How will you use maps and
inventories?

• Discuss how you will go about the process of
  developing a curricular or curricular-co-curricula
  r map and how you will label peoples entries
• Discuss how you might use inventories of
  educational practices

58
Designing or Selecting Valid Assessment Methods
that Align with Students educational Experiences

• Every assessment is also based on a set of
  beliefs about the kinds of tasks or situations
  that will prompt students to say, do, or create
  something that demonstrates important knowledge
  and skills. The tasks to which students are asked
  to respond on an assessment are not arbitrary.
• National Research Council. Knowing what
  students know The science and design of
  educational assessment . Washington, D.C.
  National Academy Press, 2001, p. 47.

59
Design or Select Assessment Methods that
Prompt Students to

• Transfer, integrate, apply, synthesize
• Value interdependence among courses and
  experiences
• Re-use and re-configure what they have learned
  (even to re-position their understanding)
• Self-reflect on their emerging learning

60
For example, do students

• Apply business principles to a student-run
  organization?
• Apply principles of effective writing to a
  proposal for an independent study or project?
• Explore multiple perspectives in solving a campus
  issue or problem?
• Self-reflect on principles underlying their
  actions or decisions?

61
Assumptions Underlying Teaching
Actual Practices
Assumptions Underlying Assessment Tasks
Actual Tasks
62

• Inference Drawing
• Validity of the Method

63
What Tasks Elicit Learning You Desire?

• Tasks that require students to select among
  possible answers (multiple choice test)?
• Tasks that require students to construct answers
  (students problem-solving and thinking
  abilities)?
• Question Consider the contexts for each of
  these kinds of tasks in your work

64
When Do You Seek Evidence?

• Formativealong the way?
• For example, to ascertain progress
• or development
• Summativeat the end?
• For example, to ascertain mastery level of
  achievement

65
Direct Methods of Assessment

• Focus on how students represent or demonstrate
  their learning (meaning making)
• Align with students learning and assessment
  experiences
• Align with curricular-and co-curricular design
  verified through mapping

66

• Invite collaboration in design (faculty,
  students, tutors?)

67
Standardized Instruments

• Psychometric approachvalues quantitative methods
  of interpretation
• History of validity and reliability
• Quick and easy adoption and efficient scoring
• One possible source of evidence of learning

68
Do Not Usually Provide

• Evidence of strategies, processes, ways of
  knowing, understanding, and behaving that
  students draw upon to represent learning
• Evidence of complex and diverse ways in which
  humans construct and generate meaning
• Highly useful results that relate to pedagogy,
  curricular design, sets of educational practices

69
Authentic, Performance-based Methods

• Focus on integrated learning
• Directly align with students learning and
  previous assessment experiences
• Provide opportunity for students to generate
  responses as opposed to selecting responses
• Provide opportunity for students to reflect on
  their performance

70
Do Not Provide

• Immediate reliability and validity (unless there
  has been a history of use)
• Usually do not provide easy scoring unless
  closed-ended questions are used.

71
Direct Methods across Students Learning
Chronology

• On-line tools
• Critical events
• Assemblage of learning objects
• Virtual learning environments or situations
  (including chatrooms and resource rooms)

72

• Scenarios
• Storyboards
• Self-directed group projects
• Magic box
• Personal and annotated websites

73

• Log book or journal tasks that explore an
  issue over time
• Event analysis
• Video clips
• Case studies over time as students move through
  courses and educational experiences

74

• Externally or internally juried reviewed projects
• Oral defense
• E-portfolio
• Aristotles finger exercises
• Interpreting visual material or data

75

• Representation concept mapping or problem
  solving (3-D)
• Practice of Artists Machetes
• Mining data
• Students drawings and modelsperceptual enhances
  understanding, analysis. and analytical ability

76

• Chronological tasks that prompt students to
  stretch over time
• Draw on knowledge/understanding to solve problem
  in a different context
• Problems with solutions Are there other
  solutions?
• Team-based projects
• Self-reflections

77

• Magnify or reduce to seek wider implications and
  relationships (a la Lewis Thomas)
• Professional/disciplinary practices
• Embedded assignments

78

• Performance on national licensure examinations
• Locally developed tests

79
Indirect Methods of AssessmentThan Can Be
Combined with Direct Methods

• Programs or Courses selected by students
• Focus groups (representative of the population)
• Interviews (representative of the population)
• Surveys

80
Other Sources of Information that May Be Useful
in Your Interpretation

• CSSE results
• Grades
• Participation rates or persistence in support
  services

81

• Course-taking patterns
• Students majors
• Transcript analyses or audits (co-curricular
  transcript?)

82
Exercise

• Using the handout, determine the degree of
  alignment of the direct and indirect methods you
  may use to asses your outcome statements.

83
Developing Standards and Criteria of Judgment

• A set of criteria that identifies the expected
  characteristics of a text and the levels of
  achievement along those characteristics. Scoring
  rubrics are criterion-referenced, providing a
  means to assess the multiple dimensions of
  student learning.
• Are collaboratively designed based on how and
  what students learn (based on curricular-co-curric
  ular coherence)

84

• Are aligned with ways in which students have
  received feedback
• (students learning histories)
• Students use them to develop work and to
  understand how their work meets standards (can
  provide a running record of achievement).

85

• Raters use them to derive patterns of student
  achievement to identify strengths and weaknesses
• Analytic
• Holistic

86
Interpretation through Scoring Rubrics

• Criteria descriptors (ways of thinking, knowing
  or behaving represented in work)
• Creativity
• Self-reflection
• Originality
• Integration
• Analysis
• Disciplinary logic

87

• Criteria descriptors (traits of the performance,
  work, text)
• Coherence
• Accuracy or precision
• Clarity
• Structure

88

• Performance descriptors (describe well students
  execute each criterion or trait along a continuum
  of score levels)
• ExemplaryCommendable Satisfactory-
  Unsatisfactory
• ExcellentGoodNeeds ImprovementUnacceptable
• ExpertPractitionerApprentice--Novice

89
Development of Scoring Rubrics

• Emerging work in professional and disciplinary
  organizations
• Research on learning (from novice to expert)
• Student work

90

• Interviews with students
• Experience observing students development

91
Consider the following guidelines as you develop
a scoring rubric for one or more of our outcomes

• Identify the purpose of the rubricfor student
  feedback, for justifying a grade, for
  program-level understanding about student
  learning
• Identify the overall formatanalytic or holistic?

92

• Identify the full range of criteria you will
  assess with indicators for these criteria
• Identify the performance descriptors within each
  cell identify leveled performance

93
Pilot-testing the Scoring Rubric

• Apply to student work to assure you have
  identified all the dimensions with no overlap
• Schedule inter-rater reliability times
• -independent scoring
• -comparison of scoring
• -reconciliation of responses
• -repeat cycle

94
Analyzing and Interpreting Results

• Seek patterns against criteria and cohorts
• Build in institutional level and program
• level discourse
• Tell the story that explains the results
• triangulate with other data, such as
• CSSE or participation rates

95

• Be able to aggregate and disaggregate data to
  guide focused interpretation
• Collectively determine what you wish to change

96
Examples of Changes

• Increased attention to weaving experiences across
  the institution, a program, or a department to
  improve student achievement
• Changes in advising based on assessment results
• Closer monitoring of student achievement--tracking

97

• Faculty and staff development to learn how to
  integrate experiences that contribute to improved
  student learning
• Changes in pedagogy and curricular and
  co-curricular design
• Development of modules to assist learning use of
  technology self-paced learning, supplemental
  learning

98
Closing the Inquiry Loop to Learn

• Implement agreed upon changes
• Re-assess to determine efficacy of changes
• Focus on collective effortwhat we do and how we
  do it

99
Structures

• Assessment Committees at the institution and
  department or program levels
• Development of task forces to assume
  responsibilities

100
(No Transcript)
101
Communication Collaborative Interpretation

• Disciplinary work groups
• Cross-disciplinary work groups
• Formal opportunities to share program-level
  findings at the institution-level opportunities
  to share institution-level findings at the
  program-level

102
Communication Decision-making Bodies

• Planning (short- and long-term planning)
• Budgeting
• Decision-making
• Allocation of Resources

103
Human, Financial, Technological Support

• Grad students or part-time support to assist
  with development of methods or research on
  methods, collection or analysis
• Analysis of results
• Faculty and staff development or resources to
  support efforts
• Development of technology to house results or to
  draw from existing data

104
Exercise

• Describe the structures, processes,
• decisions, and channels and forms of
• communication that currently exist at
• MDC, as well as your ideas for
• deepening the commitment to
• assessment (see handout).

105
What and how students learn depends to a major
extent on how they think they will be assessed.
John Biggs, Teaching for Quality Learning at
University What The Student Does. Society for
Research into Higher Education Open University
Press, 1999, p. 141.
106
Works Cited

• Biggs, J. (1999). Teaching for Quality
  Learning at University What The Student Does.
  Society for Research into Higher Education Open
  University Press, 1999, p. 141.
• Maki, P. (2004). Assessing for Learning
  Building a Sustainable Commitment Across the
  Institution. Sterling, VA Stylus Publishing,
  LLC, and the American Association for Higher
  Education.
• National Research Council. (2001). Knowing What
  Students Know The Science and Design of
  Educational Assessment. Washington, D.C.
  National Academy Press