Assessing Student Learning

1
Assessing Student Learning

• John Wood Community College
• Susan Hatfield
• Winona State University
• SHatfield_at_winona.edu

2
Plan

• 1. Background
• Evolutionary Stages of Assessment Initiatives
• Levels of Assessment Who Assesses What
• Requirements
• 2. Developing a Program Level Assessment Plan
• 3. Implementing Assessment Plans
• 4. Big Mistakes in Assessment

3

• 1. Background

4

• 1. Background
• a. Evolutionary Patterns of Assessment
  Initiatives

5
Maturing Assessment
BEGINNING
PROGRESS
MATURING
INSTITUTIONAL RESPONSIBILITY
DEPARTMENT RESPONSIBILITY
6
Maturing Assessment
BEGINNING
PROGRESS
MATURING
INDIRECT MEASURES
DIRECT MEASURES
7
Maturing Assessment
BEGINNING
PROGRESS
MATURING
PROCESS MEASURES
OUTCOME MEASURES
8
Maturing Assessment
BEGINNING
PROGRESS
MATURING
INSTITUTIONAL EFFECTIVENESS
STUDENT LEARNING
9
Maturing Assessment
BEGINNING
PROGRESS
MATURING
CLASSROOM ASSESSMENT
PROGRAM ASSESSMENT
10
Evolutionary Patterns
04
97
99
00
01
02
03
98
MATURING
MAKINGPROGRESS
BEGINNING
11
Evolutionary Patterns
04
97
99
00
01
02
03
98
MATURING
MAKINGPROGRESS
BEGINNING
12
Variables Impacting Progress
L E A D E R S H I P
R E S O U R C E S
K N O W L E D G E
C O M M I T M E N T
E X I G E N C E
MATURING
MAKINGPROGRESS
BEGINNING
13

• 1. Background
• Evolutionary Patterns of Assessment Initiatives
• Levels of Assessment

14
Levels of Assessment
Classroom
Program / Discipline
College
15
Informs
Method of Analysis
Subject of Assessment
Audience
Classroom
CATS
Learning Processes
Teaching Strategy, Short Term Learning
Course Instructor
Program / Discipline
College
16
Classroom Assessment Techniques

• One-minute paper
• Muddiest Point
• 5 point quiz
• One sentence summary
• Invented Dialogues
• Directed Paraphrase
• Whats the Principle?

17
Classroom Assessment

• Formative Assessment - valuable measure of
  progress toward learning
• Short term learning
• Measures daily or weekly learning goals
• Allows for analysis of effective and ineffective
  teaching strategies
• Benefits both teachers and students

18
Classroom Assessment

• Positively impact student learning outcomes
• Do not allow for the demonstration of the depth,
  scope, integration, or internalization of
  learning across courses.

19
Informs
Method of Analysis
Subject of Assessment
Audience
Classroom
CATS
Learning Processes
Teaching Strategy, Short Term Learning
Course Instructor
Curriculum,Long Term Learning
Program / Discipline
Faculty Administration Accreditation
Program Outcome Measures
Achievement of Program / Discipline
Learning Outcomes
College
20
Program Assessment

• Based on stated program level student learning
  outcomes (gen ed, program)
• Summative - based on learning outcomes
• Program specific, not faculty specific

21
What to Assess
AA / AS
Gen Ed
Elect
AAS
Gen Ed
Program
Certificate
Program
BA/BS
General Ed
Program
Elect
22
Informs
Method of Analysis
Subject of Assessment
Audience
Classroom
CATS
Learning Processes
Teaching Strategy, Short Term Learning
Course Instructor
Curriculum,Long Term Learning
Program / Discipline
Faculty Administration Accreditation
Program Outcome Measures
Achievement of Program / Discipline
Learning Outcomes
Institutional Effectiveness
Policy, Procedure, External Perception
Student Records Data Surveys
College
External Constituencies (Legislators, Parents, Ac
creditors)
23

• 1. Background
• Evolutionary Patterns of Assessment Initiatives
• Levels of Assessment
• Requirements

24
Requirements for Assessment

• Assessment requires
• Exertion (effort)
• Intention (focus)

25
Exertion without Intention
26
Intention without Exertion
27
Intention and Exertion
28
2. Developing a Program LevelAssessment Plan
29
Definitions
30
Language of Assessment

• A. General skill or knowledge category
• GOAL
• B. Specific accomplishments to be achieved
  STUDENT LEARNING OUTCOME
• C. Activities and Assignments to help students
  learn LEARNING EVENTS
• D. The key elements related to the accomplishment
  of the outcome COMPONENTS

31
Language of Assessment

• E. The objects of analysis OBJECTS
• F. Data indicating degree of achievement
  CHARACTERISTICS
• G. Combination of data indicating relative
  degree of achievement of the outcome INDICATORS

32
Student Learning Outcomes
33
Student Learning Outcomes

• Students should be able to
• ltltaction verbgtgt ltltsomethinggtgt

34
Student Learning Outcomes

• Learner Centered
• Specific
• Action oriented
• Cognitively Appropriate

35
COMPREHENSION
EVALUATION
APPLICATION
SYNTHESIS
KNOWLEDGE
ANALYSIS
Associate Classify Compare Compute Contrast Differ
entiate Discuss Distinguish Estimate Explain Expre
ss Extrapolate Interpolate Locate Predict Report R
estate Review Tell Translate
Analyze Appraise Calculate Categorize Classify Com
pare Debate Diagram Differentiate Distinguish Exam
ine Experiment Identify Inspect Inventory Question
Separate Summarize Test
Arrange Assemble Collect Compose Construct Create
Design Formulate Integrate Manage Organize Plan Pr
epare Prescribe ProducePropose Specify Synthesize
Write
Appraise Assess Choose Compare Criticize Determine
Estimate Evaluate Grade Judge Measure Rank Rate R
ecommend Revise Score Select Standardize Test Vali
date
Cite Count Define Draw Identify List Name Point Qu
ote Read Recite Record Repeat Select State Tabulat
e Tell Trace Underline
Apply Calculate Classify Demonstrate Determine Dra
matize Employ Examine Illustrate Interpret Locate
Operate Order Practice Report Restructure Schedule
Sketch Solve Translate Use Write
36
Program Level Student Learning Outcomes

• Professionalism
• Ethics
• Professional Skills
• Theories and Theorists

37
Program Level Student Learning Outcomes

• Design
• Critique
• Plan
• Analyze
• Direct
• Demonstrate
• Execute

38
Course Level Student Learning Outcomes

• Generally related to lower cognitive levels
  (knowledge, comprehension)
• Usually assessed through the course grading
  process
• Learning is assessed in terms of level of
  correctness or accuracy of an answer

39
Student Learning Outcomes

• WARNING SIGNS
• More than one action verb
• Passive verb instead of action verb
• Multiple outcomes

40
Student Learning Outcomes
Communication
Speak
Relate
Listen
Participate
Write
41
Learning Events
42
Learning Events

• Assignments (in class and out of class)
• Feedback on practice
• Self evaluation
• Peer evaluation
• Role Play
• Pre Tests
• Simulation

43
Learning Events
GOAL
Outcome
Outcome
Outcome
Outcome
Outcome
Learning events
Object
Object of analysis
44
Learning Objects
45
Learning Objects

• There are multiple objects that can demonstrate
  student learning
• Authentic or created specifically for purposes of
  assessment

46
Learning Objects

• Abstract, advertisement, annotated bibliography,
  biography, briefing, brochure, budget, care plan,
  case analysis, chart, cognitive map, court brief,
  debate, definition, description, diagram,
  dialogue, diary, essay, executive summary, exam,
  flow chart, group discussion, instruction manual,
  inventory, lab notes, letter to the editor,
  matching test, mathematical problem, memo, micro
  theme, multiple choice test, narrative, news
  story, notes, oral report, outline, performance
  review, plan, precis, presentation, process
  analysis, proposal, regulation, research
  proposal, review of literature, taxonomy,
  technical report, term paper, thesis, word
  problem, work of art. (Walvoord / Anderson
  1998).

47
Learning Objects
Communication
Speak
Relate
Listen
Participate
Write
Eulogy
48
Learning Objects
Communication
Speak
Relate
Listen
Participate
Write
Essay
49
Components
50
Components

• Key evaluative elements
• Outcome specific, not object specific
• Nouns, not more outcomes

51
Components
GOAL
Outcome
Outcome
Outcome
Outcome
Outcome
component
Object
Evaluative elements
component
component
52
Components
Communication
Speak
Relate
Listen
Participate
Write
delivery
Process Summary
content
organization
53
Components
Communication
Speak
Relate
Listen
Participate
Write
mechanics
Letter
style
organization
54
Performance Characteristics
55
Performance Characteristics

• Scale or description for assessing each of the
  components /traits
• Two to Five-point scales for each component /
  trait

56
Performance Characteristics
Communication
Speak
Relate
Listen
Participate
Write
delivery
Distracting - Enhancing
Sales Presentation
content
Basic - Adequate - Advanced
organization
Unidentifiable- Structured - Focused
57
MIDTERM EXAM
58
Example 1

• Gather factual information and apply it to a
  given problem in a manner that is relevant,
  clear, comprehensive, and conscious of possible
  bias in the information selected
• BETTER Students will be able to apply factual
  information to a problem
• COMPONENTS
• Relevance
• Clarity
• Comprehensiveness
• Aware of Bias

59
Example 2

• Imagine and seek out a variety of possible goals,
  assumptions, interpretations, or perspectives
  which can give alternative meanings or solutions
  to given situations or problems
• BETTER Students will be able to provide
  alternative solutions to situations or problems
• COMPONENTS
• Variety of assumptions, perspectives,
  interpretations
• Analysis of comparative advantage

60
Example 3

• Formulate and test hypotheses by performing
  laboratory, simulation, or field experiments in
  at least two of the natural science disciplines
  (one of these experimental components should
  develop, in greater depth, students laboratory
  experience in the collection of data, its
  statistical and graphical analysis, and an
  appreciation of its sources of error and
  uncertainty)
• BETTER Students will be able to test
  hypotheses.
• COMPONENTS
• Data collection
• Statistical Analysis
• Graphical Analysis
• Identification of sources of error

61
The Reality of Assessing Student Learning Outcomes

• Why you need common components and criteria

62
Course
Course
Course
Course
Course
Speaking
volume
eye contact
gestures
sources
transitions
poise
style
rate
examples
verbal variety
conclusion
appearance
evidence
organization
attention getter
63
Can our students deliver an effective Public
Speech?
volume
eye contact
gestures
sources
transitions
poise
style
rate
examples
verbal variety
conclusion
appearance
evidence
organization
attention getter
64
Indicators
65
Indicators

• Percentage of students who achieve at acceptable
  levels on the components.
• Formula Performance (component 1) Performance
  (component 2) Performance (component 3)
  provides and indicator of the degree to which the
  outcome is achieved.

66
Indicators
GOAL
Outcome
Outcome
Outcome
Outcome
Outcome
Degree to which outcome is achieved
component
Object
component
indicator
component
Characteristics
67
3. Implementing Assessment Plans
68
Communicate Effectively Demonstrate Oral
Communication Skills
History Oral Report
Does not meet
Meets
Exceeds
Verbal Delivery
13
70
17
7
Nonverbal Delivery
21
72
54
Organization
14
32
Evidence
Transitions
69
Communicate Effectively Demonstrate Oral
Communication Skills
COLLEGE WIDE COMPETENCY REPORT
Does not meet
Meets
Exceeds
Verbal Delivery
20
65
15
57
30
Nonverbal Delivery
13
58
Organization
24
18
70
4. Really Big Mistakes
71
Big Mistakes in Assessment

• Assuming that it will go away
• Trying to do too much, too soon
• Not using the same components and criteria to
  assess the same outcome
• Not considering implementation issues when
  creating plans

72
Big Mistakes in Assessment

• Borrowing plans and methods without acculturation
• Demanding statistical research standards
• Doing it for accreditation instead of improvement

73
Big Mistakes in Assessment

• Confusing institutional effectiveness with
  student learning
• Making assessment the responsibility of one
  individual
• Assuming collecting data is Doing Assessment

74
Big Mistakes in Assessment

• Expecting to get it right the first time

75
Assessing Student Learning

• John Wood Community College
• Susan Hatfield
• Winona State University
• SHatfield_at_winona.edu