The Assessment Toolbox An Overview

1
The Assessment ToolboxAn Overview

• Marty Smith Sharpe
• Old Dominion University
• Institutional Research and Assessment
• msharpe_at_odu.edu

2
Learning Outcomes of this Session

• Participants will be able to
• Articulate the rationale for and purposes of
  student learning outcomes assessment
• Respond to questions and challenges to conducting
  outcomes assessment
• Identify the characteristics of a good learning
  outcome using the SMART model
• Understand how curriculum mapping can be used to
  link outcomes to learning opportunities

3
Learning Outcomes of this Session

• Participants will be able to
• Identify broad categories of outcomes
  measures/methods
• Appreciate importance of matching the measure to
  the domain being assessed
• Articulate the steps of getting starting doing
  outcomes assessment in academic programs

4
What is Assessment?

• Assessment is the systematic collection, review,
  and use of information about educational programs
  undertaken for the purpose of improving student
  learning and development.

5
Drivers of Assessment

• A revolution in education the learning-centered
  paradigm
• Culture of continuous improvement and
  evidence-based decision making
• Calls for accountability
• Federal requirements for regional accreditation
• Disciplinary and professional accreditation
• Supports for faculty and students to improve
  their performance

6
Reasons for Assessment

• Formative
• Improve teaching and learning (or service and
  satisfaction)
• Used while learning
• is taking place
• Focus on feedback and adjustment

• Summative
• Document learning or service and satisfaction
• Occur at the end of the course or service period
• Focus on sum/total, with little feedback

7
SACS/Commission on Colleges Institutional
Effectiveness

• SACS Core Requirement 2.5 The institution
  engages in ongoing, integrated, and
  institution-wide research-based planning and
  evaluation processes that
• (1) incorporate a systematic review of
  institutional mission, goals, and outcomes
• (2) result in continuing improvement in
  institutional quality and
• (3) demonstrate the institution is effectively
  accomplishing its mission.

8
SACS/Commission on Colleges Institutional
Effectiveness

• Comprehensive Standard 3.3 - Institutional
  Effectiveness The institution identifies
  expected outcomes, assesses the extent to which
  it achieves these outcomes, and provides evidence
  of improvement based on analysis of the results
  in each of the following areas
• educational programs, to include student learning
  outcomes
• administrative support services educational
  support services
• research with its educational mission, if
  appropriate and community/public service within
  its educational mission, if appropriate

9
Learning can and often does take place without
the benefit of teaching and sometimes in spite
of it but there is no such thing as effective
teaching in the absence of learning. Teaching
without learning is just talking.
10
Student Learning Outcomes

• Essential in an assessment system
• Your destination, not your path
• What you aim to accomplish in terms of student
  learning and achievement at the completion of
  your program

11
Student Learning Outcomes Educational Outcomes,
Competencies

• In a general sense, student learning outcomes
  are
• Knowledge
• Skills
• Attitudes
• Habits of mind
• . that students take with them from a learning
  experience.

12
Student Learning Outcomes

• Become the basis for measuring student learning
  outcomes - assessment
• May begin generically, globally with broad
  objectives to be further broken down
• What we want to engender in our students what
  we want our students to know, do, and be
• Ideally, specific and measurable

13
I WAS HOPING FOR DIRECT DEPOSIT
SO, YOUD STAY HOME AND WED MAIL YOU CHECKS.
14
Purpose of Student Learning Outcomes Assessment

• To communicate to ourselves
• What we intend for students to learn
• The content and sequence of learning
• Whether students have gained
• appropriate skills
• attitudes
• and/or knowledge
• How successful a learning activity has been

15
Purpose of Student Learning Outcomes Assessment

• To communicate to students what we intend for
  them to learn
• So they can organize their efforts toward
  accomplishing the desired behavior
• So they can assess their own performance

16
Purpose of Student Learning Outcomes Assessment

• To communicate to other interested parties
  including professional community, accrediting
  agencies and the public
• The purpose and degree of success of our
  activities
• Our commitment to engage in the process of
  improvement

17
Common Myths and Misconceptions
18
Common Questions and Answers

• How can I do this and complete all of my other
  teaching and other duties?

Building the Scholarship of Assessment (Banta,
2002)
19
Common Questions and Answers

• Isnt this just an accountability problem for
  someone else to take care of besides the faculty?

Focus on student learning . has to involve
faculty
20
Common Questions and Answers

• I know how well my students are performing
  because I teach them and grade their work!!!

In God we trust ..
.. all others bring data
21
Common Questions and Answers

• Why not use grades in courses?

Grades in courses often include more than student
learning
Single grader
Results not specific enough for most program
improvements
22
Common Questions and Answers

• We have a capstone course isnt that
  assessment?

If used properly, a wonderful vehicle for
assessment
but it is only a starting point
23
Common Questions and Answers

• We do surveys and course evaluations to assess
  our students satisfaction isnt that sufficient
  assessment?

Surveys are good measures of confidence, not
competence
Surveys are good measures of satisfaction, not
student learning
24
Common Questions and Answers

• What about academic freedom?

Faculty have a responsibility to cover common
learning outcomes
Faculty can add learning outcomes and decide how
to teach common learning objectives most
effectively
Assessment is not a form of faculty evaluation .
. .
25
Jim Nichols Grade Book AnalogyHow you can use
student performance to assess course and/or
program strengths and weaknesses? - a way to
conceptualize the process
Common Questions and Answers Why Not Course
Grades?
26
Typical Grade Book
  Grading Dimensions Grading Dimensions Grading Dimensions Grading Dimensions Grading Dimensions Grading Dimensions  
Student  Paper 1  Midterm Attendance Participation Paper 2  Final    Student Performance
1            
2            
3            
4            
5            
6            
7            
8            
9            
10            
27
Typical Grade Book

• Rows represent individual students.
• Dimensions across the top most typically
  represent measures of student performance at
  various points in time and/or results of
  different measures or instruments.
• Student performance at the end of the semester is
  summarized and a course grade results.

28
Typical Grade BookFocus on the Rows

• Student performance may vary across the columns,
  but, in general, is not important in summarizing
  individual student performance in each row.
• If all/most students pass the course, presumably,
  the targeted learning objectives have been met,
  BUT have they?
• Need to focus on columns

29
Focus on the Columns Learning Objectives

• Analyzing the row data results in a summary of
  individual student performance.
• HOWEVER, analysis of the column achievement
  should identify strengths and weaknesses of the
  course which are not apparent when looking at
  student summary data only.

30

Substituting Student Learning Outcomes
  Student Student Learning Outcomes Student Learning Outcomes Student Learning Outcomes Student Learning Outcomes Student Learning Outcomes Student Learning Outcomes   Student Performance
  A B C D E F
1 1  0   1 1  1  1
2  0 1  1 1 1  1 
3 1  1 1 1  0 1
4 0   0 1 1 1 1
5 1  1 1 1 1 1
6 1   0  1 1 1  1
7 1  1 0 1 1  1
8 1  0  1 1 1  1
9 1   1  1 1  1  1
10 1   1  1 1  1  1
Outcome Performance  
84 84 84 67100 84 84 84100100
80 60 90 100 90 100
PTA Program Assessment

30
31
Substituting Program Student Learning Objectives
as Column Dimensions

• Unit of analysis changes from an individual to
  the academic program.
• Achievement of each objective may be measured at
  one or more times and in one or more ways during
  the course of the program.

32
Typical Grade Book Summary

• Rows focus on...
• Individual student performance
• Columns focus on
• Course student learning objectives
• Program student learning objectives
• Once particular strengths and weaknesses are
  identified, analysis of how the course or
  curriculum is delivered should ensue.

33
Unit of Analysis

• . Is not the faculty member
• Is not the student
• IS the PROGRAM

34
Developing Student Learning Outcomes

• Input from all stakeholders
• Process of consensus and expert opinion
• Faculty
• Professional Practice
• Accrediting and Professional Organization

35
Characteristics of a Learning Objective

• It is always expressed in terms of the learner
• It is precise and supports only one
  interpretation
• It describes an observable behavior
• It specifies conditions under which the behavior
  is performed
• It specifies criteria for accomplishment

36
SMART Model

• Specific
• Measurable
• Attainable
• Relevant
• Time - Limited

37
SMART

• Specific
• It's difficult to know what students should be
  doing if they are to pursue the goal to Write
  better." It's easier to recognize "Write an
  essay."

38
SMART

• Measurable
• It's difficult to know what the scope of "Write
  an essay" really is. It's easier to appreciate
  that effort if the objective is "Write a 300-500
  word argumentative essay with fewer than 10
  mechanical errors."

39
SMART

• Attainable The student must know what
  constitutes a condition or the objective is not
  attainable. The objective must fit in with the
  scope of the work in the class/workshop/activity.

40
SMART

• Relevant
• If the student sees no connection between the
  objective and his/her own goals and the
  assignment, the objective will not be relevant.

41
SMART

• Time-Limited
• The learning objective must include the time
  frame in which the work is to be done. As the
  final exam, the students will be able to write in
  90 minutes a 300-500 word argumentative essay
  with fewer than 10 mechanical errors.

42
ABCD Model Questions a good objective answers

• Audience Who will be performing the behavior?
• Behavior What behavior should the learner be
  able to do?
• Condition Under what conditions do you want the
  learner to be able to do it?
• Degree How well must it be done?

43
Learning Outcomes B.F.A. Acting

• Demonstrated ability to project one's self
  believably in word and action into imaginary
  circumstances, evoked through improvisation or
  text. Demonstrated ability to create characters
  convincingly from different theatrical genres and
  styles.
• A developed technique for analyzing the specific
  tasks required in performing varied characters
  from dramatic literature. Understanding of the
  specific demands of the acting styles of the
  major periods and genres of dramatic literature.

44
Learning Outcomes B.F.A. Acting

• Effective use of voice and body. The students
  will be trained in the basic stage techniques of
  the actor for theatres of varying size as well as
  for media production.
• Demonstrated comprehension of the basic business
  procedures of the actor's profession, including
  audition procedures, resumes, agents, etc.
• Related Skills An overview understanding of
  make-up materials and techniques. An overview
  understanding of technical theatre practice.

45
Linking Curriculum to Outcomes Curriculum Mapping
46
Outcomes ? Curriculum Mapping

• Curriculum Mapping should have
• Program learning objectives .
• linked to courses ..
• linked to particular measures in courses or
  outside .
• Analysis of measures should lead to course and
  curricular improvement

47
Outcomes Model
47
47
48
Curriculum Mapping Program Level
Program Objective PTA 100 PTA 150 PTA 200 PTA 220 PTA 230
1. Review the plan of care established by the physical therapist prior to initiating patient/client intervention.          
2. Provide safe interventions as directed in the plan of care and supervised by the physical therapist.          
3. Provide effective instruction to the patient/client and others to achieve the goals and outcomes as described in the plan of care.          
4. Collect data to quantify the patients/clients response to interventions as directed and supervised by the physical therapist.          
49
Put Sarahs new picture here UConn
Handout
49
50
Curriculum Mapping Course Level
Course Objectives Pro. Obj. Domain Instruct. Method Assess Method
Match the patients/clients goals to the selected interventions. 1 knowl. Lecture Case study  Rubrics, exam question
Demonstrate proper and safe body mechanics while instructing a fellow student in use of crutches on stairs. 2 skills Case study Lab activity Rubrics
Identify methods to enhance patient/client adherence to instruction. 3 knowl. skills attitudes  Lecture Case study  Rubrics Exam question
Explain the importance of data collection in the problem-solving process. 4 knowl. skills Lecture Case study Rubrics Essay Exam question
51
AssessmentMethods
52
Assessment of Student Learning Outcomes

• Learning outcomes provide the basis for assessing
  whether or not students completing your program
  have attained the agreed upon outcomes
• Program has to organize itself in such a way that
  educational outcomes are captured at various
  points in time
• Assessment results provide information to support
  program improvement

53
Assessment Methods

• Assessment tools must be compatible with the
  domain being assessed
• (1) knowledge typically assessed using
  appropriate written and oral examinations as well
  as direct observation
• (2) skills typically assessed by direct
  observation of student performances (written,
  oral, visual)

54
Assessment Methods

• Assessment tools must be compatible with the
  domain being assessed
• (3) attitudes typically assessed by interviews,
  observations, or evaluations with peers,
  supervisors, faculty, and co-workers and their
  families

55
Models of Assessment

• Standards (competency) based set a standard and
  assess the extent to which students achieve it
• Benchmarking assess how well your students
  achieve compared to other students at peer
  institutions
• Value-Added assess your students at entry and
  at a later time (end of second year graduation)
  and compare the differences
• Longitudinal assess students in each successive
  class and compare the differences between classes

56
Good Assessment Practices

• Focus on teaching and learning (or service and
  satisfaction) rather than on assessment
• Set clear expectations
• Minimize the burden of assessment
• Start small
• Start with successes
• Use assessment results for program improvement

57
Good assessments

• Measure what we intend to measure (validity)
• Give us useful, reasonably accurate information
• Yield results specific to the learning outcomes
  so that improvements can be made

58
Good assessments

• Yield consistent results across administrations
  (reliability)
• Are systematized
• Are ethical and protect the privacy and dignity
  of those involved
• Are cost effective, yielding value that justifies
  the time and expense we put into them

59
Course Embedded Assessments

• Developed and implemented within the context of
  individual course(s)
• Required courses in the major, capstone course
  ideal
• Completion of assessment measures part of course
  grades
• Contrast to separate session assessment issues
  of motivating students to complete assessment

60
Common Assessments

• Tests
• Rubrics Prompts
• Portfolios
• Surveys
• Focus Groups

61
Presumption is that, at all times, efforts are
being made to ensure that measures are valid and
reliable
62
Sample Assessment Tools Tests

• Direct measure of student learning
• Typically quantitative essay questions may be
  qualitative
• Objective if multiple choice / subjective if
  essays
• Published / normed vs. locally developed

63
Sample Assessment Tools Rubrics Prompts

• Direct measure of student learning
• Qualitative
• Subjective professional judgment required
• Multiple raters calibrated
• If appropriately developed, can result in
  consistent results (inter-rater reliability)

64
Sample Checklist
A Checklist Rubric for Critical Thinking
Clearly stated objective.
Ability to present supporting evidence.
Logical consistency.
65
Sample Rating Scale
A Rating Scale for Critical Thinking A Rating Scale for Critical Thinking A Rating Scale for Critical Thinking A Rating Scale for Critical Thinking A Rating Scale for Critical Thinking
SA A D SD
The writers purpose or position is clear and is the primary controlling force throughout the paper.
The writer fully develops clear ideas presented in support of the essays purpose..
The writer comes to well-reasoned conclusions..
66
Sample Descriptive Rubric
Rubric for Critical Thinking (ability to present supporting evidence)
Exceeds Standard The writer fully develops clear ideas presented in support of the essays purpose..
Meets Standard The writer develops ideas in support of the essays purpose..
Fails to Achieve Standard The writer fails to develop ideas in support of the essays purpose..
67
Portfolio

• What is a portfolio?
• A collection of artifacts to demonstrate that one
  has accomplished that which he/she said he/she
  would accomplish.
• A type of assessment in which students work is
  systematically collected and carefully reviewed
  for evidence of learning and development.

68
Sample Assessment Tools Portfolios

• Direct measure of student learning
• Qualitative
• Subjective professional judgment required
• A rubric can be used to evaluate a portfolio

69
Benefits of Using Portfolios

• Provide rich opportunities for reflection by both
  student and faculty.
• Allow a longitudinal (value added) view of
  student learning.
• Provide cross or co-curricular assessment of
  objectives.
• Allow for automatic tri-angulation by provide
  multiple sources of data.
• Helps students to see the interrelationship of
  the work they are doing (and its
  meaningfulness).
• Potential archive for future (professional
  portfolio).

70
Sample Assessment Tools Surveys

• Indirect measure of student learning
• Typically quantitative open-ended questions may
  be qualitative
• Subjective require professional judgment

71
Sample Assessment Tools Focus Groups

• Indirect measure of student learning
• Qualitative
• Subjective require professional judgment

72
Tasks for Planning Focus Groups

1. Define (or learn) the goals of the project and
   write research questions
2. Select participants and subsets of participants
3. Design the Moderators Guide
4. Select and Train the Moderators
5. Conduct the Focus Groups
6. Analyze the Data and Report Results

73
Assessing Student Learning Outcomes Getting
Started

1. Draft a list of student learning outcomes
   utilizing faculty, advisory committees, and other
   stakeholders

74
Outcomes AssessmentGetting Started

• Develop a curriculum map
• Identify where in the curriculum, down to the
  course and unit, where the learning experiences
  are that engender the outcome
• Identify measures within each learning experience
  which assess each learning outcome . . .

75
Outcomes AssessmentGetting Started

1. . Link the learning outcome to the measure, and
   whenever possible, use the measure in the context
   of the course (course-embedded assessment)
2. Organize results of the various assessment
   according to the individual learning outcomes and
   analyze, review, and identify weaknesses

76
Outcomes AssessmentGetting Started

1. Track the weaknesses in student learning outcomes
   back through the curriculum map to the associated
   learning experiences
2. Identify curricular improvement and implement
3. Continue assessing, reviewing, improving

77
Tools to Support Assessment
78
http//www.weaveonline.net/subscriber/odu/
79
References

• Banta, T. W. (2002). Building a scholarship of
  assessment. San Francisco Jossey-Bass.
• Nichols, J. IEA Associates Website
  http//www.iea-nich.com/
• Suskie, L. (2004). Assessing student learning A
  common sense guide. Bolton, MA Anker Publishing
  Co., Inc.

80
The Assessment ToolboxAn Overview

• Marty Smith Sharpe
• Old Dominion University
• Institutional Research and Assessment
• msharpe_at_odu.edu