Title: CS87: Evaluating Program Effectiveness Using Student Assessment Data: Basic Steps for Beginners
1CS-87 Evaluating Program Effectiveness Using
Student Assessment Data Basic Steps for
Beginners
- Retta E. Poe, PhD, Western Kentucky University
- Dennis K. George, PhD, Western Kentucky
University - SACS COC Annual Meeting, Orlando, FL.
- Monday, 12/11/2006, 215 -315 PM
- Sun C, Convention Center
2Goals for this workshop
- Clarify distinctions between student and program
assessment - Clarify distinctions between course and program
outcome statements - Show how data from student assessment products
may be used for program assessment - Provide practice in modifying typical student
assessment methods to yield program assessment
data
3Why Do It?
- Kill two birds with one stone, or in other words,
earn double mileage - Use a process with which faculty are already
familiar (student assessment) - Save ourselves some work!
4Student vs. Program Assessment
- Similarities
- Unit of measurement (the student or some product
produced by the student) - Tools/Instruments of assessment
- Differences
- Overall purpose of assessment
- Primary utilization of results
5Student Assessment
- Purpose Students assessed with respect to
mastery of material presented ensure capability
or competency of each student - Assessment tools written examinations, essays,
papers, presentations, etc. - Use of results determine grades or initiate
some remedial action (tutoring, extra work
assignment, etc.) - Unit of remediation individual student
6Program Assessment
- Purpose judge and improve overall instructional
performance provide data for making program
improvements - Assessment tools written examinations, essays,
papers, presentations, etc. - BUT data are
aggregated - Use of Results program improvements ensure
that the program is accomplishing its objectives - Unit of remediation program
7- Suppose that the assessment data you and your
colleagues have collected show that one or more
of your program outcomes were not achieved that
students didnt know or couldnt do what you had
established as your program outcomes. - What are some of the actions that you might
consider taking to address the problem?
8Program Remediation Strategies
- Curricular changes
- New courses
- Revised courses
- Revised course sequence
- Admission criteria changes
- Instructional methodology changes
9Institutional Effectiveness
- SACS focuses on Program Assessment
- Institutional Effectiveness The institution
identifies expected outcomes for its educational
programs and its administrative and educational
support services assesses whether it achieves
these outcomes and provides evidence of
improvement based on analysis of those results. - (Comprehensive Standard 3.3.1)
10Institutional Effectiveness
- Documentation of Institutional Effectiveness
requires - Evidence that outcomes for educational programs
and administrative and educational support
services are being achieved and that improvements
are the results of assessment procedures
(Handbook for Reaffirmation of Accreditation
p. 62)
11Student Assessment Feeds IntoProgram Assessment
- Aggregate Student Assessment data for Program
Assessment - Student or some product produced by the student
is unit of assessment for both - Broader view of data indicates effectiveness of
overall program - Supplements other program assessment data
12Other Data for Program Assessment
- Standardized Exams (e.g., ETS Major Field, ACAT)
- Alumni Surveys
- Exit Surveys
- Employment Data
- Employer Surveys
- Graduate School Admissions Data
13Assessment Terminology
- What to call that which is most important in a
discipline? - Core Competencies
- Educational Objectives
- Student Learning Outcomes
- Strategic Learning Goals
- Fundamental Knowledge in the Major Field
- Educational Standards
14Assessment Terminology
- SACS Principles use term outcomes
- Term adopted at WKU for our campus-wide
Assessment Plan Model
15What is an Outcome?1
- What Educational Outcomes are NOT
- Individual course outcomes or course learning
objectives. - Individual student assessment within the context
of a given course. - What Educational Outcomes ARE
- Over-arching end results of the entire academic
program. - Facultys view of the most important attributes
of the finished product.
16Preparing Statements of Educational Outcomes1
- 1. Consistent with institutional mission and
expanded statement of purpose - 2. Reasonable given the ability of the students
- 3. Reflect key concepts of program
- 4. Focus on what students will know, think, or be
able to do (not what faculty do). - 5. Clearly stated
- 6. Accomplishment is ascertainable
- 7. Rotate when validated
17Sources of Student Outcomes
- Course syllabi
- Professional associations
- Accrediting bodies
- Colleagues/programs at other campuses
- Informal or formal descriptions from employers
about expectations of graduates - Discussions with current and former students
- Faculty discussions about what are
characteristics of the ideal graduate
18Faculty Discussions
- Stimulate faculty creativity
- Outcomes owned by the program faculty
- Faculty has relative freedom to develop, i.e.,
not meeting any externally determined student
performance standards
19Short List/Long List Concept
- Statements of
- Intended Educational Outcomes
- Statements of what is currently being
assessed - __________________
- __________________
- __________________
- __________________
- __________________
- Other Valid Statements of Intended Educational
Outcomes - __________________
- __________________
- __________________
- 40. _________________
20Program Outcomes
- What a student should
- Know
- Think (i.e. attitudinal, how to think)
- Be able to do
- upon completion of an academic
- program1
- SACS requires that outcomes be identified by each
academic program
21Student Assessment Feeds IntoProgram Assessment
- KEY ISSUE!
- To the extent that a given student assessment
measures or relates to a programmatic outcome, it
can potentially be aggregated and used for
program assessment.
22 - Imagine that you are creating the syllabus for a
new course that you will teach. You have the
content all laid out. The course objectives are
determined. - What are the tools you will use to assess
individual student learning in this course?
23Commonly-used Student Assessments That May Be
Used for Program Assessment
- Comprehensive exams
- Internships
- Essays
- Portfolios
- Oral presentations
- Juried performances
- Theses
- Etc.
24Comprehensive Exam as Student Assessment
- Given in Capstone Course
- Consists of 50 multiple choice questions worth 2
points each
25Comprehensive Exam as Student Assessment
- Standard grade scale for individual students
- 90 100 A
- 80 89 B
- 70 79 C
- 60 69 D
- lt 60 F
26Comprehensive Exam as Student Assessment
- Criterion for success for individual student
- Must achieve a 70 overall (C or better)
- Remediation for individual student
- Students scoring less than 70 will receive
general tutoring
27How Can We Use the Comprehensive Exam as Program
Assessment?
- Outcomes for Environmental Health and Safety
(EHS) Program (see handout) - Outcome 1 Application
- Outcome 2 PPE
- Outcome 3 Monitoring
- Outcome 4 Agents
- Outcome 5 LEV
28How Can We Use the Comprehensive Exam as Program
Assessment?
- Link exam questions to outcomes
- Questions 1 10 assess Outcome 1
- Questions 11 20 assess Outcome 2
- Questions 21 30 assess Outcome 3
- Questions 31 40 assess Outcome 4
- Questions 41 50 assess Outcome 5
29How Can We Use the Comprehensive Exam as Program
Assessment?
- Criteria for Success for Program
- The average grade on the Senior Exam will be no
less than 70 - On no individual outcome area will the average be
less than 14 points
30Comprehensive Exam as Program Assessment
(Aggregate of Student Data)
31Comprehensive Exam as Program Assessment
(Aggregate of Student Data)
32Comprehensive Exam as Program Assessment
- Program Remediation
- Changes in course content to emphasize LEV in
certain courses - Changes in course sequence
- Creation of new course
- Re-visit importance of LEV as a Program Outcome
- Look carefully at the exam
33Comprehensive Exam as Program Assessment
- Key Points
- Correlate specific questions with specific
outcomes and aggregate data - Comprehensive Exam contributes to grade in
Capstone Course - Course-embedded assessment
- Helps solve student motivation problem
34Internship as Means of Program Assessment
- Co-op or internship experience required in many
UG and grad programs - Extremely useful for student and program
assessment - External evaluator
- Uses data from evaluations by on-site supervisors
35Internship as Means of Program Assessment
- Sample program outcome for MA Clinical Psychology
program - Graduates will demonstrate masters-level skills
in psychological assessment. - Evaluated by on-site supervisors using rating
sheet listing six aspects of assessment skills,
rated from 1 (poor) to 5 (excellent)
36Sample assessment dimensions to be evaluated in
internship
- Behavioral observation skills
- Interviewing skills
- Test selection skills
- Standardized test administration skills
- Test interpretation skills
- Report writing skills
37Internship as Means of Program Assessment
- Key Issue
- Feedback from supervisors MUST relate to outcomes
to be useful in program assessment
38Internship as Means of Program Assessment
- Sample overall program criteria for success with
aggregated data - On no individual outcome will more than 20 of
students receive a rating of 3 or below. - The average rating for all students will be at
least 22 out of 30. - Recognize multi-rater consistency issues
39Sample Internship Evaluation Data
40Essay as Means of Student Assessment
- Course objective is to evaluate a students
ability to provide a persuasive argument that
clearly supports a given resolution to an ethical
dilemma3 - Means of Assessment
- Written essay given in a course at the senior
level or other appropriate point in the
curriculum - Criterion for Success Individual Student
- Determined by the course instructor
41Essay as Means of Program Assessment
- Assume ethics is identified as a programmatic
outcome. - Outcome Ethics
- Graduates of the program will be able to provide
a persuasive argument that clearly supports a
given resolution to an ethical dilemma2
42Essay as Means of Program Assessment
- Criteria for Success Program
- Average score (as determined by a panel of
faculty) will be at least 70 out of a possible
100 points on a rubric designed to score the
written essay (see handout from Ron Miller,
Colorado School of Mines) - On no individual rubric category will the average
score be less than 60 of the possible points
43Essay as Means of Program Assessment
- Notes
- All essays need not be scored for program
assessment a sample may be sufficient - A group of faculty should be involved in scoring
the essays - The same rubric used to score each essay for
individual student assessment may also be used in
program assessment
44Essay as Means of Program Assessment
- Notes (cont)
- The same student product (essay) is used twice
once for individual student assessment and again
for program assessment - Course-embedded assessment overcomes student
motivation problem
45Essay as Means of Program Assessment
- Key
- The means of student assessment (e.g. essay) MUST
relate to or measure in some way the PROGRAM
outcome!
46Portfolio as Means of Student Assessment
- Example WKU Teacher Work Sample, completed
during student teaching - Student Outcome
- Teacher candidate must demonstrate use of
multiple assessment modes and approaches aligned
with learning goals to assess student learning
before, during, and after instruction.
47Portfolio as Means of Student Assessment
- Means of assessment
- Teacher Work Sample, scored by supervising
faculty member (see handout). - Criteria for success
- The student must achieve an average of 2.5 on all
five elements to pass. Any element scored less
than 2.0 must be re-done.
48Portfolio as Means of Program Assessment
- Program Outcome
- Teacher candidates must demonstrate ability to
use multiple assessment modes and approaches
aligned with learning goals to assess student
learning before, during, and after instruction.
49Portfolio as Means of Program Assessment
- Means of assessment
- A sample of TWS portfolios is selected. Each
portfolio is scored by at least two raters, using
the TWS rubric. - Criteria for success
- The average rating must be 2.5 or higher on the
five assessment planning elements. - On no element will the average rating be less
than 2.3.
50Portfolio as Means of Program Assessment
51Oral Presentation as Means of Student Assessment
- Sample course outcome Students are expected to
be able to deliver an effective and polished
persuasive or informative oral presentation. - Means of assessment assignment to give 5-minute
oral presentation. - Presentation is evaluated using a rubric (see
handout) that includes verbal and non-verbal
dimensions of effective presentations.
52Oral Presentation as Means of Program Assessment
- Program outcome Graduates will demonstrate
ability to deliver a polished presentation
appropriate to topic and audience.
53Oral Presentation as Means of Program Assessment
- Means of assessment Videotapes of oral
presentations are recorded. A sample is selected
for evaluation on eight variables. - Criteria for success The average overall rating
will be at least 24, with no student receiving
less than a 3 on any variable.
54Juried Performance as Means of Student Assessment
- Programs such as those in music and art
frequently require senior recital or exhibition - Course objective (example) Students are
expected to demonstrate proficiency in tone,
intonation, accuracy, rhythm, technique, and
interpretation or phrasing.
55Juried Performance as Means of Program Assessment
- Outcome
- Program graduates will be able to deliver a
polished and technically correct performance. - Means of assessment
- rubric scored by jury (see handout)
56Juried Performance as Means of Program Assessment
- Criteria for success
- Average ratings will be at least 3.0 across all
elements on the rubric. - On no element will the average be less than 2.5.
57Juried Performance as Means of Program Assessment
58Thesis as Means of Program Assessment
- Sample program outcome
- Program graduates will demonstrate skills in
planning, conducting, and communicating the
results of independent research.
59Thesis as Means of Program Assessment
- Means of assessment masters thesis and oral
thesis defense - Thesis committee members complete rubric
assessing the thesis and oral defense on quality
of literature review, research design and
methodology, description and analysis of results,
quality of discussion of results, quality of
writing, quality of oral presentation, etc.
60Thesis as Means of Program Assessment
- Rubric provides for evaluating each dimension on
a scale of 1 (poor) to 5 (excellent). - Sample criteria for success
- The average overall rating will be at least 3.75.
- No student will score less than 3 on any single
dimension.
61Student and Program Assessment
- Key Issues for Program Assessment
- Means of assessment must relate to or measure the
Program Outcome(s) - Criteria for Success must be stated in terms of
averages or aggregates of individual student
scores - Scoring must involve more than one faculty member
62Using Student Assessment Data for Program
Assessment
- Advantages
- Can be accomplished with many types of
course-embedded student products - Is something each student is already doing as
part of course that relates to a program outcome - Can be transparent to students
- Criteria for success trigger program improvement
63Using Student Assessment Data for Program
Assessment
- Guidelines
- Select a representative sample (some, most, all)
to be scored in some fashion - Use multiple faculty (even external evaluators,
such as on-site supervisors) - Can even block off names for privacy
64References
- Nichols, J.O., Nichols, K.W. (2000). The
Departmental Guide and Record Book for Student
Outcomes Assessment and Institutional
Effectiveness (3rd ed.). New York Agathon Press.
- Ron Miller and Barbara Olds, Colorado School of
Mines, Golden Colorado
65Contact Information
- Retta E. Poe retta.poe_at_wku.edu, (270) 745-4662
- Dennis K. George dennis.george_at_wku.edu, (270)
745-8723