A Process for the Direct Assessment of Program Learning Outcomes Based on the Principles and Practices of Software Engineering

1
A Process for the Direct Assessment of Program
Learning Outcomes Based on the Principles and
Practices of Software Engineering

• Robert W. Lingard
• California State University, Northridge
• ASEE Annual Conference
• June 25, 2007

2
Outline

• The Importance of Assessment
• Problems with Assessment
• Goals of an Assessment Process
• An Approach to Assessment
• Applying the Software Engineering Paradigm
• The Annual Assessment Process at CSUN
• Conclusions

3
Importance of Assessment

• Assessment is required by accrediting boards
  (e.g., ABET).
• Those with vested interests (financial and
  otherwise) in education are demanding
  accountability.
• Without assessment it is difficult to know what
  changes to make to improve learning.

4
Problems with Assessment

• Pressure for assessment can cause a rush to
  assessment, and produce meaningless results.
• Pressure to close the loop can cause premature
  decisions regarding program changes.
• There is a tendency to only assess high
  performance areas in order to showcase a
  program.
• Results are often not validated.
• The effects of changes are often not assessed.

5
Goals of an Assessment Process

• To facilitate the direct assessment of student
  learning
• To measure student retention and the ability to
  apply what they have learned
• To Provide mechanisms to ensure the continuity of
  the process
• To be an efficient and natural extension of
  normal operations
• To satisfy the requirements of ABET and the
  University for assessment

6
Approach to Assessment

• The first and only goal in education is to
  teach for long-term retention and transfer.
  Diane Halpern
• The best assessments of student learning are
  based on direct measures of achievement.
• How well students have learned is best assessed
  by looking at their ability to apply what they
  learned earlier to new situations.

7
Requirements for a Complete Assessment Process

• It must be comprehensive must cover the full
  range of learning outcomes
• It must include multiple judgments multiple
  sources of evidence must be used
• It must include multiple dimensions different
  facets of student performance must be included
• It must collect direct evidence direct measures
  of student attainment must be used

8
The Software Engineering Paradigm

• Requirements Analysis (understanding the problem)
• Software Design (planning a solution)
• Implementation/Coding (carrying out the plan)
• Testing/Validation (making sure the solution is
  correct)

9
Understanding the Problem

• Before beginning assessment, the first step is to
  decide what to assess i.e., pick the most
  important things to assess.
• Collecting information from faculty, students,
  alumni, and employers can give hints as to where
  learning problems exist.
• Surveys, faculty meetings, and student interviews
  are ways of collecting this information.

10
Planning a Solution

• Once a learning outcome is selected, a plan for
  conducting the assessment must be developed.
• It must be determined how the assessment will be
  done i.e., embedded questions in exams, a
  standard instrument, etc.
• How the results will be evaluated must also be
  determined e.g., rubrics must be developed.
• It must also be determine who will do the various
  tasks required and when they will be done.

11
Carrying Out the Plan

• The steps of the plan must be carried out, and
  the plan must be monitored to ensure successful
  completion of the assessment.
• Someone must be designated as the lead, and this
  person has the responsibility for monitoring the
  plan.

12
Making Sure the Solution is Correct

• Assessment results obtained must be validated
  this step is often omitted.
• One way to validate results is to make
  comparisons among several independent assessments
  using different approaches.
• Failure to validate the results can result in
  program changes that may not be appropriate.
• After program changes are made the effects of the
  changes must be assessed.

13
Iteration

• Like software engineering this process is
  iterative.
• Once program changes have been made, they must be
  assessed to determine whether the desired result
  has been accomplished.
• Analysis of this reassessment might indicated the
  need for further changes.

14
The Annual Assessment Process at CSUN

• Department forms Assessment Committee
• The program is divided into seven areas and a
  coordinator is chosen for each area. These
  coordinators constitute the Assessment Committee.
• Assessment Committee recommends outcomes to be
  assessed
• The Assessment considers results from previous
  assessments, informal assessments, and the length
  of time since particular outcomes have been
  assessed to determine the current set to be
  assessed.
• Department approves assessment goals and
  assessment plans are prepared
• After department approval the Assessment
  Committee decides on an assessment approach and
  develops a schedule of activities.

15
The Annual Assessment Process at CSUN

• Department approves assessment plans and
  assessments are conducted
• Program Area coordinators ensure that the
  planned assessment activities are completed and
  that a final report is prepared
• Assessment are analyzed and program changes are
  recommended
• The Assessment Committee analyzes the results of
  the completed assessments and makes
  recommendations for program changes
• Department reviews recommendations and makes
  program changes that are determined appropriate
• At a department meeting the recommendations of
  the Assessment Committee are discussed and
  proposed program changes are made only if there
  is approval from the department as a whole.

16
Conclusions

• The process has been used successfully for
  program assessment at CSUN.
• It has been embedded into the normal operations
  of the department and become accepted by faculty.
• Since all faculty are involved, no one has become
  overburdened.
• It has provided an effective means for directly
  assessing student learning with a focus on
  retention and transfer.