INSTITUTIONAL EFFECTIVENESS REQUIREMENTS

1
INSTITUTIONAL EFFECTIVENESS REQUIREMENTS

• Alexei G. Matveev, Ph.D.
• NORFOLK STATE UNIVERSITY
• Southern University at New Orleans // Workshop on
  SACS Reaffirmation Preparations
• New Orleans, LA // October 16-17, 2008

2

• Two boys are walking down the street. The first
  boy says, Ive been really busy this summer.
  Ive been teaching my dog to talk.
• His friend responds, Wow! I cant wait to have
  a conversation with your dog.
• The first boy shakes his head. I said Ive
  been teaching him. I didnt say he learned
  anything.
• from Mary J. Allen (2004).
• Assessing Academic Programs in Higher Education
• Anker Publishing, Inc. Bolton, MA.

3
Institutional Effectiveness Questions

• Does the University achieve its stated goals?
• How do we know that the University achieves its
  stated goals?
• What do we do to ensure that the University
  continues to achieve its stated goals with
  excellence?

4
Agenda

• Importance of Demonstrating Institutional
  Effectiveness
• Higher education context
• SACS Principles of Accreditation
• Components of Institutional Effectiveness
• Characteristics
• Evidence

5
Institutional Effectiveness (IE) Definition

• Systematic, explicit, and documented process of
  measuring performance against the mission in all
  aspects of an institution (SACS, 2005, p. 9)
• Intentional process of ascertaining the level of
  achievement of an institutions mission and its
  educational goals and objectives (SACS, 2005, p.
  8)

6
IE Goal

• Continuous Quality Enhancement
• The concept of quality enhancement is at the
  heart of the Commissions philosophy of
  accreditation. . .
• Although evaluation of an institutions
  educational quality and its effectiveness in
  achieving its mission is a difficult task. . . ,
  an institution is expected to document quality
  and effectiveness in all its major aspects
  (SACS, 2001/2004, p. 5)
• Purpose of IE Requirements
• Establish an approach to ensure and facilitate
  continuous quality enhancement
• Provide evidence of continuous quality
  enhancement

7
Importance of Institutional Effectiveness
8
Importance of Demonstrating IE

• Changing Student Population
• Students as consumers/customers
• Education as investment
• Too many decisions about higher educationfrom
  those made by policymakers to those made by
  students and familiesrely heavily on reputation
  and rankings derived to a large extent from
  inputs such as financial resources rather than
  outcomes.
• (The Secretary of Education Commission on the
  Future of Higher Education, 2006)

9
Importance of Demonstrating IE

• Academic Quality
• The higher education community accepts student
  achievement of expected learning outcomes as the
  key indicator of quality (AACU, 2004, p.1
  Ewell/CHEA, 2002 Massy, 2003 Ratcliff, 1997
  SACS, 2001/2004 Tagg, 2003)
• Whereas quality was once defined in terms of
  inputs and resources what the institution
  hasits now defined in terms of processes and
  outcomeswhat the institution does with what it
  has (Wergin, 2005, p. 36)

10
Importance of Demonstrating IE

• Accountability
• Postsecondary education institutions should
  measure and report meaningful student learning
  outcomes.
• The federal government should provide incentives
  for institutions to develop interoperable
  outcomes-focused accountability systems designed
  to be accessible and useful for students,
  policymakers, and the public, as well as for
  internal management and institutional
  improvement.
• (The Secretary of Education Commission on the
  Future of Higher Education, 2006)

11
IE and SACS Accreditation Requirements and
Standards

• Core Requirement 2.5 Institution-Wide Approach
  to Continuous Quality Enhancement
• The institution engages in ongoing, integrated,
  and institution-wide research-based planning and
  evaluation processes that
• (1) incorporate a systematic review of
  institutional mission, goals, and outcomes
• (2) result in continuing improvement in
  institutional quality and
• (3) demonstrate the institution is effectively
  accomplishing its mission.

12
IE and SACS Accreditation Requirements and
Standards (Contd)

• Comprehensive Standard 3.3.1 Program-Level
  Evidence of Continuous Quality Enhancement /
  Outcomes Assessment. The institution
• 1. identifies expected outcomes for
• Educational programs (including student learning
  outcomes for educational program)
• Administrative support services
• Educational support services
• Research
• Public/community service
• 2. assesses whether it achieves these outcomes
  and
• 3. provides evidence of improvement based on
  those results.

13
CR 2.5 and CS 3.3.1

• CR 2.5 requires that an institution have an
  effective process for producing improvement and
  accomplishing its mission.
• CS 3.3.1 requires that an institution identify
  outcomes (resulting from the process required in
  CR 2.5), evaluate achievement of those outcomes,
  and demonstrate improvement based on the results
  of that evaluation.

14
IE and SACS Principles of Accreditation

• Even though the concept of institutional
  effectiveness may not be explicitly referenced in
  all of the comprehensive standards, the
  accreditation process assumes that all programs
  and services wherever offered within the context
  of the institutions activity are reviewed as
  part of the institutional effectiveness process.
  (SACS, 2005, p. 9)

15
IE and SACS Selected Accreditation Requirements
and Standards (Contd)

• CR 2.9 Learning Resources and Services
• CR 2.10 Student Support Services
• CR 2.11.1 Financial Resources
• CR 2.12 Quality Enhancement Plan
• CS 3.2.8 Qualified Administrative Officers
• CS 3.4.10 Responsibility for Curriculum
• CS 3.8.3 Qualified Staff Learning Resources
• CS 3.9.3 Qualified Staff Student Affairs
• CS 3.11.3 Physical Facilities
• FR 4.1 Student Achievement

16
Example IE and CR 2.10

• Support documentation and assessment data is
  lacking to indicate that student support
  programs, services, and activities support the
  institution's mission to promote student learning
  and development. Throughout the narrative
  reference is made to using assessment of services
  utilizing the CAS Professional Standards for
  Higher Education and Assessment Practices in
  Student Affairs but no examples of assessment
  instruments or results are given (SACS
  reviewers comment)

17
Example IE and FR 4.1

• Sufficient evidence is lacking regarding job
  placement data, graduate school acceptance,
  certification/licensures, etc., for those
  programs not represented by an external
  accrediting body, as well as some programs with
  external accreditation (SACS reviewers comment)

18
CR 2.5 Institutional Effectiveness

• The institution engages in ongoing, integrated,
  and institution-wide research-based planning and
  evaluation processes that (1) incorporate a
  systematic review of institutional mission,
  goals, and outcomes (2) result in continuing
  improvement in institutional quality and (3)
  demonstrate the institution is effectively
  accomplishing its mission.

19
IE What does SACS staff say?

• Donna Wilkinson, VP SACS-COC
• Conditions for Effective Planning and Evaluation
  Processes
• Characteristics of Sound Planning and Evaluation
  Processes

20
IE What has been learned from commonly cited
issues?

• Mission-Driven IE Process
• The review process by the Council for Assessment
  and Planning covers the number of reports
  completed and submitted, but it does not serve to
  assess the value of the reports with regard to
  assessment of institutional mission. The process
  lacks a link to the institutional mission, stated
  or implied, at any level -college/department or
  upper-level administration. (SACS reviewers
  comment)

21
IE What has been learned from commonly cited
issues? (Contd)

• Systematically Reviewed IE Process
• While the recent approval of the Mission and
  Goals by the Board provides evidence that the
  college periodically reviews its mission and
  goals, it does not provide evidence of a
  systematic review of the mission and goals.
  (SACS reviewers comment)

22
IE What has been learned from commonly cited
issues? (Contd)

• Aligned/Coherent IE Process
• While there are institution-wide, integrated,
  data-based planning processes built on a
  well-defined strategic plan with appropriate
  goals and objectives that are adhered to by all
  the units on campus, there is no immediately
  obvious linkage between a particular goal, an
  assessment method, specific criteria for success,
  and actions taken to improve. (SACS reviewers
  comment)

23
IE What has been learned from commonly cited
issues? (Contd)

• Systemic and Documented IE Process
• An institutional effectiveness process began
  in 2005. Not all units have completed the
  process or if they have, evidence is not
  shown. (SACS reviewers comment)

24
IE What has been learned from commonly cited
issues? (Contd)

• IE Process Explicitly Linked to Analysis,
  Decision-Making, and Budgeting
• The extensive assessment reports from all the
  units on campus contain a wealth of specific
  indicators of success and a lot of raw data, but
  there appears to be no attempt to interpret the
  meaning of the data and link them to actions in a
  way that closes the assessment loop.  Neither
  does the narrative address how funding decisions
  are linked to the assessments. (SACS reviewers
  comment)

25
IE What has been learned from commonly cited
issues? (Contd)

• Improvement-Focused IE Process
• The name of the institutions plan, however,
  is an accountability plan, requiring annual
  reports of enrollment, retention, graduation
  rates, etc. with no clear evaluation process and
  no description of how the results of the reports
  will be used. (SACS reviewers comment)
• The report did not provide explicit description
  of use of results for continual improvement.  A
  model program review system was in place, but
  again, use of results was not specified.  (SACS
  reviewers comment)

26
IE/CR 2.5 Important Indicators What has been
learned from commonly cited issues? (Contd)

• Mature IE Process
• The institutions processes have been
  strengthened greatly over the past four years and
  the institution documents a commitment to
  continuing improvement of its processes and the
  use of their results.  However, the institution
  is still developing those processes and has not
  yet compiled results that may be used to document
  continuing improvements.  The institution needs
  to document that it has attained results from its
  institutional effectiveness efforts and used the
  results for continuous improvements. 
  (SACS reviewers comment)

27
Components of Institutional Effectiveness Process
28
IE Process A Model
29
Components of IE Process

• Planning
• Budgeting
• Accountability and Assessment
• Feedback Mechanisms
• Coordination
• IE Program Support

30
Components of IE Process Analysis

• Structural Element
• Characteristics
• Evidence

31
Components of IE Process 1. Planning

• 1.1 Environmental scans
• Characteristics
• Periodic
• External and Internal
• Triangulated
• Mission-based interpretation
• Evidence
• Participants and data collection methods
• Findings and analyses
• Communication/distribution

32
Components of IE Process 1. Planning (Contd)

• 1.2. Strategic Plan development and
  implementation
• Characteristics
• Explicitly mission-driven (mission statements are
  typically reviewed during the strategic planning
  processes)
• Environmental scan data-based
• Enforced, integrated in annual planning and
  reporting
• Evidence
• Strategic Plan
• Strategic Plan flowchart
• Strategic Plan Committee meeting minutes
• Annual Implementation Reports

33
Components of IE Process 1. Planning (Contd)

• 1.3 Alignment between internal plans and external
  mandates
• Characteristics
• Alignment, not hierarchy
• Evidence
• Crosswalks between plans
• Common indicators

34
Components of IE Process 2. Budgeting

• 2. Budgeting
• Characteristics
• Mission-driven and explicitly aligned with
  Strategic Plan
• Assessment and evaluation data-based
• Open and participatory process
• Evidence
• Budget
• Budget development process flowchart
• Budget request forms and instructions
• Budget Committee meeting minutes

35
Components of IE Process 3. Accountability and
Assessment

• 3.1.1 Annual Reporting
• Characteristics
• Standardized Annual Report template based on the
  Strategic Plan goals
• Activities and outputs-based
• Unit/provider-focused
• Evidence
• Annual Reports (Units, Divisions, University)
• Annual Report templates and instructions

36
Components of IE Process 3. Accountability and
Assessment

• 3.1.2 Personnel Evaluation
• Characteristics
• Based on the Strategic Plan objectives
• Transparent process
• Evidence
• Personnel Handbooks
• Job descriptions
• Personnel evaluation process flowchart
• Evaluation forms and samples

37
Components of IE Process 3. Accountability and
Assessment

• 3.2.1 Unit/Program Assessment
• Characteristics
• Formative
• Mission-driven and Outcomes-based
• Open
• Student/customer/client-focused
• Evidence
• Matrix of submitted and accepted reports and
  sample reports
• Assessment report templates and instructions
• Departmental/Committee meeting minutes

38
Components of IE Process 3. Accountability and
Assessment

• 3.2.2 Core Competency/Gen Ed Assessment
• Characteristics
• Based on coherent Gen Ed rationale and
  well-articulated outcomes aligned with the
  University Mission
• Assigned to designated individual or group (e.g.,
  Gen Ed Council) but Participatory
• Comprehensive
• Representative
• Evidence
• General education assessment process flowchart
• General Education Program assessment report

39
Components of IE Process 3. Accountability and
Assessment

• 3.2.3 University Assessment
• Characteristics
• Representative
• Focused on campus climate/environment
• Based on perceptions, satisfaction and
  self-reported growth
• Evidence
• Matrix of survey administration
• Reports analyzing and interpreting survey data
• Summary reports, displays, and presentations

40
Components of IE Process 3. Accountability and
Assessment

• 3. 3 Program Review
• Characteristics
• Comprehensive and summative
• Periodic
• Based on perceptions, satisfaction and
  self-reported growth
• Evidence
• Schedule of program reviews and specialized
  accreditation visits
• Program review criteria, process flowchart, and
  instructions
• Program review reports and self-studies

41
Components of IE Process 4. Feedback Mechanisms

• Strategic Plan
• University Annual Theme
• President and VPs annual objectives
• Annual Reports
• Unit Plan for upcoming year
• Personnel Evaluations
• Improvements Needed section
• Assessment Reports
• Improvement Plans and Improvements Made sections

42
Components of IE Process 5. Coordination

• Presidents Cabinet / Strategic Planning Council
• Institutional Research
• Enrollment data analysis and reporting
• Planning and Budget
• Financial planning and reporting
• Institutional Effectiveness and Assessment
• Outcomes assessment and quality assurance

43
Components of IE Process 6. Support

• Senior Leadership support and guidance
• Designated individuals/offices to coordinate IE
  process
• Policies
• Procedures, process flowcharts, forms and
  templates
• Handbooks and Manuals
• Faculty and Staff Workshops
• Software, technology
• Websites

44
Sample Outline for CR 2.5 Narrative

• Introduction
• Planning
• Strategic planning process
• Budgeting
• Feedback Mechanisms
• Evaluation
• Annual reports
• Personnel evaluation
• Assessment
• Unit/program
• Core learning areas
• University
• State accountability program reviews and
  specialized accreditations

• 5. Coordination of IE process
• 4. Continuous Quality Enhancement
  Mission-Critical Indicators
• 1. Student success
• 2. Management structures
• 3. Funding
• 4. Improvements in IE process
• 6. Conclusion

45
Review How Well Did the Narrative Address the
IE Requirement? (from Green Book)

• What are the institutions processes for
  systematic, ongoing, integrated, research-based
  reviews that result in continuous improvement?
• How does the institution demonstrate a sustained,
  documented history of planning evaluation cycles,
  including the use of results for improvement, to
  accomplish the institutions mission?
• Is there appropriate institutional research and
  budgetary support for assessment programs
  throughout the institution?

46
Review How Well Did the Narrative Address the
IE Requirement? (from Green Book)

• What is the evidence that data from various
  sources concerning the effectiveness of programs
  and services are being used to make decisions for
  improvement?
• How is the institutional effectiveness process
  related to the budget?
• Are appropriate internal and external
  constituents and stakeholders involved in the
  planning and assessment process?

47
Questions and Discussion
48
INSTITUTIONAL EFFECTIVENESS REQUIREMENTS

• Alexei G. Matveev, Ph.D.
• NORFOLK STATE UNIVERSITY
• Southern University at New Orleans // Workshop on
  SACS Reaffirmation Preparations
• New Orleans, LA // October 16-17, 2008