Creating Effective, Viable Plans to Assess Institutional Effectiveness

1
Creating Effective, Viable Plans to Assess
Institutional Effectiveness
Pittsburgh Regional Workshop September 8, 2005

• Linda Suskie, Executive Associate Director
• Middle States Commission on Higher Education
• 3624 Market Street, Philadelphia PA 19104
• Web www.msche.org E-mail LSuskie_at_msche.org

2
Assessment as a Four-Step Cycle
1. Goals
2. Programs, Services Initiatives
4. Using Results
3. Assessment/ Evaluation
3
What Goals Are We Talking About?

• Institutional goals (mission strategic plan)
• Administrative goals
• Division goals
• Administrative unit goals
• Student learning goals
• Institutional
• Gen Ed curriculum
• Academic programs
• Student development programs
• Support programs

4
Institutional Effectiveness Are We Achieving
Mission Goals
Community Service
Scholarship
Student Learning
Diversity
Productivity/ Efficiency
Access
Revenue Generation
5
1. Setting Effective Institutional Goals
6
Effective goals focus on the ends, not the means.

• Root cause analysis
• Ask Why?
• Easier to identify alternative strategies
• Easier to identify appropriate assessments

7
Effective goals stem from institutional mission
priorities.

• Frequently mentioned as important
• Educational quality outcomes
• Responsiveness to fundamental public priorities
• Addressing regional economic, civic social
  needs
• Responsible stewardship of resources efficiency
• Other aspects of institutional mission
• Scholarship
• Service
• Every institution has unique goals.

8
Effective goals have appropriate targets.

• Standards-based
• Specific number enrollment goal, retention rate,
  dollars raised
• Benchmark
• Compared to peers Better than average, top 10
• Best practice Highest/best among peers
• Value-added
• Improvements from beginning to end (esp. student
  learning)
• Longitudinal
• Improvement over time

9
2. Strategies to Achieve Goals

• Objectives
• Initiatives
• Plans
• Operational goals
• Unit/division goals

10
Provide adequate guidance support for plans
initiatives.

• Time!
• Suspend less important activities?
• Technical support
• Coordination
• Professional development opportunities
• Direction/guidelines
• Assistance

11
3. Strategies to Assess Institutional Goals
12
Assessments of student learning

• Direct evidence (clear, visible, convincing)
• Tests examinations
• Assignments, papers, projects
• Portfolios
• Field experience evaluations
• Indirect evidence
• Retention, graduation, placement rates
• Surveys of students alumni

13
Performance indicators

• Measures that monitored in order to determine
  the health, effectiveness, efficiency of an
  institution
• Key performance indicators (KPIs)
• Performance measures
• Performance metrics
• Balanced scorecard
• Dashboard indicators

14
Popular performance indicators

• Student retention graduation rates
• Job placement rates
• Racial/ethnic enrollment breakdowns
• Dollar value of sponsored research grants
• Licensure certification exam pass rates
• Faculty workload
• Student/faculty ratio
• Average credit enrollment per FTE faculty

15
Program reviews (academic other)

• Common criteria for academic program reviews
• Quality
• Resources, activities, outcomes, etc.
• Need
• Demand for the program
• Competing programs
• Centrality to mission
• Cost and cost-effectiveness

16
Baldrige National Quality Program

• Leadership
• Strategic planning
• Customer market focus
• Measurement, analysis knowledge management
• Human resource focus
• Process management
• Results

17
Other assessment strategies

• Surveys focus groups
• Secret shoppers
• Online institutional portfolios
• Quality improvement tools
• Run charts, histograms, pareto analyses, six
  sigma analyses

18
Good Assessments

• Provide useful information.
• Are focused, simple, cost-effective.
• Focus on a few priority goals.
• Yield reasonably accurate, balanced, truthful
  information.
• Multiple measures
• Are systematized.
• Consistent across the organization visible not
  once done
• May be quantitative or qualitative.

19
4. Using Assessment Results
20
How can assessment results be used?

• Inform stakeholders of how well the institution
  is achieving its goals.
• Transparency Clear, relevant communications
• Revise institutional unit-level goals.
• Plan modifications to current programs, services,
  initiatives.
• Plan new programs, services, initiatives.
• Inform budget decisions.
• Improve assessment/evaluation efforts.

21
Promoting a climate for using assessment results

• Use results to inform important decisions.
• Base decisions on systematic, compelling
  evidence, not whim or anecdote.
• Use results ethically responsibly.
• Never base a major decision on only one result.
• Dont use results punitively.
• Recognize reward efforts to improve
• Even if theyre not, at first, successful.
• Encourage risk-taking innovation

22
What Does Middle States Want?
23
Guidance Support

• Are assessments guided by the mission a common
  vision?
• Is there adequate support for assessment?

24
Clearly Articulated Goals

• Are goals clearly articulated at every level?
• Institutional, unit-level, program-level,
  course-level
• Do goals have appropriate interrelationships?

25
Clearly Articulated Learning Goals

• In addition
• Do the undergraduate curriculum requirements
  address
• Institutional learning outcomes?
• MSCHEs general education competencies?
• Are learning outcomes of sufficient rigor?

26
Implemented Assessment Strategies

• Are assessments already implemented for an
  appropriate proportion of goals?
• Are assessments systematic, not anecdotes?
• Ongoing, not once-and-done?
• Do assessments clearly relate to goals?
• Are there multiple assessments?
• Do student learning assessments include direct
  evidence of student learning?

27
Planned Assessment Strategies

• In addition
• Do plans have sufficient detail and ownership?
• Are timelines appropriate?
• Are plans simple and practical enough to be
  feasible?
• Are you confident that plans will be implemented
  as scheduled?

28
Evaluation of the Assessment Plan

• Are assessment efforts reviewed regularly?
• Have the reviews led to improvements in
• Assessment plans?
• Assessment processes?
• Support for assessment?

29
Assessment Results

• Do assessment results provide convincing evidence
  that the institution is achieving its mission and
  goals?
• Institutional goals
• Unit-level goals
• Program-level goals

30
Uses of Assessment Results

• Have assessment results been appropriately shared
  discussed?
• Have assessment results been used to make
  appropriate decisions about
• Curricula and pedagogy?
• Programs and services?
• Resource allocation?
• Institutional goals and plans?

31
What if an institution is not in full compliance?
32
Is there a culture of assessment?

• Do institutional leaders understand value
  assessment?
• Are assessment efforts encouraged, recognized,
  valued?
• Are efforts to improve teaching honored valued?

33
Where do you see the institution going with
assessment?

• Does the institution have sufficient engagement
  momentum on its own?
• How much has been implemented to date?
• Will momentum slow after the team visit?
• Are there any significant gaps in assessment
  plans documentation?
• What Commission action will most help the
  institution keep moving?