Assessing the Effectiveness of

1

• Assessing the Effectiveness of
• Academic Programs
• Quality, Potential, Costs
• Operationalizing the Variables

2
Foundation of Cost Study

• Followed National Delaware Study assumptions
  and best practices
• 300 Public and Private Schools participate
• Delaware Study methodology does not include
  revenue or overheads
• Millikin developed assumptions and methodologies
  for allocating overhead

3
Definitions

• SCH student credit hours (i.e., 25 students in
  a 3 credit class generates 75 SCH)
• Direct costs faculty salaries, benefits,
  supplies, and duplicating
• Overhead administrative expenses, physical
  plant, dining, student services
• Non-operating Income/Expense annual fund,
  unrestricted endowment income, property sales,
  summer school and immersion courses

4
Definitions Continued

• Contribution Margin Net Revenue less Direct
  Costs
• Margin from Operations Contribution Margin less
  Overhead
• Unrestricted Margin Margin from Operations less
  Non-operating Income/Expense
• Athletic sports expenses are included in overhead

5
Reminders

• Each colleges discount rate reflected under
  scholarships
• Physical Plant Overhead allocated based on square
  footage, 40 to academic units and 60 to general
  overhead
• Use of only unrestricted operating budget, not
  endowment or restricted dollars
• Now have 2004, 2005, and 2006 comparison data

6
Effectiveness of Academic Programs
The QPC Model
Program Effectiveness
7
QPC Project in Context

• 2002 Project Renewal to achieve financial
  stability
• PACE Launched
• 3 Academic Programs eliminated Religion, French,
  German
• Full time faculty reduced from 159 in 2004 to 138
  in 2005
• Project Renewal ultimately successful
• FY06 budget fourth in a row to end in surplus
• Endowment grew from 54.6 million in 2003 to 80
  million in 2006
• Currently 145 faculty members
• Cost Study to determine how to re-allocate
  resources to support cost-effective programs
• Cost Study incorporated into a more robust study
  of program effectiveness -- Quality, Potential
  and Cost
• Situated in the context of the re-accreditation
  self study
• Concurrent with University Studies Assessment
  Process

8
Project Development

• VPAA Deans reviewed best practices
• VPAA Deans generated key dimensions of the QPC
  variables
• Appropriate sharing and support
• Chairs and Directors Briefings (March 06)
• Feedback led to clarification and to expanded
  matrix (from 8 cells to 27 cells)
• Department Reports written (July 06)
• Executive Summaries (August 06)
• Operationalization of QPC variables (Fall 2006)
• Pilot Application of coding scheme
• Chairs and Directors Briefing (January 2007)

9
Guiding Philosophy - Moving from Good to Great

• If we focus only on program efficiency, we will
  merely have a prosperous university, not a great
  one. Economic growth is the means, not the
  definition, of a great university.
• paraphrased from Jim Collins
• Good to Great and the Social Sector
• To be a great university, we must embrace the
  market-smart, mission-driven interconnections
  among Quality, Potential, and Costs

10
Quality

• External validation of program quality
• Apply to Admit (freshmen and transfers)
• Admit to Enrollment yield (freshmen transfers
• Specialized accreditation
• External awards or recognition
• External grants
• Results from external reviews
• External validation from media
• Other validations of quality

11
Quality

• Inputs
• Faculty Credentials
• Number of FT, PT and associated credit hour
  production
• Degree, rank, experience, contributions for all
• Support for Faculty Development
• Student Preparedness
• Student Demographics
• Support for Student Development
• Student organizations
• Honor societies
• Funding for research, conference participation,
  competitions, etc.

12
Quality

• Outputs
• Retention/graduation rates
• Time to completion
• leaving/staying in program
• leaving/staying at MU
• Placement reports
• Employer information
• Graduate feedback
• Former student publications and creative works
• Other notable student achievement

13
Quality

• Curriculum and Program Issues
• Meets MU standards for learning outcomes
  assessment
• Includes integrated learning, experiential
  learning, and connection between theory and
  practice
• Contains and integrates MU elements of sequential
  learning for major programs
• Incorporates relevant global and international
  issues

14
Quality

• Curriculum and Program Issues, cont.
• Appropriate integration of technology
• Includes opportunity for service learning
• Reflects current discipline-related standards for
  content and coherence
• Appropriate program-related library holdings
• Class size ( of courses lt 8, of courses 8-25,
  of courses gt 25, of courses gt 50)
• Facilities
• Function (technology, equipment, classroom
  management, safety, etc.)
• Aesthetics (clean, modern, appealing learning
  environment)

15
Potential Growth Development

• National Demand
• Local Demand Trend
• 10 year trend of declared majors
• 10 year trend of number of graduates
• 10 year trend of number of graduates with minor
• External Transfer student enrollment
• Exploratory student major selection
• Service to Interdepartmental majors
• Size
• as measured by of total student population on
  top 2 above measures
• as measured by SCHs generated annually

16
Potential Growth Development

• Internal Impact
• Degree to which program serves other departments
  (as required courses) or general education
• Voluntary offerings (University Studies)
• Required for single major program
• Required for general education
• Popular electives

17
Potential Growth Development

• Essentiality to Mission
• Consistent with Strategic Goals
• Professional or liberal arts program with
  increasing trend line or is required to provide
  substantial service
• Serves external constituencies
• Enriches campus culture or life of learning for
  campus community members
• Other justification, future opportunities

18
COSTS

• Revenue production for past 3 years
• Discount rate for past 3 years
• Contribution margin Direct costs minus revenue
  (Delaware Study, National Benchmarks)
• Margin from operations Contribution margin
  after allocating overhead costs per SCH
• Unrestricted margin Margin from operations
  after allocating non-operating income/expense per
  SCH
• Productivity/costs per credit hour SCH/faculty
  expenses

19
Useful Decision-Making Tool

• Includes comprehensive list of program
  evaluation criteria
• Promotes more complete understanding of the
  program mix
• Provides systematic, transparent method for
  prioritizing academic programs
• Identifies opportunities for re-allocation of
  resources
• Encourages market-smart, mission-driven decisions

20
Implementation Assumptions

• Enrollment will stabilize at 2700
• 2200 (T), 425 (P) and 75 (G)
• Number of majors should be reduced
• NCA recommendation
• Dramatically uneven dispersion of students
• Need to allocate resources to moderate-high
  potential programs
• Number of course offerings should be reduced
• Minimize low enrollment sections
• Reduce use of adjuncts
• Need to strengthen University Studies and ensure
  majors are grounded in and inspired by liberal
  arts

21
QPC Rubrics

• Breaker variables for each of the three QPC
  dimensions were identified from the longer list
  of evaluative criteria (just reviewed)
• Each of these variables were evaluated separately
  and scaled as low, moderate or high
• Process varied a bit for each variable and each
  QPC dimension because available data varied
• Objective, quantitative data
• Subjective, qualitative data
• Overall rating for dimension calculated from
  scores on breaker variables for that dimension

22
Quality Rubric Operationalized

• Each criterion rated 3high, 2moderate, 1low
• Average rating used to determine overall Quality
  rating
• External Validation
• Inquiry to apply, apply to admit, admit to enroll
  
• External reputation
• Accreditation
• If available yes 3, no 1
• If not available not computed in score
• Inputs (faculty and students)
• Outputs
• Curriculum

23
Potential Rubric Operationalized

• Variables rated 3high, 2mod, 1low,
  0Non-existent
• Average used to determine overall Potential
  rating
• Number of graduates
• used completes from 05, matched 10 year pattern
• High 4, Moderate 2-4, low lt 2
• External Demand anticipated demand
• Internal Demand courses serving other majors
• Essential fit to mission and the 3 prepares
• Capacity for growth at capacity or with
  potential rated higher

24
Costs Rubric Operationalized

• Each of the three criterion variables were
  ranked, then rated 3high, 2mod, 1low
• Two of three determined overall rating (H, H,
  M) H (few exceptions with one of each)
• Costs per student credit hour
• Total direct costs divided by the total number of
  student credit hours in department
• Total Contribution Margin
• Total net revenue minus total direct costs by
  department
• Total Student Credit Hours
• Total student credit hours generated by
  department

25
QPC Matrix now expanded to 27 Cells
Med. Quality
Low Quality
High Quality
Low Potential Med Potential High
potential
Low Potential Med Potential High
potential
Low Potential Med Potential High
potential
Low Cost
Med Cost
High Cost
26

• Applying the Operationalization Rubrics
• Examples across the curriculum