The Role of IR in Assessment and Planning

1
The Role of IR in Assessment and Planning

• Presentation by
• Victor M. H. Borden
• February 20, 2003

2
Overview

• The Scratched Surface
• Assessment and its Information Needs
• Planning and its Information Needs
• The Big Picture

3
The Scratched Surface (1)

• Applied Research
• Evaluation
• Basic Research
• Problem Identification
• Action Research
• Policy Analysis

4
The Scratched Surface (2)
Data Scrubbing View Preparation
Insitution's
Operational
Information
Systems
Reporting
Local
DESIGN
OBTAIN FEEDBACK
Surveys
Surveys and
Assessments
Assessment
Data
Presenting
COLLECT?STORE?EXTRACT/SCRUB?ANALYZE?REPORT?PRESENT
5
The Scratched Surface (3)

• Enrollment Management
• Decision Support
• Information support as an enabling process
• Campus Planning
• Program Review and Assessment
• Accreditation

6
Planning

• Annual Planning and Budgeting
• Performance Indicators
• Accountability

7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
Performance Indicators
11
Program Review and Assessment

• Surveys, Surveys, Surveys
• Academic Program Review Support
• Program Evaluation Support

12
Survey Research

• Entering students
• Freshmen
• Non-returning students
• Continuing students
• Internal satisfaction survey
• National Survey of Student Engagement
• Recent Alumni - one year out

• Alumni supervisors
• Further out alumni for specific depts
• Faculty
• Staff
• Client surveys
• IT organization
• Academic programs
• Schools
• Administrative Offices

13
Program Evaluation and Assessment
14
Accreditation

• Institutional Portfolio
• Focused Self-Study
• Teaching Learning
• Civic Engagement
• Civic Engagement Inventory

15
(No Transcript)
16
So What Else is There to Know?

• More depth on what is being supported
• Assessment
• Planning
• How it all fits together

17
What is Assessment?

• The process of providing credible evidence of
  the outcomes of higher education undertaken for
  the purpose of improving programs and services
  within the institution.
• Trudy W. Banta

18
Ted Marcheses Insights

• Good teachers, like "reflective practitioners" in
  other professions, constantly test, adjust, and
  reframe their models of practice on the basis of
  experience and reflection.

19
Forerunners of Assessment (1)

• Student Learning in College
• Contributors
• Feldman Newcomb, Astin, Pace, Stern, Murray,
  Chickering, Pascarella Terenzini
• Contributions
• Taxonomies of outcomes
• Models of student growth
• Student engagement
• Tools
• cognitive examinations longitudinal and
  cross-sectional surveys quasi-experimental
  designs

From Peter Ewells chapter, An Emerging
Scholarship in Trudy Bantas (2002) The
Scholarship of Assessment
20
Forerunners of Assessment (2)

• Retention and Student Behavior
• Contributors
• Tinto, Spady, Astin, Noel, Lenning, Beal, Sauer,
  Bean Metzner, Pascarella, Terenzini, Ewell
• Contributions
• Student tracking studies
• Specially configured surveys
• Multivariate analytical techniques
• Action research

21
Forerunners of Assessment (3)

• Scientific Management and Program Evaluation
• Contributors
• Jones Ewell, Enthoven, Walhaus, Dressel, Guba
  Lincoln
• Contributions
• Cost/benefit studies
• Resource allocation models
• Social return on investment
• Student use and perception of programs
• Qualitative program evaluation methods.

22
Forerunners of Assessment (4)

• Mastery Learning
• Contributors
• K-12 mastery movement Thornton Bynham Alverno
  College Faculty Other Experimental Colleges
  (Evergreen, Empore State, Regents, Antioch,
  DePaul School for New Learning)
• Contributions
• Articulation of learning goals
• Authentic assessment techniques (e.g., Portfolios)

23
Three IR Roots Plus

• Student Learning in College
• Student Retention
• Scientific Management and Program Evaluation
• Mastery Learning

24
Underlying Philosophies

• Rational, positivistic, scientific and objective
  program evaluation models
• Subjectivist, intuitionist ethic, that values the
  tacit knowledge of professional authorities

25
Techniques of Assessment

• Tests and examinations
• Projects, assignments, and performances
• Portfolios
• Case studies
• Journal analysis
• Student profile trends
• Student tracking
• Critical point assessments
• Surveys
• Paper/web Focus Groups Interviews

26
Sources of Assessment Data

• Departmental and Course Records
• performance appraisals, program participation,
  facility usage
• Institutional Records
• admissions, registration, grades, financial aid,
  alumni
• Surveys
• questionnaires, interviews and focus groups
• mail, phone, and in-person
• External data sources
• National exams, peer institution benchmarks,
  environmental scanning

27
A little ir in Everyone

• Assessment is everybodys business
• Faculty
• Administrative Managers
• Central IR can be both facilitator and supporter
• Expert consultation on research and measurement
• Staff support as needed
• Action research has potential

28
Example - Principles for Undergraduate Learning
(PULs)

• A common language for general education
• Core communication and quantitative skills
• Critical thinking
• Integration and application of knowledge
• Intellectual depth, breadth, and adaptiveness
• Understanding society and culture
• Values and ethics

29
Core Communication and Quantitative Skills

• The ability of students to write, read, speak,
  and listen, perform quantitative analysis, and
  use information resources and technology and the
  foundation skills necessary for all IUPUI
  students to succeed.
• This set of skills is demonstrated, respectively,
  by the ability to
• express ideas and facts to others effectively in
  a variety of written formats
• comprehend, interpret, and analyze texts
• communicate orally in one-on-one and group
  settings
• solve problems that are quantitative in nature,
  and
• make efficient use of information resources and
  technology for personal and professional needs.

30
Critical Thinking

• The ability of students to analyze information
  and ideas carefully and logically from multiple
  perspectives. This skill is demonstrated by the
  ability of students to
• analyze complex issues and make informed
  decisions
• synthesize information in order to arrive at
  reasoned conclusions
• evaluate the logic, validity, and relevance of
  data
• solve challenging problems, and
• use knowledge and understanding in order to
  generate and explore new questions. 

31
Programs Map Learning Goals to PULs
32
PULs Integrated into Campus Surveys
33
Program Evaluation and Assessment
34
Information Support to Planning

• Trend analysis
• Models revenues, costs, enrollments, etc
• Program evaluation effectiveness and efficiency
• Performance indicators

35
Trend Analysis

• Examining changes over time in
• Enrollments
• Revenues and expenditures
• Staffing
• Faculty activity
• Student, faculty, and staff satisfaction

36
Trend Indicators for Planning
37
Models

• Cost models
• Activity-based costing - macro and micro
• Enrollment models
• Projections and targets
• ICLM
• Other models
• Delaware instructional productivity

38
Program Evaluation

• Value added
• Difference in outcomes with appropriate controls
• Process improvement
• Academic program review
• Guided self-study and site visit
• Portfolio assessment
• Student, teaching, institutional, program

39
Indicators for Program Review
40
Assessment of Institutional Effectiveness

• Achievements of students, faculty,
• staff, graduates
• Findings from questionnaires, interviews, focus
  groups
• Special studies (retention, large classes)
• Annual reports using performance indicators
• Campus
• Units (overall and assessment)

41
An Information Support Pyramid
Performance Indicators
Trend Analysis
Modeling
Program Evaluation
42
Putting it All Together

• Planning and assessment are part of the same
  cycle
• The vast majority of institutions are not that
  good at eitheryet
• A new way of business for higher education
• Even if good at planning or assessment, linking
  them together is yet another matter

43
Planning
Plan
Implement
Improve
Assess
44
The Planning Cycle A Full Contact Sport

• How does or can it all come together at your
  institution?
• Classroom learning
• Academic programs
• Service units
• Administrative units

45
Strategies for Alignment Some Elements
46
Alignment Strategies A Few Examples

• First-year experience programs
• K-16 consortia
• Enrollment management
• Faculty/staff development
• Student portfolios
• Constituent surveys
• Integrative policy committees
• Integrated planning/budgeting process

47
The Essential Elements

• Leadership at all levels
• Common vision and complementary goals
• Capable processes
• Effective information support
• Accessible, timely, relevant, contextualized,
  constructive

48
IR as a Critical Component

• Takes the enterprise view
• Supports the culture of evidence
• An enabling support
• Part of a team
• Planning, Assessment, IR, IT, Libraries, Faculty
  and Staff Development, Budget and Finance,
  Academic and Administrative Management

49
An Expanded View of the Information Cycle