Conducting a Program Review from Beginning to End

1
Conducting a Program Review from Beginning to End
Cathy A. Buyarski, Assistant Dean Michele J.
Hansen, Director of Assessment, University
College Chris Maroldo, Coordinator of Academic
Success Programs University College Indiana
University-Purdue University Indianapolis
2
Presentation Overview

• University College Academic Advising
• Program Review
• Assessment Planning and Approaches
• Review Process
• Using the Results

3
IUPUI and University College
4
IUPUI Context

• Large urban, commuter, public university.
• Many incoming students possess characteristics
  that place them at a greater risk for academic
  failure and attrition.
• Many students have not completed a rigorous high
  school college-preparatory curriculum.
• Attend classes part-time.
• Majority live off-campus.
• Significant off-campus work commitments.
• Over one-half of fall beginning freshmen are
• first -generation college students.

5
University College

• Academic unit formed in 1997
• Houses numerous first-year programs including
  advising, orientation, honors, Math Assistance
  Center, Writing Center, student mentoring
  programs.
• Develops and implements academic support courses
  including learning communities, first-year
  seminars, and critical inquiry.
• Serves Over 6,500 students

6
Mission Statement

• The University College Advising Center is
  committed to the academic success of individual
  students as they transition into college and to
  their degree-granting school. We partner with
  students to develop a coherent and meaningful
  plan for academic success.

7
Primary Functions of University College Advisors

• Individual Advising
• New Student Orientation
• Learning Communities
• Work with at-risk students
• Reinstatement decisions
• Workshops
• Liaisons to other Schools and Departments
• Contributions to University College

8
Program Review Process
9
Institutional Objectives for Program Review

• Provide planning information
• Directing internal resource allocation
• Providing monitoring to ensure quality
• Internal review process

10
Linking Program Review to our Mission

• Selection of reviewers
• Self-study must address the campus mission and
  goals, relating them to the units
• Preliminary questions to reviewers
• Process is an example of best practices

11
Elements of Program Review

• Self Study
• Review by Respected Peers
• Recommendations
• Follow-up

12
Empowerment Evaluation And Program Review

• Empowerment evaluation has an unambiguous value
  orientation -- it is designed to help people help
  themselves and improve their programs using a
  form of self-evaluation and reflection. Program
  participants -- including clients conduct their
  own evaluations an outside evaluator often
  serves as a coach or facilitator (p. 1).
• Fetterman and Eiler (2001)

13
Literature Suggests

• Research on implementing effective
  institutional change suggests that efforts to
  improve organizational functioning often do not
  lead to expected outcomes because of unintended
  negative reactions among individuals responsible
  for implementing improvements (e.g., Armenakis
  Bedeian, 1999 Olson Tetrick, 1988 Wanberg
  Banas, 2000).

14
Program Review Examines

• Resources
• Credentials of staff, facilities, budget,
• equipment
• Processes
• Courses / Curricula, services, program
  offerings, how business is conducted
• Outcomes
• Achievement of stating learning outcomes,
• retention and graduation, adjustment
  to
• college, external recognition

15
Sources of Evidence Used for Outcomes Assessment
in Self-Studies
Satisfaction Surveys
Portfolios
Survey Self-Reports
Focus Groups
Standardized Tests
GPAs
Interviews
Classroom Assessment Techniques
Academic Performance (GPAs DFWs)
National Survey of Student Engagement
Retention
Graduation Rates
Campus Climate for Diversity Survey
Degree Attainment
Focus Groups
Interviews
16
The Site Visit

• 2 ½ day visit
• Meet with
• Faculty
• Alumni
• Community representatives
• Students
• Staff
• Members of related departments

17
Following the Review

• Unit reviews the recommendations
• Unit prepares written response
• Unit presents response in administrative hearing
• Responsible administrators agree upon responsive
  action(s)
• Responsive actions are implemented
• Re-review ensures action

18
Empowerment Evaluation Tenants Applied to Service
Unit Program Reviews

• Unit director empowerment through active and
  on-going participation (the self-study).
• Reviewer feedback promotes collaboration,
  dialogue, and collective analysis.
• Active learning and discovery fostered by
  critical reflection process.
• Data-driven action plans developed following
  review.

19
Benefits Of Program Reviews for Service Units

• Establishes atmosphere of openness and trust
  regarding assessment data.
• Creates positive affect regarding usefulness of
  assessment.
• Leverages university opposing forces.
• Brings external legitimacy to the service unit.

20
Step 1 Assessment Planning
21
Assessment Plan

• Stakeholders
• Purpose of Assessment
• Articulated Program Goals
• Mapped out Processes and Learning Outcomes
• Sources of Evidence
• Methods of Gathering Evidence

22
Assessment Committee(comprised of stakeholders)

• University College Advisors
• Full-time
• Part-time
• Graduate Assistants
• Faculty member who developed previous campus
  advising survey
• UCOL Director of Assessment

23
Purpose of Advising Assessment

• Improvement in student learning/satisfaction
• Improvement in advisor training/satisfaction
• Efficiency and effectiveness in all areas
• Accountability

24
Goals for UC Academic Advising

• Promote holistic student learning through
  academic, personal, and career development
• Empower students to make informed decisions
• Promote positive and collaborative relationships
  with other campus units
• Facilitate students transition to academic and
  campus life
• Educate and provide students with accurate
  information and resources on academic policies,
  procedures and requirements

25
Process and Learning Outcomes

• Most difficult part of the process
• Be sure to focus on advising outcomes, not
  outcomes for college experience
• Process/learning outcomes intertwined
• What should students be able to know or do as a
  result of an advising interaction?
• What do we have to do to facilitate this learning?

26
We focused on types of advising interactions

• New student orientation
• First-year seminars
• Students on probation
• Students being reinstated
• General advising meetings

27
Program ReviewGuiding Questions

• Assessment plan guides all of our operations
  too encompassing for the program review
• Established guiding questions to help focus the
  assessment done for the unit self-study

28
Step 2Conducting Assessment
29
Overall Assessment Approaches

• Sought involvement of key stakeholders in
  planning and implementation (formed advising
  assessment committee).
• Selected outcome measures that were valid,
  reliable, aligned with goals and learning
  outcomes.
• Attempted to understand what processes lead to
  particular outcomes the why and the what.
• Employed qualitative and quantitative methods.
• Used multiple measures from different sources.
• Employed summative and formative approaches.

30
Multiple Sources of Information

• Spring 2005 web-based survey (random sample of UC
  students)
• Pre-post questionnaire administered in First Year
  Seminar Courses
• Spring 2006 web based survey (students recently
  certified into schools)
• Spring 2006 survey for UC professional advisors
  

31
Institutional Review Board

• All assessment was vetted through the
  Institutional Review Board as good practice in
  addition to ensuring that we could present or
  publish on any findings.

32
Factor 1 Interaction Style
33
Factor 2 Knowledgeable
34
Factor 3 Student Familiarity
35
Factor 4 Connections
36
Factor 5 Professionalism
37
Factor 6 Academic Goal Facilitation
38
Statistically Significant Relationships
39
Improvements Over Time 1999 and 2005
Improvements on all 5 factors are statistically
significant!
40
Students Who Met with Same Advisor Better
Outcomes
41
What Students Learned From AdvisingPre-Post
Results (N294)

• Academic Goal Setting
• I have a good understanding of my academic
  goals.
• I have learned to accept responsibility for
  achieving my academic goals.
• Career Decision Making
• I am able to identify links between my chosen
  major and possible careers.
• I have a good understanding of how to decide on a
  major or future career.
• Goal Persistence
• At the present time, I am energetically pursuing
  my academic goals.
• There are lots of ways around any school-related
  problems that I may face.
• Engagement
• I have a good understanding of ways to become
  engaged at IUPUI (co-curricular and campus life
  activities).
• I feel a sense of belonging at IUPUI.

42
Significant Predictors of Fall 2005 GPA (N358)

• Academic Success Strategies (e.g.," I can
  maintain a balance between school and my personal
  life.)
• Confidence in Degree Completion (e.g., I feel
  confident that I will complete my degree in a
  timely manner.)
• Academic Goal Persistence (e.g., At the present
  time, I am energetically pursuing my academic
  goals.)

43
Most Important Aspects to Students

• 1.  Treats me with respect.
• 2.  Is trustworthy.
• 3.  Provides accurate information.
• 4. Is knowledgeable about the general
  requirements needed for a degree in my school.
• 5. Is knowledgeable about the degree requirements
  needed for a degree in my school.
• 6. Is a good listener.
• 7. Treats me fairly.
• 8. Understands my schools (e.g., Liberal arts,
  business, public and environmental affairs,
  science, etc.) Academic rules and policies.
• 9. Provides adequate information.
• 10. Is approachable.

44
Most Likely to Experience/Satisfaction

• 1. Treats me with respect.
• 2.  Is friendly.
• 3.  Treats me fairly.
• 4.  Understands university rules and policies.
• 5.   Is trustworthy.
• 6.   Is approachable.
• 7.   Is a good listener.
• 8.   Is open-minded when making decisions.
• 9.   Provides information in a timely manner.
• 10. Provides accurate information.

45
Step 3The Actual Review
46
Self-Study Report

• Completed self-study report guided by
  expectations set by institutional program review
  guidelines
• Current state of affairs
• Strengths/areas for improvement
• Thorough reporting of all assessment outcomes

47
External Reviewers

• Nancy King, Vice President for Student Success
  and Enrollment Management, Kennesaw State
• Charlie Nutt, Executive Director, NACADA
• Josh Smith, IUPUI Assistant Professor of
  Education and NACADA Research Committee
• Miriam Langsam, Retired Associate Dean of Liberal
  Arts
• Angela Allen, Guidance Counselor, Indianapolis
  Public Schools

48
Review Day

• Charge and context with the Vice Chancellor for
  institutional improvement
• Meetings with key staff and stakeholders
  (including students)
• Wrap-up with Vice Chancellor and discussion of
  preliminary conclusions

49
Review Report

• Received within 60 days of visit by external
  reviewers
• Submit response to reviewers report within six
  months
• Discussion of report and response with Vice
  Chancellor for Institutional Improvement, Dean of
  Faculties and Dean of University College

50
Step 4Planning and Action
51
Unit Actions

• Presented results of assessment and review to
  staff on several occasions
• Every staff member received a copy of the
  reviewers report
• Established a program review action committee
• Public presentations to University assessment
  committee and UC faculty

52
Mission Review

• Gathered information from staff through
  activities at several staff meetings
• Review action committee drafted statement of
  mission, values and vision
• Held day-long retreat to refine mission statement
  and establish committees based on review
  recommendations

53
Moving toward the Future

• Committees formed to address
• Assignment of advisors
• Meeting the needs of non-traditional students
• New Student Orientation
• Advisor development (inquiry and scholarship)
• Advisor training
• Health advising network

54
Important Considerations

• How can we effectively continue to assess the
  processes and outcomes of UC advising?
• What are we learning from assessment results?
• How can we leverage assessment results to improve
  advising?
• How can we sustain the momentum for quality
  assessment and improvement?

55
http//uc.iupui.edu/staff/research.asp
56
Contact Information

• Cathy A. Buyarski (cbuyarsk_at_iupui.edu)
• Michele J. Hansen (mjhansen_at_iupui.edu)
• Chris Maroldo (cmaroldo_at_iupui.edu)