What Works in Promoting Student Success

1
What Works in Promoting Student Success

• By Steve Robbins AVP Research, ACT, Inc.
• Presented as part of the University of Michigan
  Forum on Diversity, Merit and Higher Education

2
Putting the Pieces Together Hoping They Work
Concept and original artwork by Jorge O.
Calvillo, Canyon Springs High Schools, Moreno
Valley, California, winner of the ACT High School
Impact poster contest.
3
Agenda

• Why College Students Stay
• What Works in Promoting College Success
• Risk, Service Use, Success A Case Study
• Putting It All Together

4
About ACT Research The Propeller Heads

• Access to Longitudinal Data
• 8th - 10th 12th linked files 140,000 per
  cohort
• ACT/COMPASS linked to college transcript/outcomes
  (n 1 million and counting)
• Student Readiness study 15,000 students at 48 2-
  and 4-year institutions moving into year 6
• Use of National Student Clearinghouse data
• Policy, Statistical, Measurement, Career
  Transition, I/O, and Survey Research Staff

5
Harris, S. (1991) Cant you guys read? Cartoons
on Academia. Rutgers University Press. New
Brunswick, NJ, p.74
6
Why College Students Stay 4-year Colleges

• First-year GPA has large effects on likelihood of
  retention and transfer
• Motivation (Academic Discipline) and
  pre-collegiate academic preparation have indirect
  effects on retention and transfer by working
  through 1st-year GPA
• Social connection has a direct effect on
  retention.
• SES predictive of transfer behavior
• Higher SES students transfer while poor students
  give up
• African-American Students have high commitment
  but difficulty with classes resulting in higher
  drop-out rates.

Robbins et al. (2006) Allen et al. (in press)
7
Why College Students Stay 2-year Colleges

• Pre-collegiate academic preparation is the
  strongest predictor of all outcomes
• Motivation (Academic Discipline) distinguishes
  retained and graduating students from transfer
  and drop out
• Social connection has effects only for those
  students who transferred to 4-year institutions
• Socioeconomic status distinguishes all groups
  from drop-out higher SES kids are likely to
  transfer and low SES kids drop out

8
Common Findings across 2- 4-year Studies

• Academic preparation, Socio-Economic Status (SES)
  and Academic Discipline are all critical
• 1st year GPA essential for 4-year students
• Students socially connected are more likely to
  transfer upon 2 year graduation or stay (4 year)

9
Were looking for a more comprehensive research
strategy than simply Google it.
Harvard Business Review. (July-August 2007)
10
What Works . . .
Meta-Analysis and Validity Generalization as Key
Tools

• Summarizing the effect of something over multiple
  data points
• Create confidence intervals of the true effect
  size
• Interpretation of multiple studies, better than
  any individual study

11
Testing Integrated Meta-Analytic Path Analysis
The Effects of College Interventions on College
Outcomes as Mediated by PSFs
Robbins et al. (2007)
12
Categorizing College Interventions

• Orientation (21 hours) summer, early fall,
  time-limited
• Freshman Year Experience (45 hours)
• Academic (8 hours)
• Study skills
• Learning strategies
• Note-taking
• Self-Management (6 hours)
• Stress management
• Self-control
• Anxiety management
• Hybrid of Academic Self-Management (12 hours)

13
Categorizing Psychosocial Factors (PSFs)
3 Categories

• Motivation
• Academic Discipline
• Commitment to College
• Self-Regulation
• Emotional Control
• Academic Self-Confidence
• Social Engagement
• Social Connection
• Social Activity

Robbins, S., Allen, J., Casillas, A., Peterson,
C., Le, H. (2006) Robbins, et al. (2004)
14
Effects of Intervention on Outcomes
EFFECT SIZE 0 No Effect
.1 - .2 Small .2 - .4 Moderate
.4 Strong
15
Effects of Intervention on Psychosocial Factors
EFFECT SIZE 0 No Effect
.1 - .2 Small .2 - .4 Moderate
.4 Strong
16
Effect Sizes of PSFs on College Outcomes
EFFECT SIZE 0 No Effect
.1 - .2 Small .2 - .4 Moderate
.4 Strong
Robbins, et al. (2004)
17
Testing Integrated Meta-Analytic Path Analysis
PSFs - Motivation - Self-Regulation
Indirect .11, .11, -.13
Total .24, .21, .57

• Intervention
• Academic
• Self-Mgt
• Hybrid

GPA
Direct .13, .10, .70
About 50 of the effect of Academic and Self-Mgt
interventions on GPA are through relevant
PSFs The effect of Hybrid intervention on GPA is
fully direct
18
Testing Integrated Meta-Analytic Path Analysis

• PSFs
• - Motivation
• Self-Regulation (only for Self-Mgt)
• Social Engagement (only for FYE)

Indirect .06, .05
Total .27, .05

• Intervention
• Self-Mgt
• FYE

Persistence
Direct .21, .00
Self-Mgt Intervention has a strong direct effect
on persistence Though small, 100 of the effect
of FYE intervention on persistence is through
relevant PSFs
19
What it means

• Interventions with academic focus are key
• Boost academic interventions using self
  management strategies, i.e., Hybrid
• Align specific interventions to narrowed outcomes
  (PSF and/or success) to increase treatment effect
• Rethink goals focus of Freshman Year Experience
• Understand mediating role of motivation and
  self-regulation factors to promote student success

20
Giving Guidance
Herzog Miller (1985)
21
Risk, Service Use, Success A Case Study

• Public Southwestern University
• 4-year Institution
• Over 13,000 Undergraduate Students and 1,100
  Faculty
• 31 Dropout Rate after Freshman Year
• Implemented Card Swiping System to Monitor
  Resource Use
• Robbins et al. 2007

22
3 Critical Elements

• Coordinate resource service use options
• Academic services
• Recreational resources
• Social resources
• Academic referrals
• Advisory / career services
• Determine student levels of risk by using
  admission test scores and Psychosocial Factors
• Target and contact students for help

23
Program Model

• Identify At-risk Students by testing entire
  entering class and identify bottom 24 on
  retention risk
• Use SRI Scale scores to match student needs with
  University and ACT/SRI resources (e.g., advising,
  FYE, academic support)
• Provide interpretive feedback to students on
  importance of motivation and academic success
• Evaluate results

24
Resource Services Utilization
Robbins et al. (2007)
25
Association of Risk Level Academic Service Use
on Retention 1st-year GPA
.08
.24
26
Retention Recommendations based on Case Example

• Designate a visible individual to coordinate a
  campus-wide Retention Planning Team
• Conduct Systematic Analysis
• Academic/Non-academic Characteristics/Needs
• Persistors Non-persistors broken down by
  race/ethnicity
• Implement Early-Alert Assessment and Monitoring
  System
• Academic/Non-academic Factors
• Identify At-Risk Students

27
Putting It All Together

• Psychosocial Factors Supplement but Do Not
  Replace Traditional Indicators

28
Ensuring Student Success

• Academic preparation performance are at the hub
  of all else
• Be clear on Goals
• Satisfaction, learning, persistence are not the
  same
• Be strategic in your use of resources
• Move the Mountain to the students
• Dont be afraid of intrusive
• advising

29
Herzog Miller (1985)
30
Questions?
31
References

• ACT, Inc. (2008). What We Know about College
  Success Using ACT Data to Inform Educational
  Issues. Iowa City, IA Authors.
• ACT, Inc. (2007). State of College Readiness for
  Latino Students. Iowa City, IA Authors.
• ACT, Inc. The National Council for Community
  and Educational Partnerships. (2007). Using
  EXPLORE and PLAN data to evaluate GEAR UP
  programs. Iowa City, IA Authors.
• ACT, Inc. (2004). Schools Involving Parents in
  Early Postsecondary Planning. Iowa City, IA
  Authors.
• ACT, Inc. (2002). Creating Seamless Educational
  Transitions for Urban African American and
  Hispanic Students. Iowa City, IA Authors.
• Allen, J., Robbins, S., Casillas,A., Oh, I.-S.
  (in press). Why college students stay Using
  academic performance, motivation, and social
  engagement constructs to predict third-year
  college retention and transfer. Research in
  Higher Education
• Braxton, J., Sullivan, A., Johnson, R. (1997).
  Appraising Tintos theory of college student
  departure. In J. C. Smart (Ed.) Higher Education
  Handbook of Theory and Research, 12, 107-158. New
  York Agathon.
• Bucheri, C., Hampton, T., Voelker, V. (eds.)
  (1991). The Student Body Great Cartoons from the
  Kappan. Phi Beta Kappa. Bloomington, IN.

32
References (cont.)

• DesJardins, S. L., Kim, D. O, Rzonca, C. S.
  (2002-2003). A nested analysis of factors
  affecting bachelors degree completion. Journal
  College Student Retention, 4, 407-435.
• Habley, W. McClanahan, R. (2004). What Works in
  Student Retention All Survey Colleges. ACT,
  Inc. Iowa City, IA.
• Harris, S. (1991) Cant you guys read? Cartoons
  on Academia. Rutgers University Press. New
  Brunswick, NJ
• Herzog, K. Miller, M. P. (eds.) (1985).
  Scholarship More Great Cartoons from the Kappan.
  Phi Beta Kappa. Bloomington, IN.
• Horn, L. Nevill, S (2006). Profile of
  undergraduates in U.S. postsecondary education
  institutions 2003-2004 With a special analysis
  of community college students (NCES 2006-184).
  U.S. Dept. of Education. Washington, DC National
  Center for Education Statistics.
• Larson, Gary (1995). The Far Side Gallery 5.
  Universal Press Syndicate. Kansas City, MO
• Le, H., Casillas, A., Robbins, S., Langley, R.
  (2005). Motivational and skills, social, and
  self-management predictors of college outcomes
  Constructing the Student Readiness Inventory.
  Educational and Psychological Measurement, 65,
  482-508.
• Lotkowski, V., Robbins, S., Noeth, R. (2004).
  The role of academic and non-academic factors in
  improving college retention. ACT Policy Report.
  Iowa City, IA ACT, Inc.

33
References (cont.)

• Pascarella, E. T., Terenzini, P. T. (2005). How
  College Affects Students A Third Decade of
  Research. San Francisco Jossey-Bass.
• Peterson, C. H., Casillas, A., Robbins, S. B.
  (2006). The Student Readiness Inventory and the
  Big Five Examining social desirability and
  college academic performance. Personality and
  Individual Difference, 41, 663-673.
• Porter, S.R. (2003-2004). Understanding Retention
  Outcomes Using Multiple Data Sources to
  Distinguish Between Dropouts, Stopouts, and
  Transfer-Outs. Journal of College Student
  Retention Research, Theory Practice, 5(1),
  53-70.
• Robbins, S. B., Allen, J. Casillas, A., Akamigbo,
  A., Saltonstall, M., Cole, R., Mahoney, E.
  Gore, P.A. (2007). Associations of Resource and
  Service Utilization, Risk Level, and College
  Outcomes. Manuscript submitted for publication.
• Robbins, S. Allen, J., Casillas, A., Peterson,
  C., Le, H. (2006). Unraveling the differential
  effects of motivational and skills, social, and
  self-management measures from traditional
  predictors of college outcomes. Journal of
  Educational Psychology, 98, 598-616.
• Robbins, S. B., Lauver, K., Le, H., David, D.,
  Langley, R., Carlstrom, A. (2004). Do
  psychosocial and study skill factors predict
  college outcomes? A meta-analysis. Psychological
  Bulletin, 130, 261-288.
• Robbins, S., Oh, I., Button, C., Le, H. (2007).
  The effects of college interventions on
  psychosocial mediators and academic and
  persistence outcomes An integrated
  meta-analysis. Manuscript submitted for
  publication.
• Swail, W. S. (2004, January 23). Legislation to
  improve graduation rates could have the opposite
  effect. The Chronicle of Higher Education, B16.

34
What Works in Promoting Student Success

• Correspondence regarding this presentation should
  be addressed to
• Steve Robbins, AVP Applied Research, ACT, Inc.,
  PO Box 168, Iowa City, IA 52243-0168 or email
• steve.robbins_at_act.org
• phone [phone]