Findings from a National Longitudinal Study of the RISE Program

1
Findings from a National Longitudinal Study of
the RISE Program

• P. Wesley Schultz
• California State University, San Marcos

Presented at the 2009 NIGMS Council Meeting,
January 23, 2009, Washington DC.
2
Findings from a National Longitudinal Study of
the RISE Program

• Research Team
• Mica Estrada-Hollenbeck Richard Serpe
• Anna Woodcock David Morolla
• Randie Chance Paul Hernandez
• Maria Aguilar Perla Rivas
• Victor Rocha Brian McDonald

3
Minority Training Programs

• NIH values diversity
• Underrepresentation is an important issue
• STEM and biomedical sciences
• Women in science
• Racial and ethnic representation (Black, Native
  American, Hispanic/Latino)
• Many NIH-funded programs promote research careers
  among members of underrepresented groups
• Primarily through NIGMS-MORE

4
Do These Programs Work?

• 1. National-level Data
• 80 increase in minority PhDs since 1977
• 62 increase in minority population during same
  period
• 2. Site-specific Data
• Each NIH program submits yearly progress reports,
  and longer-term evaluation results
• Typically track their student success, but what
  about a control group?

5
Do These Programs Work?

• 3. Large-Scale Evaluation
• Assessment of NIH Minority Research and Training
  Programs (Phase III). Science, 2006.
• Surveyed students from 47 programs (1970 -99)
• Substantial limitations in finding former
  students
• 5,371 identified participants, of which data was
  obtained from 739 (only 83 trainees funded as
  undergraduates--19 USTAR, 31 Bridges, 33 COR)

6
Problems with Existing Data

• 1. Programs can cherry pick students who are
  likely to succeed
• - These students are likely to succeed without
  the program (growing new talent versus harvesting
  abilities)
• 2. No control group
• 3. Long-term evaluations outside of funding scope
  
• 4. No way to examine the mechanisms of success
• 5. Retrospective accounts can be biased

7
Overview The Science Study

• Prospective, propensity matched control
• Longitudinal study of R.I.S.E. and MARC students
• Participants from 25 programs nationwide
• Twice yearly surveys from students
• Matched control group
• For each R.I.S.E. or MARC student, we find a
  similar student who does not go through the
  program
• Matching variables ethnicity, gender, major,
  GPA, intention to become a scientist, enrollment
  level (LD, UD, Grad)
• Secondary matching age, parental education,
  community college transfer, English as first
  language

8
Overview The Science Study
9
Longitudinal Panel

• Entering Panel N1,228 (Fall 2005)
• 457 R.I.S.E./MARC
• 457 Matched Control
• 157 Other funded (and 157 matched)

10
Longitudinal Panel
11
Survey Data Collection

• Data collected through web interface
• www.TheScienceStudy.com

12
Survey Measures

• Psychological constructs
• self-efficacy, scientific identity, task-goal
  orientation, value orientation, ethnic identity,
  integrative identity, satisfaction, self- esteem,
  etc.
• Accomplishments/milestones
• publications, presentations, research
  participation, conference attendance, degree
  completion, etc.
• Perceptions of the academic environment
• mentorship, belonging, financial support and MTP
  experience

Survey coverage 93.1
Response rates
13
Questions we can (ultimately) answer
1. Does participating in the RISE/MARC program
cause an increase in the likelihood that a
minority student will pursue a career in the
biomedical sciences? (summative) 2. What are the
types of activities that students involved in
RISE/MARC programs are exposed to? 3. Are there
some types of students who benefit more from the
RISE/MARC program than others? 4. Are there
elements of the RISE/MARC program that are linked
with student success? (Research experience,
faculty mentoring, financial support, motivated
peers, resume of accomplishments)
14
Preliminary Findings

• Basic effects (summative outcome)
• Intention to become a scientist
• Graduation rates
• Applications and acceptance to doctoral programs
• 2. Program elements (mediators)
• Financial support, research experience, faculty
  mentorship
• 3. Reasons for success (process)
• Self-efficacy, identity, values

15
Q To what extent do you intend to pursue a
career as a biomedical scientist? (0 10)
Note Juniors and Seniors at screening (W0).
Change over time analyses conducted as a
hierarchical linear model, with both linear and
quadratic terms. RISE students continuously
funded, and MATCH students never funded by any
program. Dropped students who were at one time
enrolled in RISE but did not complete it (N55).
16
C1.58
Q To what extent do you intend to pursue a
career as a biomedical scientist? (0 10)
Note Juniors and Seniors at screening (W0).
Change over time analyses conducted as a
hierarchical linear model, with both linear and
quadratic terms. RISE students continuously
funded, and MATCH students never funded by any
program. Dropped students who were at one time
enrolled in RISE but did not complete it (N55).
17
Note Juniors and Seniors at screening (W0).
Change over time analyses conducted as a
hierarchical linear model, with both linear and
quadratic terms. RISE students continuously
funded, and MATCH students never funded by any
program. Dropped students who were at one time
enrolled in RISE but did not complete it (N55).
18
Baccalaureate Graduation

• W0 Juniors and Seniors.
• MTP Minority Training Program

19
Graduate School Applications
W0 Juniors and Seniors who have graduated with
B.A./B.S. MTP Minority Training Program.
20
Graduate Medical School Enrollment
1
13
10
W0 Juniors and Seniors who have graduated with
B.A./B.S. MTP Minority Training Program.
21
2. Program Elements (Mediators)
Invited Speakers Workshops
Paid Tuition Stipends
Tutoring
Research Experience
Support to attend professional conferences
Faculty Mentorship
22
2. Program Elements (Mediators)
Note Table reported at the 2007 NIH progress
report meeting. Results based on survey responses
from 25 RISE directors, and 457 RISE students.
23
2. Program Elements (Mediators)

• Mediation analyses of programs
• Focus on three program elements
• Research Experience
• Faculty Mentorship
• Financial Support (hours spent in employment off
  campus)

24
Mediation Research Experience
Research Experience
a.36
b2.38
Intention to Pursue a Career in the Biomedical
Sciences
c1.58
RISE Status
c .68
Note a, b and c paths are unstandardized
coefficients. p lt .05, p lt .001 Bootstrapped
Indirect Effect Mean 0.68, CI99 1.31 to
0.45 Sobel Z 4.67, p lt .001
25
Note Change over time analyses conducted as a
hierarchical linear model, with both linear and
quadratic terms. Analyses are based on students
who were undergraduates (jr. or sr.) at W0.
Propensity score (W0) used as time invariant
covariate. RISE students continuously funded,
and MATCH students never funded by any program
and enrolled on a RISE campus. Research is any
research experience ever during undergraduate
education. Intention to pursue career as
biomedical scientist.
26
Mediation Faculty Mentor
FacultyMentor
a.32
b.78
Intention to Pursue a Career in the Biomedical
Sciences
c1.58
RISE Status
c 1.33
Note a, b and c paths are unstandardized
coefficients. p lt .05, p lt .001 Sobel Z
2.08, p lt .05
27
Mediation Financial Support
FinancialSupport
a-.26
b-.42ns
Intention to Pursue a Career in the Biomedical
Sciences
c1.60
RISE Status
c 1.49
Note a, b and c paths are unstandardized
coefficients. p lt .05, p lt .01 Sobel Z
.85, ns Financial support is hours spent working
off campus
28
(No Transcript)
29
3. Process of Influence
30
3. Reasons for Success (process)
Scientific Self-Efficacy (rule)
b.98
a.28
Intention to Pursue a Career in the Biomedical
Sciences
c2.55
Research Experience
c2.28
Note a, b and c paths are unstandardized
coefficients. p lt .05, p lt .01 Bootstrapped
indirect effect mean .27, CI99 .06 to
.56 Sobel Z 2.57, plt.01
31
3. Reasons for Success (process)
Scientific Identity (role)
b1.34
a.56
Intention to Pursue a Career in the Biomedical
Sciences
c2.55
Research Experience
c1.80
Note a, b and c paths are unstandardized
coefficients. p lt .05, p lt .01 Bootstrapped
indirect effect mean .75, CI99 .33 to
1.23 Sobel Z 4.37, plt.001
32
3. Reasons for Success (process)
Scientific Values (value)
b1.08
a.24
Intention to Pursue a Career in the Biomedical
Sciences
c2.51
Research Experience
c2.24
Note a, b and c paths are unstandardized
coefficients. p lt .05, p lt .01 Bootstrapped
indirect effect mean .27, CI99 .01 to
.60 Sobel Z 2.43, plt.01
33
Key Points

• 1. NIH programs for undergraduates (RISE and
  MARC) have a strong effect on student intentions
  to pursue a career in the sciences.
• 2. RISE and MARC undergraduates have higher
  graduation rates, applications to doctoral
  programs, and enrollment in graduate programs
  (preliminary).
• 3. RISE works primarily because it provides
  students with research experience.
• 4.

34
(No Transcript)