Why Do College Coaching Interventions Work?

1
Why Do College Coaching Interventions Work?

• Scott Carrell, UC Davis and NBER
• Bruce Sacerdote, Dartmouth College and NBER

2
US College Completion Ranks 11-13thth In OECD

Cascio Gordon Clark JEP 2008
3
US Dept of Ed Trio Programs

• 879 million in FY 2011
• GEAR up, Upward Bound, Talent Search
• Fundamental tenets
• Catch students early..by 8th grade to get them
  college ready
• Avoid discussion of randomized evaluation
• Bruce and Scott dont know what they are talking
  about

4
US Economist/Ed School Programs

• Send letters to top students (Hoxby and Turner)
• Auto fill the FAFSA (Bettinger Long Oreopolous
  Sabonmatsu)
• Send text messages (FAFSA reminders or deadline
  reminders) (Castleman and Page)
• Offer remote coaching or application help
  (Phillips and Reber)
• Carrell and Sacerdote in person help, financial
  incentives, likely letters
• Clearly different populations in all of these

5
2010 Cohort Frequency Counts 10th Grade Math
Scores for Non College Goers
6
The Mission

• Ask whether simple interventions (bonuses,
  mentoring, likely letters) very late in the game
  can have a meaningful impact on long term
  outcomes
• Attending, persisting, graduating, income
• Use Clearinghouse data to track into College
• Are these marginal college going students very
  different than the average?
• For whom does it work? Try to infer mechanisms
  using baseline interests, preparation,
  personality measures
• SAT Questionnaire, survey data

7
The Answer

• Yes
• If you are female
• Or a guy who has not taken the SATs
• If you do not have strong parental and teacher
  support for completing applications
• If you are not extraverted
• What doesnt work
• Likely letters
• Letters of encouragement (Delaware, NH)
• Cash Bonuses alone

8
Hypotheses

• Behavioral
• Fear of process
• Lack of easily obtained information
• Procrastination/disorganization
• Largely rational have already optimized
• Need expert/ adult help

9
What is the Intervention

• Guidance counselors identify high school seniors
  at risk of not applying to college
• Ideally students who have expressed interest in
  applying
• But have taken few steps
• Typically identify in mid-December
• We randomly choose half for the treatment group
• Guidance staff then invites treatment group to
  participate

10
What is the Intervention (2)

• We visit once per week for 3-5 weeks
• Pair up each HS student with a Dartmouth student
• 1.5-4 hours per session
• Start and complete applications
• Send transcripts as needed
• Pay application fees
• Complete common apps, two year schools
• Start the FAFSA
• Sign up for SATs send scores
• Incentives
• Free coaching
• App fees paid
• 100 cash bonus upon completion
• Get out of class
• Pizza

11
The Process
12
Mission Accomplished (?)
13
Challenges Faced By HS Students And Guidance
Counselors

• Horrible internet access and computers
• In first year, many schools had BLOCKED the
  application sites for all NH Community Colleges
• When that was fixed, we still had credit card
  problems
• In some cases we brought our own laptops and MiFi
  cards
• Students and parents unfamiliar with Common App,
  FAFSA, online forms
• Students were often selected to be the
  procrastinators

14
Treatment and Control Standardized Reading Scores

15
Standardized Math Scores Treatment Group Versus
All Non Experimental

16
Table 1 Summary Statistics for Treatment and
Control Groups
    Control Group     Control Group     Control Group     Mentoring Treatment     Mentoring Treatment     Mentoring Treatment  
Variable Obs Mean Std. Dev Obs Mean Std. Dev
             
Accepted Treatment 902 0 0 871 0.454 0.498
10th Grade Math Score (Standardized) 798 -0.480 0.937 778 -0.286 0.943
10th Grade Reading Score (Standardized) 799 -0.436 0.928 772 -0.278 0.966
Math gt 50th Percentile in State 798 0.312 0.464 778 0.335 0.472
Reading gt 50th Percentile in State 799 0.350 0.477 772 0.398 0.490
Math gt75th Percentile 798 0.164 0.371 778 0.185 0.389
Reading gt 75th Percentile 799 0.213 0.410 772 0.224 0.417
Free and Reduced Lunch Eligible 902 0.277 0.448 871 0.286 0.455
Male 902 0.548 0.498 870 0.575 0.495
Non-white 902 0.173 0.378 871 0.201 0.401
Any College (Clearinghouse) 902 0.438 0.496 871 0.592 0.492
Four Year College (Clearinghouse) 902 0.169 0.375 871 0.276 0.447
Persist for First Two Years Post Grad 902 0.195 0.397 871 0.240 0.427
Persist in a Four Year College 902 0.094 0.292 871 0.115 0.319
Enrolled 3 Semesters   902   0.237   0.426   871   0.292   0.455  

17
Table 2 Show Treatment Status Unrelated to
Pre-Treatment Characteristics
  (1) (2)
  Treatment Status Men Treatment Status Women
     
Standardized 10th Grade Math Score 0.001 0.041
    (0.012) (0.025)
Standardized 10th Grade Reading Score -0.025 -0.006
    (0.014) (0.020)
Free Reduced Lunch Eligible -0.043 0.073
    (0.027) (0.046)
Student is Nonwhite 0.019 -0.038
    (0.032) (0.057)
     
Observations 1216 866
R-squared 0.355 0.321
F Pre-Treat Variables 1.281 2.109
p-value 0.294 0.098

Students are randomly assigned to treatment
within high school. Data include 2009-2014
cohorts. Regressions include high schoolcohort
dummies which is the level at which randomization
occurred. Standard errors are clustered at the
high schoolcohort level. Regressions include
birthyearcohort dummies to control for students'
age within grade.
18
Effect of Treatment Status on Applying
  (1) (2) (3) (4)
  Apply to Any College (Survey) (OLS) Apply to Any College (Survey) IV Estimate Women Apply to Any College (Survey) OLS Men Apply to Any College (Survey) OLS
         
Mentoring 0.274 0.375 0.294 0.243
Treatment   (0.050) (0.068) (0.049) (0.078)
         
Observations 859 859 391 468
R-squared 0.234 0.309 0.293 0.231
Survey data with 50 response rate. Control mean
is .72. Standard errors in parentheses, includes
high school cohort f.e. , age controls,
male significant at 10 significant at
5 significant at 1
19
Baseline Treatment Effects on Enrollment in Any
College
  (1) (2) (3) (4) (5)
  Whole Sample Women Men Did Not Take SAT Took SAT
           
Effects on Enrollment Any College          
Mentoring Treatment (OLS) 0.060 0.146 0.007 0.083 0.035
    (0.018) (0.042) (0.025) (0.026) (0.035)
Transcript Only (OLS) -0.005 0.005 0.000 0.035 -0.049
  (0.019) (0.034) (0.021) (0.034) (0.035)
           
Mentoring Treatment (IV) 0.133 0.299 0.017 0.160 0.086
    (0.041) (0.087) (0.061) (0.047) (0.085)
First Stage for IV          
Mentoring Treatment 0.463 0.500 0.429 0.511 0.444
  (0.039) (0.044) (0.042) (0.033) (0.070)
Observations 2,623 1,114 1,509 1,453 1,170

Outcome variable is a dummy equal to 1 if the
student has any enrollment in college including 2
year or four year colleges. Outcome variables
are based on the Nation Student Clearinghouse
data. Data include 2009-2014cohorts. Regressions
include high schoolcohort dummies which is the
level at which randomization occurred. Standard
errors are clustered at the high schoolcohort
level. plt0.01, plt0.05, plt0.1
20
Baseline Treatment Effects on Enrollment in A
Four Year College
Effects on Enrollment Four Year College Whole Sample Women Men Did Not Take SAT Took SAT
Mentoring Treatment (OLS) 0.057 0.107 0.020 0.103 -0.005
  (0.018) (0.031) (0.028) (0.026) (0.033)
Transcript Only (OLS) 0.001 0.007 0.003 0.002 -0.038
  (0.015) (0.022) (0.028) (0.012) (0.030)
Mentoring Treatment (IV) 0.125 0.222 0.047 0.202 -0.018
    (0.037) (0.062) (0.068) (0.048) (0.083)
First Stage for IV          
Mentoring Treatment 0.463 0.500 0.429 0.511 0.444
  (0.039) (0.044) (0.042) (0.033) (0.070)
Observations 2,623 1,114 1,509 1,453 1,170

21
Treatment Effects on Persistence in College
(Women)
  (1) (2) (3) (4) (5)
  Women Women Women Men Women
        No SAT Data  
  Enrolled in 3 Semesters Enrolled Any College Both School Years Post Graduation Enrolled Four Year College Both School Years Post Graduation Enrolled Four Year College Both School Years Post Graduation Enrolled Second Year Conditional on Enrolled First Year
           
Mentoring Treatment 0.129 (0.053) 0.105 (0.042) 0.097 (0.030) 0.014 (0.041) -0.040 (0.066)
  0.129 (0.053) 0.105 (0.042) 0.097 (0.030) 0.014 (0.041) -0.040 (0.066)
           
Observations 535 535 535 445 263
R-squared 0.172 0.123 0.105 0.220 0.165

Outcome variables are four different ways to
measure persistence into the second year of
college. Sample is limited to women in the
2009-2012 cohorts.
22
Evidence on Mechanisms Interaction of Mentoring
Treatment with Sources of Assistance on
Applications
  (1) (2) (3) (4) (5) (6)
SAT Questionnaire Measure Coefficients on Treatment SAT/ Survey Measure Coeff on Treatment Indicator Coeff on SAT/ Measure N Mean SAT/Surv indicator regressed on Male Dummy
             
Do Not Need Help With Educ Planning -0.116 (0.059) 0.126 (0.058) 0.049 (0.039) 1302 0.829 0.015 (0.004)
Survey Measure            
Parents Help With -0.131 0.118 0.133 724 0.468 0.014
College Applications (0.067) (0.045) (0.041)     (0.037)
             
Teacher Helps With -0.165 0.112 0.089 646 0.172 -0.023
College Applications (0.091) (0.030) (0.062)     (0.030)
             
Guidance Counselor -0.009 0.0541 0.037 724 0.312 -0.0982
Helps with College Application (0.069) (0.037) (0.057)     (0.034)

Dependent Variable is Enrollment in Any College
.
23
Interaction of Mentoring Treatment with Beliefs
About Wages/ Tuition
  (1) (2) (3) (4) (5) (6)
Survey Measure Coeff on Treat Survey Measure Coeff on Treat Indicator Coeff on Survey Measure N Mean Survey indicator regressed on Male Dummy
             
Log (Hourly wage at Age -0.159 0.545 -0.052 354 2.931 0.307
30 w. only HS Diploma) (0.078) (0.233) (0.099)     (0.052)
             
Log (TuitionFees -0.019 0.246 -0.023 506 9.052 0.032
Community College) (0.036) (0.310) (0.031)     (0.091)
             
Log (Tuition Fees NH -0.028 0.343 0.015 502 10.076 -0.040
Public University) (0.030) (0.295) (0.023)     (0.085)
             
Need College Degree for -0.010 0.027 0.273 663 0.777 -0.089
Stated Career Goal (0.112) (0.094) (0.055)     (0.032)

Dependent Variable is Enrollment in Any College
.
24
Evidence on Mechanisms Interaction of Mentoring
Treatment with Sources of Assistance on
Applications
  (1) (2) (3) (4) (5) (6)
Survey Measure Coefficients on TreatmentSurvey Measure Coefficient on Treatment Indicator Coefficient on Survey Measure N Mean Survey indicator regressed on Male Dummy
             
Individual Measures            
Likes to meet new people -0.305 0.280 0.150 530 0.723 -0.096
  (0.086) (0.085) (0.055)     (0.039)
Enjoy Amusement Rides -0.287 0.259 0.097 530 0.696 0.031
  (0.136) (0.103) (0.087)     (0.040)
Composite Measures            
Meets Deadlines/ Organized 0.083 (0.189) 0.030 (0.082) 0.096 (0.133) 530 0.343 0.011 (0.022)
Adventuresome -0.275 0.239 0.144 530 0.657 0.017
  (0.179) (0.143) (0.146)     (0.021)
Self-Esteem -0.097 0.136 0.143 552 0.672 0.007
  (0.128) (0.096) (0.092)     (0.028)

Dependent Variable is Enrollment in Any College
.
25
Interaction of Mentoring Treatment with
Personality Traits
  (1) (2) (3) (4) (5) (6)
Survey Measure Coefficients on TreatmentSurvey Measure Coefficient on Treatment Indicator Coefficient on Survey Measure N Mean Survey Measure regressed on Male Dummy
             
Self Esteem            
Believes In Self -0.025 0.083 0.096 552 0.663 0.003
  (0.088) (0.076) (0.053)     (0.041)
Deals Well With Problems -0.063 (0.092) 0.110 (0.075) 0.101 (0.064) 552 0.601 -0.044 (0.042)
Change Important Things -0.057 (0.111) 0.108 (0.088) 0.080 (0.076) 552 0.672 0.027 (0.040)
Solves Problems -0.151 0.186 0.097 552 0.739 0.048
  (0.081) (0.064) (0.070)     (0.038)
Not Easily Pushed Around 0.022 (0.088) 0.055 (0.069) -0.001 (0.052) 552 0.683 0.003 (0.040)

Dependent Variable is Enrollment in Any College
.
26
Interaction of Mentoring Treatment with
Personality Traits
  (1) (2) (3) (4) (5) (6)
Survey Measure Coeff on TreatSurvy Measure Coeff Treat Indicatr Coef Survey Measre N Mean Survey Measure on Male Dummy
Skips Homework -0.047 0.076 0.059 530 0.408 0.075
  (0.088) (0.060) (0.056)     (0.043)
Lose Papers Easily -0.049 0.063 -0.080 530 0.157 0.012
  (0.128) (0.046) (0.086)     (0.032)
Not Organized 0.119 0.022 0.022 530 0.306 0.048
  (0.087) (0.057) (0.055)     (0.040)
Wastes Time 0.055 (0.058) 0.042 (0.056) -0.039 (0.046) 516 0.479 -0.117 (0.044)
Waits Until Last Minute -0.012 (0.093) 0.073 (0.065) 0.103 (0.080) 516 0.411 0.057 (0.044)
Surprised By Deadlines 0.071 (0.078) 0.041 (0.060) 0.106 (0.056) 516 0.481 -0.027 (0.044)

Dependent Variable is Enrollment in Any College
.
27
Some Takeaways

• We find a boots on the ground approach is
  effective
• Appears to compensate for lack of or non take up
  parental or teacher help
• No evidence that its more effective for people
  who struggle with deadlines, planning,
  organization
• Helps students less far along in process, ie non
  SAT takers
• Much more robust effects for the women..lack of
  effects for men very much related to labor market
  opportunities
• Cant find evidence for effects from
• Likely letters/ letters of encouragement
• Cash bonuses along..Though cash bonuses may be
  very important for takeup
• Texting of CCSNH students has not yielded effects
• Nor did letter of encouragement to DE graduated
  seniors

28
More Takeaways

• We find evidence in favor of optimization story
  and need skilled help story
• We dont find evidence for super naïve behavioral
  story
• Or that failure to attend is driven by
  forgetfulness or lack of organization
• But parents/mentors could indeed be solving this
• We suspect that letter based interventions or
  simple text reminders may only work with specific
  groups or in specific contexts

29
Also Mention If Time

• Huge drop off in participation when we removed
  the cash bonus
• Men very likely to say they have a job they
  prefer to college
• And no evidence in ACS of OLS return to two years
  of college
• Still have low average persistence in both treat
  and control
• Cant make the Big Five measures work

30
A Very Few Words on Returns to CollegeOreopoulos
and Salvanes
31
Caveats on Earnings Returns to College

• Large number of papers but sources of ID very
  different
• We havent actually shown that the marginal
  students in current college going interventions
  have earnings gains
• Bettinger, Gurantz, Kawano and I having trouble
  finding earnings effects from winning the
  Cal-Grant
• The one thing we have learned is just how
  ridiculously noisy the outcome is

32
Evidence on Mechanisms Interaction of Mentoring
Treatment with Sources of Assistance on
Applications

Dependent Variable is Enrollment in Any College
.
33
Do We Affect Type of College Attended?

34
Do We Affect Type of College Attended? (2)

Outcome variables measured in IPEDS data. Sample
only includes students in college and for whom we
have IPEDS data
35
Split Sample By Test Score

36
Interaction of Treatment with Immigration Status

Data are from Manchester West 2010,2011 Cohorts.
Sample is roughly 9 immigrants.
37
Mechanisms

• Bonus? Incentive?
• Same or Cross Gender Mentoring?
• What do participants say?
• More effective in schools with fewer resources?

38
Evidence From 2012 Cohort (Coaching Plus 100
Bonus Versus Bonus Alone)

39
Post-Survey Evidence on 100 Bonus

40
Take Up Rates Within Mentoring Group
41
What Aspects Helped the Most?

• 19 of 19 mentioned in person help with
  applications
• 5 of 19 mentioned the 100 bonus
• 3 of 7 men versus 2 of 10 women
• 12 of 19 mentioned our paying for application
  fees
• Rational bc a lot more than 100 for some
  students

42
Treatment Effect by High School
43
Average College Going Versus Effect Size For Women

44
Average College Going Versus Guidance Counselors
Per Student

45
Mentoring Treatment Interaction with Sources of
Disadvantage
46
Mentoring Treatment Interaction with Sources of
Disadvantage
47
Mentoring Treatment Interaction with Sources of
Disadvantage
48
Stories from the HS Students

• gt hey Cambell, its Daniel M from west, one of
  the student you hepled apply for college. If i
  remember correctly, you told me to let you know
  what college i get into. Well, i was recently
  accepted to NHTI in Concord.
• There are still some things i need to take care
  of, but its nothing serious.
• gt P.S. Thanks for you help. Sorry for all the
  flirting

49
Cross Tab of Student Male and Assigned Male Mentor

Mentors were assigned on a first come first
served basis, but when multiple arrivals occurred
at the same time, we had a modest bias towards
same gender pairings. Regressions include a
dummy for being assigned to treatment but not
showing up to be assigned a mentor.
50
Is Cross Gender Mentoring More Effective?
51
Four Theories As To Why It Only Works for Women

• Treatment is actually a complement to ability or
  persistence
• Look for other ways in which treatment might
  interact with sources of advantage, help or test
  scores
• Guys are primed to either be near the top of an
  endeavor or quit
• Dont respond well to feedback that they are in
  the middle
• Labor economics
• Short run returns to college in NH are lower for
  guys
• Are guys negatively selected into the treatment
  group?
• In part bc of 100 bonus?

52
Survey Data on Male And Female Study Subjects

53
Survey Data on Male And Female Study Subjects
(Aspirations and Expectations)

54
How Do Returns to College Differ for Men Versus
Women in NH (Ages31-40)?

55
How Do Returns to College Differ for 21-30 year
old Men Versus Women in NH?

56
Other Interventions Find Effects Differ By Gender?

• Moving to Opportunity only saw decreases in crime
  and increases in school engagement for girls
• Angrist and Lavy bonuses in Israel for obtaining
  Bagrut
• Study one of Oreopoulos and Angrist bonuses plus
  mentoring for U of Toronto students
• Study two did not find any effects
• Dynarksi Ichino find bigger effects of offereing
  finacial aid for women
• Job Training programs affect women more

57
Other Interventions Find Effects Differ By Gender?

• But not at all clear this is true for charter
  school studies
• Boston area charter studies find bigger effects
  for boys
• Are effects bigger for students who are weaker in
  that domain
• Did boys already attend college up to zero
  marginal returns?
• Girls need a shot of confidence or more help
• Or is the treatment a complement to skill?
• Is it ultimately about personality measures,
  follow through
• Our intent is to collect ex-post survey measures
  of personality traits, patients, follow through

58
Additional Interventions

• Send one group a letter offering admission to
  their closest public college and also community
  college
• Weve had fantastic cooperation from admissions
  officers at NH institutions (UNH, UNH Manchester,
  SNHU)
• Not much evidence of an effect yet
• Compare to Hoxby Turner which has large effects
  from letters sent to students in top 10 of SAT /
  ACT distribution
• Our population is very different and perhaps
  needs much more in person help hand-holding

59
Additional Interventions (2)

• Have students meet with Admissions Officers from
  Community College System
• Have Lets Get Ready run the program.
• Adds a serious SAT component
• Ask how much results change when program is
  scaled up/ PI is no longer involved
• And when population of students is broadened

60
Additional Interventions (3)

• Use text messaging to Community College students
  to prevent Summer Melt, encourage persistence,
  help students make the transition to 4 year
  institutions

61
Bottom Lines

• Big differences by gender..critical to better
  understand that..Quite possibly its not a general
  boy phenomenom
• These marginal students persist at same rate as
  average
• Program is enormously successful for immigrants
• And for resource poor high schools
• Suggests that we are in some way compensating for
  missing component rather than reinforcing
  advantage
• Difficult or impossible to evaluate this sort of
  program using endogenous take up
• May be real downsides to offering cash bonuses
• May be real upsides to expensive in person help

62
Could We Estimate the Effects for Women Without a
Randomized Control Group?

63
Compare to Existing Work

• HR Block (FAFSA) study
• The Role of Simplification and Information in
  College Decisions Results from the HR Block
  FAFSA Experiment
• Eric P. Bettinger, Bridget Terry Long, Philip
  Oreopoulos, and Lisa Sanbonmatsu
• Moved enrollment rates from 26.8 to 34.5 percent
• 7 percentage point effect in same age group
• We have a more intensive intervention and
  (arguably/ debatably) twice the effect size
• COACH program (Avery and Kane)
• Kane college mentoring in CA
• 5 percentage point effect on enrollment rates
• Berman Ortiz and Bos (ongoing in LA Unified)
• Phillips and Reber remote coaching in LA and CA

64
Estimate the Effects By Comparing Those That
Accept Treatment to Those That Don't?

65
Modifications in Follow Up Work

• Try W/o the 100 bonus
• May actually increase treatment on treated for
  men
• Eliminate noise of people signing up for wrong
  reason
• Have Lets Get Ready run the intervention
• Include SAT prep. Include juniors
• Try letters of encouragement from college
  admissions departments
• Extensive pre and post surveys to learn more
  about students personalities, career plans,
  stick-toitiveness
• Try emails and texts to spur students to apply

66
Partnered with NH DOE

• NH DOE has been very supportive and flexible
• Commissioners and staff have leant their support
  to the project
• Key pre-treatment and outcome variables collected
  by NH DOE
• Natl Student Clearinghouse subscription
• Test scores and demographics
• Part of a larger Data Warehouse project underway
• Future projects being designed jointly

67
What our High Schools Look Like