Data Driven Retention Strategies for Online Students

1
Data Driven Retention Strategies for Online
Students
Dr. Jeffrey P. BartkovichMarie J.
FetznerFebruary 21, 2004
2
I. Introduction and Welcome

• Welcome and presenter introductions
• Brief audience survey
• Handouts
• Session evaluation

3
Presentation Agenda

• 1230pm 130pmIntroduction, Overview and
  Presentation of Study 1
• 130-140pm Break
• 140pm 220pmPresentation of Study 2
• 220 230pm Break
• 230 315pm Research to Inform Practice
• 315 330pmQuestions and Discussion

4
Purpose of Presentation

• Provide an overview of two online student
  retention/withdrawal studies
• The problem, the research, the findings and the
  applications
• Demonstrate retention strategies and policy
• Discuss a research agenda to inform practice
• Theory development, defining retention, managing
  services and student-to-student advice

5
II. MCC Overview

• MCC is part of SUNY
• Founded in 1961 13,326 FTE in 2003
• Joined SUNY Learning Network (SLN) in 1997
• Spring 2004 MCC SLN Summary Data
• Headcount 3,434
• Online FTE 328.43
• Courses 86 Sections 121
• Sharing in the development and delivery of SLN
  courses

6
MCC Online Courses and SectionsFall 1997 to
Spring 2004
7
MCC Online Program Support

• MCC is western NY SLN faculty training site
• MCC online faculty supported via The Monroe Model
• SLN and local campus participation
• Cross divisional membership
• Comprehensive on-site support
• Students supported via SUNY HelpDesk and MCC
  online and on-site resources

8
MCC Educational Technology Services (ETS)
Organization
9
III. MCC Online Student Retention Study 1

• Characteristics and Attitudes of Non-Retained
  Online Students
• Office of Educational Technology ServicesMonroe
  Community College Dr. Jeff Bartkovich and Marie
  Fetzner

10
The Research

• The Problem/Research Question
• Method
• Study limitations
• Descriptive Statistics
• Correlations
• Significant findings
• Conclusions and Applications to Practice

11
The Problem

• Why are certain MCC online students (those with
  grades of F/W) not succeeding in their online
  courses?

12
Online Retention Study Method

• Archival Data Analysis
• Grades and demographics for online courses
  retrieved from student records database
• Student Survey
• Survey created and administered to sample of
  students who received an F or W in an online
  course for the Fall semesters in 2000, 2001 and
  2002

13
The Survey

• Basic demographic data
• Questions derived from
• Online Faculty
• MCC Student Retention Survey
• E-resources

14
Survey Instrument

• Total of 45 questions in three areas
• Expectations at time of registration
• Satisfaction at time of withdrawal
• Reasons for withdrawal
• Likelihood of enrolling in another online course

15
Research Limitations

• Specific
• Sample of convenience
• Problems with generalizability (n201)
• Difficulty in contacting students who do not
  successfully complete the course
• General
• Orientation procedures and other online student
  services evolve over time
• Course design and faculty experience improve
  constantly

16
Basic Student Demographics OL and Site-based, By
Percent

Spring 2002 Fall 2002
OL Site-based OL Site-based
Female 68 55 71 55
Male 32 45 29 45
Minority 20 25 19 24
Non-Minority 80 75 81 76
Less than 20 18 27 15 30
20-24 33 33 35 32
25-29 16 11 15 11
30 and over 33 29 34 27
17
Archival Data Analysis

• MCC Online vs. On-site
• Grade Distribution
• Grade Rates
• Success Rates in Matched (ftf vs. same section
  online) Courses

18
Overall Grade DistributionOL vs. Site-based, By
Percent
Spring 2002 Fall 2002
Grade OL Site-based OL Site-based
A 34.5 32.8 34.0 28.7
B 20.5 26.1 19.7 26.2
C 10.0 15.3 10.6 15.2
D 3.9 5.0 3.7 5.1
F 12.9 8.2 14.1 7.4
W 16.3 11.4 14.5 10.7

19
Grade RatesOL vs. Site-based, Fall Semester

1999 2000 2001 2002
OL N 970 1,417 1,916 2473
C and Higher 69.7 65.8 65.3 63.6
F/W 24.8 26.4 29.7 30.2

MCC N 43,779 45,235 48,288 52,428
C and Higher 70.3 69.7 70.7 70.7
F/W 20.0 21.3 20.6 20.5
20
Success Rates in Matched Online and Site-based
Courses
On-campus, n 16,291 Online, n 1,719 C or Better C or Better Percentage Point Difference
On-campus, n 16,291 Online, n 1,719 Online Site-based Online Site-based Percentage Point Difference
Total 64.2 69.6 -5.4
Full-time Part-time 56.4 72.3 70.3 66.7 -13.9 5.6
1st time FT 1st time at risk 1st time not at risk 39.7 47.2 66.3 72.2 66.6 74.0 -32.5 -19.4 -7.7
Under 25 years of age 25 years of age or older 54.5 75.2 67.8 75.7 -13.3 0.5
21
Success Rates in Matched Online and Site-based
Courses
On-campus, n16,291 Online, n 1,719 C or Better C or Better PercentagePoint Difference
On-campus, n16,291 Online, n 1,719 Online Site-based PercentagePoint Difference
Part-time 72.3 66.7 5.6
1st time, PT not at risk 73.5 66.4 7.1
Taking Eng, History, Math, Speech, Comm, Music, HVAC 51.6 65.3 -13.6
Not Taking Eng, History, Math, Speech, Comm, Music, HVAC 74.1 73.7 0.4
22
Retention Sample

• MCC vs. Retention Sample
• Gender
• Age
• Ethnicity

23
MCC, MCC SLN and Sample Profile Gender and Age
2002-03 MCC Profile MCCSLN Profile Sample Profile
Female Male 54.7 45.3 70.8 29.2 62.2 34.8
Under 20 20-24 25-29 30-34 35-44 45-59 60 and over 31.8 32.2 11.0 7.6 11.2 6.0 0.2 14.3 36.9 14.0 11.6 16.7 6.5 0.0 14.3 42.9 15.7 8.6 11.4 7.1 0.0
24
MCC, MCC SLN and Sample Profile Ethnicity
2002-03 MCC Profile MCC SLN Profile Sample Profile
Black Hispanic Amer. Ind. NRA Asian Unknown White 16.2 5.1 0.6 0.5 3.2 0.0 74.4 11.0 4.4 0.4 0.3 2.1 0.0 81.7 12.9 5.5 0.0 0.0 3.5 4.5 73.6
25
First-time vs. Experienced Online Student Sample
by Gender

• First-time online students (n 132 68)
• Experienced online students (n 62 32)

82
50
43
19
n 194 7 Unknown
Experienced
First-time
26
First-time vs. Experienced Online Student Sample
by Ethnicity
27
Expectations at Time of Registration
First-Time Students Experienced Students
Can Begin Anytime 41.4 18.8
Online Easier than On-Campus 40.4 22.3
Less Homework than On-Campus 40.0 20.0
Faculty Interaction Required 35.6 15.0
Need Basic Computer Skills 28.3 14.1
Participate Independently 18.9 35.0
Definitely what I expected Sort of what I
expected
28
Satisfaction at Time of Withdrawal
First-Time Students Experienced Students
Own Performance in Course 30.0 14.0
Technical Help with the Course 28.6 13.5
Registration/Orientation 26.4 9.8
SLN in General 23.4 13.9
Directions Provided by Faculty 22.8 14.0
Directions to Get Started 10.9 15.1
Somewhat satisfied Extremely satisfied
29
Explanation for Non-Successin Online Course
Top Ten Factors First-Time Students Experienced Students
Lack of Motivation 46.3 24.4
Course Taking too Much time 43.0 18.8
Instructors Teaching Style 43.0 18.4
Too Many Technical Difficulties 41.9 19.8
Got Behind/Couldnt Catch Up 41.7 22.1
30
Explanation for Non-Success in Online Course
Top Ten Factors (Cont) First-Time Students Experienced Students
Too Much Reading 41.4 20.9
Signed Up for too Many Courses/Had to Drop 41.1 20.4
Course too Unstructured 41.0 21.1
Not Interested in Subject 40.8 20.3
Couldnt Handle Study Plus Other Responsibilities 38.9 21.6
31
Number One Reason for Non-SuccessRanked Order

1. Got behind and couldnt catch up
2. Too much reading
3. Course taking too much time
4. Course too unstructured
5. Too many technical difficulties

32
Number One Reason for Non-SuccessRanked order

• 6. Not interested in subject matter
• 7. Course too difficult
• 8. Signed up for too many courses
• 9. Couldnt handle school with other duties
• 10. Didnt know where to go for help

33
Likelihood to Take Another Online Course
Survey Not a chance/not likely Its possible Somewhat likely/very likely
Fall 2000 31.8 11.6 56.5
Fall 2001 58.1 12.9 29.0
Fall 2002 55.9 11.8 32.3
3 Year Total 48.5 12.2 39.3
34
Likelihood to Take Another Online Course by Age
Age Range Not a chance/not likely Its possible Somewhat likely/very likely
lt20 43.59 15.9 40.6
20-24 34.4 15.6 50.0
25-29 46.4 14.3 39.3
30-34 64.1 7.7 28.2
35-44 63.6 0 36.4
45-59 50.0 0 50.0
60 and over N/A N/A N/A
35
Likelihood to Take Another Online Course by Gender
Not a chance/not likely Its possible Somewhat likely/very likely
Female 52.4 8.1 39.5
Male 40.9 19.7 39.4
36
Likelihood to Take Another Online Course by
Ethnicity
Ethnicity Not a chance/not likely Its possible Somewhat likely/very likely
Black 48.0 16.0 48.0
Hispanic 45.5 18.2 36.3
Asian 49.2 1.6 49.2
White 41.3 11.7 40.0
37
General FindingsArchival Analysis

• When compared to site-based courses at MCC
• Online students are more likely to earn a grade
  of C or better if they are gt25 years of age and
  are part-time students
• First-time, full-time online students are least
  likely earn a grade of C or better

38
General FindingsSurvey Analysis

• Levels of Satisfaction
• First-time students more satisfied at time of
  withdrawal in general, and with their own
  performance
• Experienced students less satisfied with
• Registration procedures
• SLN in general
• Directions from faculty

39
General FindingsSurvey Analysis

• Expectations
• First time students more likely to expect
• Fluid beginning and end dates
• Less homework
• Online easier than site-based courses
• Experienced students more likely to expect
• Limited faculty interaction
• Independent participation in course

40
General FindingsSurvey Analysis

• Expectations
• Online format expectations not accurate
• Level of online interaction with faculty and
  other students not accurate
• Technical expectations (needed PC and typing
  skills) not accurate
• Percentage of W/F students likelihood to take
  another online course is decreasing
• Approximately 1/3 are likely or somewhat likely

41
General FindingsSurvey Analysis

• Main reasons for withdrawal
• Got behind and couldnt catch up
• Course had too much reading, took too much time,
  was too unstructured and too difficult
• Had too many technical difficulties and didnt
  know how to get help
• Not interested in subject matter and dropped
  course to cut course load
• Couldnt handle school, family, work, etc.

42
Significant Positive Correlations

• Likelihood to take another online course
• SLN in general (.604)
• Interaction with other students (.518)
• Directions provided by faculty (.491)
• Directions to get started (.483)
• Interaction with the faculty (.428)
• Technical help with course (.408)

(GPA and credit hours not significant with
likelihood to take another course) Significant
at 0.01 level)
43
Significant Positive Correlations

• Lacked PC skills and
• Lacked typing skills (.970)
• Too many tech difficulties (.741)
• Lack of access to PC and
• Lacked PC skills (.766)
• Financial problems (.731)
• Too many tech difficulties (.688)
• Course too difficult/lacked PC skills (.629)

(Significant at 0.01 level)
44
Significant Negative Correlations

• Likelihood to take another online course
• Didnt like the online format (-.569)
• The course was too unstructured (-.485)
• Felt too alone, not part of the class (-.435)
• Didnt know where to get help (-.324)

(Significant at 0.01 level)
45
Factor Analysis Summary

• Factor analysis conducted
• Data converged into four key groupings in seven
  iterations
• Further discussion on this analysis will be
  conducted later in this presentation

46
Advice from the Literature

• The course design/level of interaction
• The experience of the faculty
• The level of technical support
• A student pre-course orientation
• Manage student expectations upfront
• Professional development and training for faculty
• Standardized course management system

47
Summary of Key Findings

• Retention of online (W/F) MCC students is
  impacted by these characteristics
• First-time (vs. Experienced) online students
• Amount of previous higher education credits for
  full-time online students
• Student age (lt25)
• Lack of PC skills and lack of access to a PC

48
Research Informing PracticeApplications

• Online student orientation CD project
• Development and distribution of the CD
• Phone calls made to first-time online students
• On-site student orientations implemented
• Development of Ten Myths Videostream
• Expansion of PC access for MCC students
• Wireless laptops for check-out in Libraries
• Learning Center support for online students

An analysis was conducted on online student
performance outcomes for those students who
attended on-site MCC orientations
49
Research Informing Practice

• Additional materials development
• SLN Student Awareness Sheet
• Retention strategies included in online faculty
  course development trainings
• Emphasis placed on pre-course activities and
  first week of course interactions
• Promote self-paced free technology training to
  MCC students
• Preliminary discussion of MCC policies
• Forwarded to Academic Leadership Council

50
Next Steps

• Conduct further analysis to investigate the
  relationship between online student reasons for
  non-success, and satisfaction variables for
  successful online students
• Investigate students technology readiness
• Continue evaluation of services and activities
  that were implemented as a result of the
  retention study

51
Summary and Questions

• Questions on retention survey 1
• Time for a break!

52
IV. MCC Online Student Retention Study 2

• An Analysis of SLN Students Performance and
  Differentiation
• Office of Institutional ResearchMonroe Community
  CollegeAngel Andreu

53
The Study Research

• The Problem/Research Question
• Observations
• The Data
• Analysis
• Conclusions and Policy Implications

54
The Problem

• It is observed that the failure and withdrawal
  (F/W) grades for online students are increasing
  in number and percentage.

55
The Observation

• Percentage of F/W Grades, Fall Semester
• 1998 1999 2000 2001 2002
• Online 19 25 26 30 30
• College 17 20 21 21 21
• N in 1998 was 409 2002 was 2,473

56
The Research

• Previous research identified characteristics of
  students at risk of F/W grades in online courses.
  These were
• First-time online students
• Full-time students with less than 30 earned
  credits
• Students under the age of 25
• Minority students

57
The Research

• The MCC Andreu study affirmed these
  characteristics and looked specifically at
• Time of registration
• Basic academic skills

58
The Data Time of Registration

• Only 7 of the C or better students registered
  during the first week of classes, compared to 16
  of the F/W students
• Students who registered during the first week of
  classes had a 50/50 chance of earning a grade of
  C or better
• The odds of getting C or better decreased by 13
  for each week closer to the first week of classes

59
The Data Age

• 65 of the F/W grades were among students under
  the age of 25
• Between the ages of 21 and 22, a student had a
  50/50 chance of earning a C or better

60
The Data Age
61
The DataAge and Time of Registration

• Age and time of registration are each significant
  contributors to grade performance
• Their interaction is not significant
• The best chance of getting a C or better
• Register 5 weeks before the start of the semester
• Be age 25 or above

62
The Data Experience(i.e. number of accumulated
credit hours)

• The rate of F/W grades between first and second
  year students is significant
• Experience is not significant when combined with
  age or time of registration

63
The Data Part/Full Time Status

• Part/Full time status is not significant
• The interaction of status and age was significant

64
The DataPart/Full Time Status
65
The DataBasic Academic Skills

• MCCs ACCUPLACER tool
• The variables used included reading and sentence
  scores
• The metrics are placement scores at the college
  English level (78 for reading, 83 for sentences)

66
The DataBasic Academic Skills

• For students to have a 50/50 chance of earning a
  C or better in an online course, their basic
  academic skills should be at the college English
  level

67
What Does the Analysis Show?

• Factors important to success in online courses
• Time of registration
• The age of the student
• The academic preparedness of the student

68
What Does the Analysis Show?

• Factors in the decision model
• College is enrollment driven
• The online program is supporting enrollment
  growth
• The demographics of enrollment growth for the
  College are the same demographics of non-success
  in online courses

69
What Does the Analysis Show?

• Policy Factors
• Should policy account for motivation
• At what odds of failure do we restrict
  enrollment
• Should online growth be allowed to continue to
  support college enrollment growth
• What is the balance between student success and
  online enrollment growth

70
Policy Development 1

• In all cases, a student with an ACCUPLACER score
  below 78 (reading) and/or 83 (sentence) is
  prohibited from taking an online course at MCC.
  
• This prohibition is lifted at such time that the
  student successfully completes the Transitional
  Studies coursework appropriate to address the
  deficiency.
• All online sections will be closed to
  registration one week before the start of class.
  Qualified students (see definition on next slide)
  who wish to enroll after this date will be
  allowed to green slip in until the Colleges
  Add deadline.

71
Policy Development 1

• Qualified students include mature students with a
  record of success in college-level coursework.
  Student must also have satisfactory ACCUPLACER
  scores to have successfully completed the
  Transitional Studies coursework appropriate to
  address any deficiency. Students who have
  successfully completed an online course
  previously are also considered qualified. Note
  that data indicate that students 22 years old or
  older have a significantly greater chance of
  success in online coursework.

72
Policy Development 2

• Considerations
• Forget policy, make it a course prerequisite
• Must be programmable
• Cannot rely upon faculty involvement for
  exceptions due to availability
• Does resolution of the online problem become a
  standard

73
Policy Development 3

• Draft X
• Students in remedial English courses are
  prohibited from enrolling in online courses
• All registrations for online courses are closed
  on the first day of classes

74
Policy Development 4

• Current Policy
• Registration for online courses closed week
  before the first day of classes
• Late entry by permission

75
Summary and Questions

• Questions on retention survey 2
• Time for a break!

76
V. Research to Inform PracticeAn Online
Research Agenda

• MCCs Online Research Agenda Level 1
  Descriptive
• Basic demographics
• Basic production
• Basic enrollment
• Basic finance

77
An Online Research Agenda

• MCCs Online Research Agenda Level 2
  Evaluative
• Basic performance
• Student and faculty satisfaction
• Integration with campus IR
• Modify existing surveys to fit

78
An Online Research Agenda

• MCCs Online Research Agenda Level 3
  Comparative
• Statewide benchmarks
• National benchmarks
• Complete surveys-be a case study
• MCCs Online Research Agenda Level 4
  Theoretical
• External in the literature
• Internal on the campus

79
Research to Inform PracticeSUNY
ResearchReasons for Taking an OL Course

• Reasons for Taking Course Online
• Conflict with Personal Schedule 49
• Family Responsibilities 23
• Distance or Transportation 10
• Other 10
• Course Not Offered on Campus 5
• Interest in Technology/Internet 3
  Retained Students

80
SUNY ResearchReasons for Taking an OL Course
MCC vs. all other SUNY Students

• Primary Reason MCC SUNY
• Personal Schedule Conflict 49 31
• Family Responsibilities 23 16
• Distance or Transportation 10 20
• Other 10 12
• Not Offered On Campus 5 15
• Interactive Technology 3 7
• Retained Students

81
Research to Inform PolicyTheory Development

• Four major barriers to persistence in distance
  learning (Garland, 1993)
• Situational students general environment
  (social, economic, family)
• Institutional colleges programs, policies, and
  procedures
• Dispositional students personal background
• Epistemological academic environment (course,
  pedagogy, prerequisites)

82
Theory Development 2

1. Classified the 22 reasons for non-persistence
   according to Garlands four barriers
2. Conducted Principle Components Factor Analysis
3. Analysis and Definition of the Factors

83
Research to Inform PracticeOn-campus Orientation

• Question
• Is there value to an on-campus orientation
  course for online performance?

84
On-campus Orientation

• On-Campus Orientation Results
• Most students felt they had the necessary skills
  to be successful
• Skill preparation is not related to performance
• Students who attended the on-campus program were
  more likely to earn a C or better
• C/Better F/W
• On-campus Orientation 78 18
• Online Orientation only 64 30

85
On-campus Orientation

• To Continue the Course
• Practice Demonstration Course
• Structure not Skills
• Market the grade differential
• Keep faculty involved in the program to provide
  authentic experience

86
Research to Inform PracticeStudent Advice

• Question to online students who received F/W
  grade on their online course
• What advice would you give a fellow student
  considering registering for an online course?

87
Student Advice

• Results Top Five Topics
• Be ready for independent learning. Must be
  self-motivated.
• Once you get started, stay on track.
• Online courses require lots of reading, homework
  and research. Get ready for the demands.
• Must develop a schedule to go online, must manage
  your time.
• Talk to other people and to other students before
  you take the class, and during the class

88
Student Advice

• Practical Use of Online Student Advice
• Integrate into brochures, orientation seminars
• Do mock testimonies for video displays
• Share with Counseling and Advising staff
• Share with Faculty

89
Research to Inform PracticeManaging Services

• Manage the Expectations
• Additional pre-start information sharing
• Additional orientations offered
• Manage the Support Services
• Greater integration with support team
• Additional points of access to services
• Manage the Academics
• Faculty course design issues and training
• Emphasis on multiple interactions

90
Managing Services

• Manage the Expectations
• Welcome letter from Academic Vice President has
  been expanded
• Face-to-face student orientations offered
• MCC Online Orientation CD distributed
• Phone calls made to all Fall 2002 online students
  and 1st-time Spring 2003 online
  studentsadditional analysis conducted

91
Managing Services

• Manage the Support Services
• Student Services web page expansion
• Revisions made to the service gateway to online
  learning web page
• Brochures at Records and Registration and at the
  Counseling Center

92
Research to Inform PracticeManaging Services

• Manage the Academics
• Retention strategies integrated into faculty
  trainings
• Discipline-specific retention studies
• Discussions on other academic supportonline
  tutoring, writing centers, etc.
• Pilot test CourseSpace (use of online template
  for web course enhancement)

93
Research to Inform PracticeDefining Retention

• Question How do you define retention?
• It depends
• Researchtwo basic presentations
• As an enrollment statistic
• As a performance indicator
• Policy
• Be consistent, clarify your statistics

94
Defining Retention and Withdrawal Rates
Retention as an enrollment statistic
All SLN Courses Fall 2002 Annual 2002-03
First day enrollment 1851 4154
First week enrollment 1857 4225
Census day enrollment 1714 3847
Last day enrollment 1716 3832
95
Online Retention Rates
Fall 2002 1st Week Census Day Last Day
1st Day 1.02 .94 .95
1st Week .92 .92
Census 1.00
All 2002-03 1st Week Census Day Last Day
1st Day 1.02 .93 .92
1st Week .91 .91
Census 1.00
96
Defining Retention and Withdrawal Rates
Retention as a performance measure (All SLN Courses)
Grade Fall 2002 Annual 2002-2003
A to D 1,253 2,755
F, W, Other 463 1,077
97
Withdrawal Rates
Fall 2002 FW Grades Spring 2003 FW Grades 2002-2003 FW Grades
All MCC 20.5 19.6 19.5
Online Only 30.0 29.5 28.1
98
Next StepsThings to Consider

• Services
• Policies
• Definition of Retention for data collection
• Replicate studies at other institutions

99
Summary

• Conclusions
• Questions
• Discussion
• Completion of Session Evaluation Forms

100
Contact Information

• Dr. Jeffrey P. Bartkovichjbartkovich_at_monroecc
  .edu
• Marie J. Fetznermfetzner_at_monroecc.edu
• MCC Web Page www.monroecc.edu
• MCC Online Learning pagehttp//www.monroecc.edu/
  depts/distlearn/index.htm