A Distributed Learning Impact Evaluation: What UCF Knows About Its Online Students, Faculty, Successes, and Challenges

1
A Distributed Learning Impact EvaluationWhat
UCF Knows About Its Online Students, Faculty,
Successes, and Challenges

• Charles D. Dziuban
• Patsy D. Moskal
• University of Central Florida

2
The University of Central Florida
3
UCF terminology for courses utilizing web
instruction

• Web courses delivered entirely over the Web,
  with no regular class meetings
• Mixed-mode courses some face-to-face
  instruction is replaced with web instruction so
  that on-campus time is reduced
• Enhanced courses delivered entirely in
  face-to-face mode, but with web enhancements

4
UCF Enrollment by Web modality
Enrollment
Spring 2000
Fall 2002
Semester
5
Scope of UCFs Online Program

• Half (600) of all full-time faculty engaged
• Half (300) of these trained through IDL6543
• 1,500 courses in WebCT
• 900 courses built with production support
• Four online bachelors degree completion programs
• Five online masters degree programs
• Four online graduate certificate programs

6
The Evaluation
7
Principles that guide our evaluation

• Evaluation must be objective.
• Evaluation should conform to the culture of the
  institution.
• Uncollected data cannot be analyzed.
• Data do not equal information.
• Qualitative and quantitative approaches must
  complement each other.
• We must show an institutional impact.
• Our results may not be generalized beyond UCF.

8
Distributed Learning Impact Evaluation components
Faculty
Online programs
Real-time surveys
Writing project model
Modified instructional theories
Student evaluation of instruction
Large online classes
9
Student Results
10
Student satisfaction in fully online and
mixed-mode courses
44
Fully online (N 1,526)
41
39
38
Mixed-mode (N 485)
11
9
9
5
3
1
Very Satisfied
Neutral
Very Unsatisfied

Unsatisfied
Satisfied
11
Students likelihood of taking another web course
Fully online (N 1,526)
49
47
Mixed-mode (N 485)
29
23
16
11
10
8
6
2
Definitely
Probably Not
Probably
Not Sure
Definitely Not

12
What students like most about W courses
41
29
18
Flexibility
Convenience
Improved Logistics
13
What students like least about W courses
16
16
11
11
5

Reduced or no F2F time
Lack of instructor availability
Facilitates procrastination
Technical problems
Excessive requirements

14
Success rates of modalities excluding computer
science -- Spring 00 through Summer 01
F2F
Total N 57,950 students
E
M
W
95
95
94
93
92
91
90
90
89
89
89
88
88
88
86
86
85
84
84
82
15
An example of a fractured data set
Course Type
Course Prefix W M E F2F POS2041 EDF
6481 NUR6465 SYG2010
16
A segment model for success
Overall
85.9 n11,286
Arts Sciences, Business Admin., Hospitality
Mgmt.
Health Pub. Affairs
Education
Engineering
85.8 n6,460
72.7 n378
91.5 n2,079
86.7 n2,369
F2F, E, M
W
F2F
E, M, W
E, M
F2F
94.1 n1,036
89.1 n1,043
64.7 n148
79.6 n230
86.5 n5,639
74.8 n821
females
males
AS
BA Hosp. mgmt
88.4 n3,263
84.1 n2,376
78.5 n526
68.9 n298
17
Student Behavior Types
18
Research on reactive behavior patterns

• Theory of William A. Long, University of
  Mississippi
• Ambivalence brings out behavior patterns
• Provides a lens for how types react to
  different teaching styles

19
Resources

• Personality
• Emotional maturity
• Sophistication level
• Level of intellect
• Educational level
• Character development

20
A description of Long behavior types

• Aggressive Independent
• high energy
• action-oriented
• not concerned with approval
• speaks out freely
• gets into confrontational situations
• Passive Independent
• low energy
• not concerned with approval
• prefers to work alone
• resists pressure from authority

• Aggressive Dependent
• high energy
• action-oriented
• concerned with approval
• rarely expresses negative feelings
• performs at or above ability
• Passive Dependent
• low energy
• concerned with approval
• highly sensitive to the feelings of others
• very compliant

21
A description of Long behavior traits

• Phobic
• exaggerated fears of things
• often feels anxious
• often sees the negative side
• doesnt take risks
• Compulsive
• highly organized
• neat, methodical worker
• perfectionist
• strongly motivated to finish tasks

• Impulsive
• explosive
• quick-tempered
• acts without thinking
• frank
• short attention span
• Hysteric
• dramatic and emotional
• more social than academic
• artistic or creative
• tends to overreact

22
Distribution of Long types and traits for fully
online students
75
PD 7
51
AI 21
AD 54
30
26
PI 18
(N1,437)
(N1,520)
23
Distribution of Long types and traits for
mixed-mode students
76
PD 8
54
AI 17
AD 52
32
23
PI 23
(N472)

24
Distribution of Long types and traits for
Composition I students
PD 14
53
50
40
38
AI 20
AD 44
PI 23
(N1,054)

25
Long types and traits for Web, mixed-mode, and
general education students
Web (N1,533) Mixed-mode (N491) Comp I (N1,054)
Aggressive Dependent 54 52 44
Passive Dependent 7 8 14
Compulsive 74 76 53
Impulsive 26 23 38
Types
Traits
26
Faculty Results
27
Time to develop course as compared with a
comparable face-to-face section
A lot more time
More work
A little more time
52
77
About the same
A little less time
A lot less time
43
21
Equal to or less than
5
2
W n56
M N43
Modality
28
Time in weekly course administration activities
as compared with a comparable face-to-face section
A lot more time
43
More work
A little more time
60
About the same
A little less time
38
20
A lot less time
15
19
Equal to or less than
2
4
W n55
M N42
Modality
29
Time in weekly course delivery activities as
compared with a comparable face-to-face section
A lot more time
A little more time
About the same
A little less time
More work
9
5
A lot less time
15
13
28
37
Equal to or less than
30
29
20
15
W n55
M N42
Modality
30
Amount of interaction in Web classes compared to
comparable F2F sections
Increased
Somewhat increased
45
More interaction
62
About the same
Somewhat decreased
30
Decreased
16
Equal to or less than
13
15
8
2
3
7
W n55
M N40
Modality
31
Quality of interaction in Web classes compared to
comparable F2F sections
Increased
Somewhat increased
About the same
30
35
Better interaction
Somewhat decreased
Decreased
37
33
Equal to or less than
22
19
9
14
2
W n55
M N43
Modality
32
Faculty satisfaction compared with a comparable
face-to-face section
38
Very satisfied
44
49
Satisfied
Positive
Neutral
58
Unsatisfied
44
38
Very unsatisfied
Neutral or negative
7
6
5
5
7
W n55
M N43
F2F N64
Modality
33
Faculty willingness to teach Web courses in the
future
Definitely
Probably
Probably not
Definitely not
Positive
81
69
16
13
Neutral or negative
2
10
4
6
W n71
M N53
Modality
34
Relationships of faculty satisfaction with class
interaction and workload (TAU-b)
W M (n53) (n38) Amount of
interaction .39 .34 Quality of
interaction .43 .51 Time to develop .16
.09 Time to administer .10 .01 Time to
deliver .06 .10
plt.05 plt.01
35
What faculty like most about teaching on the web
26
26
19
11
Convenience of location
Flexibility of time
Interaction with students
Enjoy technology
36
What faculty like least about teaching on the web
35
18
14
Lack of face-to-face contact
Technical problems
Requires too much time
37
Student Ratings
38
Student Evaluation of Instruction

• Same form used for all classes -- F2F, and Web
• Some concern among Web faculty that standard
  form not appropriate for asynchronous courses
• Concern among faculty that Web faculty received
  lower evaluations than F2F

39
Student Ratings by Modality
Very Modality Excellent Good Good Fair Poor
F2F 42.00 29.50 19.00 7.20 2.40 (N628,623) E 44
.00 29.10 17.40 6.90 2.60 (N6,632) M 40.60 28.6
0 20.60 7.70 2.40 (N11,450) W 55.40 25.20 12.10
4.90 2.50 (N5,435)
40
Research Initiative for Teaching Effectiveness

• For more information contact
• Dr. Chuck Dziuban
• [phone]
• dziuban_at_mail.ucf.edu
• Dr. Patsy Moskal
• [phone]
• pdmoskal_at_mail.ucf.edu
• http//pegasus.cc.ucf.edu/rite

41
Rationale for the Study

• Teaching evaluation data for a 3-year period were
  available to allow a comparison of two different
  sets of items (UCF/BOR). 
• Responding to faculty interest, the UCF Faculty
  Senate requested that an evaluation of the
  Student Evaluation of Instruction measures be
  performed. (FS 1995-96-11)

42
The Study Layout

• Approximately 450,000 student responses
• Five Colleges
• Arts and Sciences
• Business Administration
• Education
• Engineering
• Health and Public Affairs

• Three Levels
• Lower Undergraduate
• Upper Undergraduate
• Graduate

• Three Years
• 1996-97
• 1997-98
• 1998-99

43
Rules Leading to Overall Rating of Excellent
Questions Ex VG G F P
Facilitation of Learning ?
Communication of Ideas and Information ?
Probability .96
N46,805
44
Rules Leading to Overall Rating of Excellent
Questions Ex VG G F P
Facilitation of Learning ?
Communication of Ideas and Information ?
Organization of the Course ?
Assessment of Student Progress ? ?
Probability .85
N3,462
45
Rules Leading to Overall Rating of Excellent
Questions Ex VG G F P
Facilitation of Learning ?
Communication of Ideas and Information ? ?
Organization of the Course ?
Instructor Interested in Your Learning ?
Probability .78
N6,215
46
Overall Excellent Ratings by CollegeUnadjusted
and Adjusted for Rules
Adjusted for Rule Adjusted for Rule Adjusted for Rule
College Unadjusted 1 2 3
Arts Sciences 42 95 85 80
Business 36 95 83 77
Education 54 97 86 75
Eng. Comp Sci. 32 96 84 78
Health Public Affairs 48 96 86 74