Assessing TechnologyEnhanced Learning

1
Assessing Technology-Enhanced Learning

• Charles D. Dziuban
• Patsy D. Moskal
• University of Central Florida

2
The University of Central Florida
3
UCF terminology for courses utilizing web
instruction

• Web Courses delivered entirely over the Web,
  with no regular class meetings
• Mixed-mode Courses some face-to-face
  instruction is replaced with web instruction so
  that on-campus time is reduced
• Enhanced Courses delivered entirely in
  face-to-face mode, but with web enhancements

4
Principles that guide our evaluation

• Evaluation must be objective.
• Evaluation should conform to the culture of the
  institution.
• Uncollected data cannot be analyzed.
• Data do not equal information.
• Qualitative and quantitative approaches must
  complement each other.
• We must show an institutional impact.
• Our results may not be generalized beyond UCF.

5
Distributed Learning Impact Evaluation
Students
Faculty
6
Student Results
7
Success rates by modalitySpring 01 through
Spring 03
F2F
M
Total N 139,444 students
W
Percent
8
Success rates by modality for Health Public
Affairs
F2F
M
Total N 26,073 students
W
Percent
9
Success rates by modality for Arts Sciences
F2F
M
Total N 49,460 students
W
Percent
10
Success rates by modality for Education
F2F
M
Total N 10,822 students
W
Percent
11
A segment model for success
Overall
85.9 n11,286
Arts Sciences, Business Admin., Hospitality
Mgmt.
Health Pub. Affairs
Education
Engineering
85.8 n6,460
72.7 n378
91.5 n2,079
86.7 n2,369
F2F, E, M
W
F2F
E, M, W
E, M
F2F
94.1 n1,036
89.1 n1,043
64.7 n148
79.6 n230
86.5 n5,639
74.8 n821
females
males
AS
BA Hosp. mgmt
88.4 n3,263
84.1 n2,376
78.5 n526
68.9 n298
12
Student satisfaction in fully online and
mixed-mode courses
44
Fully online (N 1,526)
41
39
38
Mixed-mode (N 485)
11
9
9
5
3
1
Very Satisfied
Neutral
Very Unsatisfied

Unsatisfied
Satisfied
13
Students positive perceptions about blended
learning

• Convenience
• Reduced Logistic Demands
• Increased Learning Flexibility
• Technology Enhanced Learning

Reduced Opportunity Costs for Education
14
Students less positive perceptions about blended
learning

• Reduced Face-to-Face Time
• Technology Problems
• Reduced Instructor Assistance
• Overwhelming
• Increased Workload

Increased Opportunity Costs for Education
15
Some characteristics of the generations

• Matures (prior to 1946)
• Dedicated to a job they take on
• Respectful of authority
• Place duty before pleasure
• Baby boomers (1946-1964)
• Live to work
• Generally optimistic
• Influence on policy products

• Generation X (1965-1980)
• Work to live
• Clear consistent expectations
• Value contributing to the whole
• Millennials (1981-1994)
• Live in the moment
• Expect immediacy of technology
• Earn money for immediate consumption

16
Students who were very satisfied by generation
55
38
26
Percent
Boomer n328
Generation-X n815
Millennial n346
17
Better able to integrate technology into their
learning
67
48
34
Percent
Boomer n328
Generation-X n815
Millennial n346
18
Because of the web I changed my approach to
learning
51
37
Percent
23
Boomer n328
Generation-X n815
Millennial n346
19
College Level Academic Skills Test (CLAST)
English scores
953
782
548
n 1,268
n 8,861
n 6,164
20
Faculty Results
21
Time to develop course as compared with a
comparable face-to-face section
A lot more time
More work
A little more time
52
77
About the same
A little less time
A lot less time
43
21
Equal to or less than
5
2
W n56
M N43
Modality
22
Quality of interaction in Web classes compared to
comparable F2F sections
Increased
Somewhat increased
About the same
30
35
Better interaction
Somewhat decreased
Decreased
37
33
Equal to or less than
22
19
9
14
2
W n55
M N43
Modality
23
Faculty willingness to teach Web courses in the
future
Definitely
Probably
Probably not
Definitely not
Positive
81
69
16
13
Neutral or negative
2
10
4
6
W n71
M N53
Modality
24
Student Behavior Types
25
Research on reactive behavior patterns

• Theory of William A. Long, University of
  Mississippi
• Ambivalence brings out behavior patterns
• Provides a lens for how types react to
  different teaching styles

26
Resources

• Personality
• Emotional maturity
• Sophistication level
• Level of intellect
• Educational level
• Character development

27
A description of Long behavior types

• Aggressive Independent
• high energy
• action-oriented
• not concerned with approval
• speaks out freely
• gets into confrontational situations
• Passive Independent
• low energy
• not concerned with approval
• prefers to work alone
• resists pressure from authority

• Aggressive Dependent
• high energy
• action-oriented
• concerned with approval
• rarely expresses negative feelings
• performs at or above ability
• Passive Dependent
• low energy
• concerned with approval
• highly sensitive to the feelings of others
• very compliant

28
A description of Long behavior traits

• Phobic
• exaggerated fears of things
• often feels anxious
• often sees the negative side
• doesnt take risks
• Compulsive
• highly organized
• neat, methodical worker
• perfectionist
• strongly motivated to finish tasks

• Impulsive
• explosive
• quick-tempered
• acts without thinking
• frank
• short attention span
• Hysteric
• dramatic and emotional
• more social than academic
• artistic or creative
• tends to overreact

29
Student Ratings
30
A decision rule based on student evaluation
responses and the probability of faculty
receiving an overall rating of Excellent
If...
Excellent Very Good Good Fair
Poor
Facilitation of learning
Communication of ideas
Then...
The probability of an overall rating of Excellent
.93 The probability of an overall rating
of Fair or Poor .00
31
A comparison of excellent ratings by college
unadjusted and adjusted for instructors
satisfying Rule 1
College Unadjusted Adjusted Arts
Sciences 41.6 92.4 Business 34.9 90.9 Educati
on 56.8 94.8 Engineering 36.2 91.3 HPA 46.1 9
3.9 (N441,758) (N147,544)

32
A comparison of excellent ratings by course
modality--unadjusted and adjusted for instructors
satisfying Rule 1
Course Modality Unadjusted Adjusted

F2F 42.0 92.2 E 44.0 92.3 M 40.6 92.0 W 55.4 9
2.7 ITV 20.9 86.7
N709,285 N235,745
33
Research Initiative for Teaching Effectiveness

• For more information contact
• Dr. Chuck Dziuban
• [phone]
• dziuban_at_mail.ucf.edu
• Dr. Patsy Moskal
• [phone]
• pdmoskal_at_mail.ucf.edu
• http//rite.ucf.edu