Evaluating Distance Learning: Feedback from

1
Evaluating Distance Learning Feedback from
Distressed Students

• Christopher Essex
• Kursat Cagiltay
• Indiana University
• AECT National ConventionDenver, ColoradoOctober
  27, 2000

2
Our Background in Online DE

• Essex has worked for the Indiana University SOE
  Distance Education Program since 1994
• Essex has taught online since Summer 1998
• Cagiltay taught Turkish students online from USA

3
Outline

• Definition of Distress
• Study Background
• Course and Student Profile
• Method
• Findings Positive and Negative Feedback
• Recommendations
• Limitations

4
Online Distance Education Student Distress

• Key References
• Hara, N. and R. Kling (in press). Students'
  distress with a web-based distance education
  course. Information, Communication and Society.
  (Earlier draft is on-line Available
  http//www.slis.indiana.edu/CSI/ wp99_01.html)
• Mendels, P. (Sept. 22, 1999). Study Finds
  Problems With Web Class. New York Times Online
  Edition. (On-line). Available
  http//www.nytimes.com/library/tech/99/09/cyber/ed
  ucation/22education.html
• Gale, C. (2000, January). Online learning A
  student perspective. Syllabus. pp. 52-53

5
Online Distance Education Student Distress

• Distress situations that the students...find
  particularly troublesome.
• Social Isolation
• Overwhelming Email Communication
• Lack of Instructor Feedback
• Technical Problems
• Ambiguous Instructions
• (Hara and Kling)

6
Study Background

• The Course
• Focused on the evaluation of Internet resources
  for use by K-12 educators and students
• Course activities
• Required students to navigate through various
  websites, the course website along with a number
  of others, send and respond to email messages,
  and use a web-based conferencing tool for class
  discussion purposes
• Instructors role/pedagogy
• Facilitator, guided students through PBL-based
  activities and moderated class discussion

7
Study Background

• Our role as evaluators
• The researchers conducted the study from a
  third-party perspective, observing impartially
  and not involved in making any changes to the
  program based on our evaluation.

8
Course Profile

• 100 online course, utilizing course website,
  email and asynchronous web-based conferencing
• Masters-level, three-credit hour course
• Designed and taught by advanced doctoral student
• Part of an online Masters degree program

9
Student Profile

• Of 11 registered students, 9 responded to at
  least one of various survey instruments. Seven
  students filled out all the instruments
• 100 graduate-level students
• Students were located in Indiana, Hawaii, Iowa,
  Tennessee, North Dakota, and Sweden
• 4 in on-campus degree program
• 1 in online masters degree program
• 10 enrolled for educational and business-related
  reasons--Only 1 registered for personal reasons

10
Methods

• Interviewed instructor
• Reviewed course content/activities
• Developed questions based on Kirkpatricks Four
  Levels of Evaluation
• Developed online survey instruments
• Integrated surveys into course website

11
Focus of Kirkpatrick Levels

• Level 1 (Student satisfaction)
• Course materials
• Course activities
• Instructor performance
• Overall rating for course
• Level 2 (Student learning)
• Level 3 (Transfer of learning)
• Level 4 (Cost/benefit impact)

12
Positive Student Feedback

• Level 1
• Overall, students gave positive comments
  regarding this course.
• Level 2
• The students reported a moderate level of
  learning.
• Level 3
• Students expected professional benefits in the
  future from taking the course.
• Level 4
• Students responded that the course cost them more
  money than on-campus course, but saved them time.
• The majority of the students felt that the
  cost/benefit ratio of the course was very
  favorable.

13
Negative Feedback Distress

• Level 1
• Many students were not very satisfied with their
  interactions with the instructor.
• Level 2
• A small number of students were not well-prepared
  for the technological requirements of this
  course, which caused them frustration.
• Level 4
• Students responded the course cost them more
  money than an on-campus course.

14
Negative Feedback Distress

• Levels 1, 2, 3
• One student, in a moment of distress, made the
  following statement "I am totally frustrated. I
  absolutely do not know how this class is
  organized and how to access the information I
  need. I hate Long Distance education and I never
  plan to do this ever again. It has made me
  rethink even using the Internet in my class at
  school. I hate this. I hate this. I hate this."

15
Distress Comparison to Hara and Kling

• Distress situations that the students...find
  particularly troublesome.
• Social Isolation (not found)
• Overwhelming Email Communication (not found)
• Lack of Instructor Feedback (found)
• Technical Problems (found)
• Ambiguous Instructions (found)
• (Hara and Kling)

16
Recommendations for DE Instructors and Developers

• The instructor should review her practices in
  responding to students email and web-based
  conferencing posts, to ensure that she is
  providing sufficient and appropriate feedback.
• The instructor should specify the technological
  requirements in the syllabus, and arrange for
  technical support.
• The instructor should conduct a usability tests
  on the course requirements and other instructions
  to ensure that they are clear and non-ambiguous.
• The distance education program should provide an
  on-campus face-to-face orientation. For students
  that cannot be on-campus, a video tape should be
  provided.

17
Limitations

• Small sample size (though entire class)
• Education students may be atypical
• Lack of follow-up evaluation
• Lack of ability to measure Level 3 (Transfer of
  Learning)

18
Questions?
Christopher Essex cessex_at_indiana.edu Kursat
Cagiltay kursat_at_indiana.edu Presentation Online
at http//php.indiana.edu/kursat/evalde-aect-den
ver.ppt