Evaluating your e-learning course

1
Evaluating your e-learning course

• LTU Workshop11 March 2008

2
Common practice

• End-of-course student questionnaire
• Standard version with last-minute adaptations
• changes to how the course elements are named
• including some additional items
• Usually with similar problems
• low return rates
• no follow up returns, and sometimes do not
  analyse results
• poor quality responses
• limited enhancement function

3
Research shows

• Evaluation of e-learning courses mainly affected
  by
• distributed configuration of teams and
• distance of students
• Strategies for evaluating courses are problematic
  even at compliance level
• Sense of evolution looking for the right
  strategy
• Teams worry about response rates but dont use
  data available
• Undefined responsibilities
• Staffs lack of skills, confidence motivation

4
Common problems

• Evaluation is not planned (in advance) as part of
  the overall course design
• Feedback is gathered only from students
• Feedback is gathered only at the end of the
  course/module
• Feedback is gathered only through traditional
  methods
• Feedback requested do not consider all e-learning
  aspects

5

• Recommendations Strategies

6
Collect feedback from tutors, administrators and
technical support

• Usual context team meetings affected by
  distribution of staff (part time or at a
  distance).
• Swinglehurst (2006) approach tips for
  e-learning course teams
• Formally organise frequent staff meetings, where
  tutors, administrators and support staff meet to
  analyse course. These meetings could be
  face-to-face and/or online.
• Define an agenda for each meeting, and rotate the
  member of staff responsible for this agenda, to
  cover key issues related to the different aspects
  of the course (e.g. student support, academic
  feedback, encouraging discussions)
• Limit the recording of the discussion at these
  meetings to a minimum, briefly documenting the
  topics covered and the decisions made, and
  disseminate them.
• Questionnaire to staff

7
Collect feedback during the run of the course

• Daly et al. (2006) approach - offers a simple and
  effective mechanism for the collection of student
  feedback during the run of a course
• embedding evaluation activities, encouraging
  students to think about how the course
  design/materials/activities have support their
  learning.
• Main benefits to identify difficulties when they
  students are experiencing them, and the
  opportunity to explore students experience.
• Online learning diaries - running throughout the
  course, in which students are encouraged through
  brief questions to post their thoughts regarding
  their learning process and how the course has
  supported them.

8
Collect feedback at the end of the course/module

• Questionnaires
• Questionnaire/question features language,
  questions types
• Mode of application
• Online (using an online survey tool)
• By e-mail
• Paper-based
• Timing at which feedback is collected
• Responsibility for collection and analysis
• Focus groups

9
Use alternative strategies

• Computer logs that the particular e-learning
  system in use provides (e.g. Course Statistics in
  the Blackboard VLE).
• Monitoring basic statistics such as last login
  date, number of messages sent by users, areas of
  content and discussion boards/forums visited by
  users.
• Statistics are not an indications of the quality
  of the student/tutor participation or of a
  satisfactory online experience but useful tool
  for monitoring online presence, to obtain an
  overall picture of the ongoing activity, as well
  as to detect problems

10
Include aspects relevant to the use of technology

• Wide range of aspects to include, and this list
  would vary depending on context and on the
  objectives and audience for the evaluation
• Main issues
• quality, usefulness and frequency of use of
  course components (online activities, resources,
  face-to-face events, readings, online
  discussions/seminars, tutor support, technical
  support, etc.)
• how well online activities run (timing,
  frequency, sequence, instructions, interactions,
  feedback, time on task, etc.)
• e-learning experience (workload, involvement,
  online participation facilitators and
  restrictions, etc.)
• role of tutors (engagement, feedback, support,
  etc.)
• Warwick University and Bristol Universitys
  guidelines IHEPs Quality on the Line report
  (Phipps and Merisotis, 2000)UCLs Quality
  Framework (Greenhalgh, 2001)