A Case Study of Usability Testing on a Webbased Course

1
A Case Study of Usability Testing on a Web-based
Course

• Mingzhuo Liu
• East China Normal Univeristy

2
Items

• Theoretical considerations
• Goals
• Methods
• Results
• Tasks
• Questionnaire
• Questions

3
Theoretical Considerations
4
Theoretical Considerations

• Usability is an important indicator for the
  quality of interactive IT products. According to
  ISO 9241-11 (1998) usability is the extent to
  which a product can be used by specified users to
  achieve specified goals with effectiveness,
  efficiency and satisfaction. Francis() described
  usability as five dimensions (5Es)-effective,
  efficient, engaging, error tolerant and easy to
  learn.

5
5E Model

• Effective How completely and accurately the work
  or experience is completed or goals reached
• Efficient How quickly this work can be
  completed
• Engaging How well the interface draws the user
  into the interaction and how pleasant and
  satisfying it is to use
• Error Tolerant How well the product prevents
  errors and can help the user recover from
  mistakes that do occur
• Easy to Learn How well the product supports both
  the initial orientation and continued learning
  throughout the complete lifetime of use.

6
5 dimensions for the usability testing

• Effective Watch for the results of each task,
  and see how often they are done accurately and
  completely. Look for problems like information
  that is skipped or mistakes that are made by
  several users.
• Efficient Time users as they work to see how
  long each task takes to complete. Look for places
  where the screen layout or navigation make the
  work harder than it needs to be.
• Engaging Watch for signs that the screens are
  confusing, or difficult to read. Look for places
  where the interface fails to draw the users into
  their tasks. Ask questions after the test to see
  how well they liked the product and listen for
  things that kept them from being satisfied with
  the experience
• Error Tolerant Create a test in which mistakes
  are likely to happen, and see how well users can
  recover from problems and how helpful the product
  is. Count the number of times users see error
  messages and how they could be prevented.
• Easy to Learn Control how much instruction is
  given to the test participants, or ask
  experienced users to try especially difficult,
  complex or rarely-used tasks. Look for places
  where the on-screen text or work flow helpsor
  confuses them.

7
Survey Instruments

• According to Riihiaho (2000) usability evaluation
  can be divided into two broad categories user
  testing and usability inspection.
• Several of these survey instruments were
  suggested by IBM (Lewis, 1995)
• PSSUQ- the Post-Study System Usability
  Questionnaire
• CSUQ the Computer System Usability Questionnaire
• ASQ the After-Scenario Questionnaire
• QUIS the Questionnaire for User Interface
  Satisfaction
• SUS the system Usability Scale
• From http//www.usability.serco.com/trump/documen
  ts/Suschapt.doc) or the Website Analysis and
  Measurement Inventory (http//www.ucc.ie/hfrg/
  Questionnaires/wammi/).

8
Possible factors affecting learners learning
experience

• Unclear navigation
• Bad information architecture
• Unsteady interface
• Media is monotonous
• Colors mismatching
• Learning path and position unclear
• Cant see the details about the graph or video
• Audio is unclear
• Too many pages of text
• Learning activities pay much attention on
  delivering rather that interaction
• Be short of necessary learning support
• Accessing speed is slow
• Cant play automatically because of plugins

Those factors show that the most important is
learners experience. But how can we know the
learners experience in an online course? Thats
usability testing. We will have to go back to
learner-centered design.
9
(No Transcript)
10
(No Transcript)
11
Introduction to the tested course
The name of the course is Introduction to
Psychology. http//ecourse1.ecnudec.com Username
test password test
12
Architecture of the course
13
The design of the course
14
Five-step learning for each module

• Warm-up story-based description, such as flash,
  comic. From the scenarios description, the
  subject can be raised so as to excite the
  learners learning interest.
• Learning Guide This part includes learning goal,
  learning suggestion, those important or difficult
  learning points and key words of this present
  module. From the learning guide, the students
  will have a brief view about this module and they
  will have an idea about how to learn this module.
• Lectures The same content will be taught
  (displayed) in three styles text, audio and
  video so that the learners can choose which way
  they like according to their personal learning
  orientation.
• Practice/interaction activities There are kinds
  of activities prepared for the students, such as
  forums, chats, quizzes, assignments,
  supplementary resources, cases and so on. The
  students are asked to take part in or complete.
• Tests In this course, it provides two kinds of
  tests - standard test and performance evaluation.
  In this performance evaluation, the teacher gives
  open questions and some answering clues to the
  students. The students should submit their test
  paper online, the teacher will mark it online and
  also other students should comment on it. For
  those outstanding works, the teacher will share
  them to all of the students.

15
The First Page
16
The Content Page
17
When is the right time?
e-Learning Product Life Cycle
18
Usability testing to the prototype of this course
(initial design stage)
19
Purpose

• The main purpose of this study is to know if the
  design of this course is reasonable for the
  learners or welcomed by the learners. This study
  will try to answer the following questions
• v Could the participants easily find the content
  that they want to learn? That is, could the
  participants complete their tasks successfully?
• v Are those newly added activities and learning
  tools meaningful or welcomed by the participants?
• v Is the course attractive to the participants?
  Do the participants like the course, such as its
  interface design, color and the way that the
  teachers teach?

20
Methods

• Interview Test participants were interviewed
  about their expectations to the web-based courses
  before they saw them.
• Solving test tasks Test participants were asked
  to carry out tasks using the web-based courses.
  These tasks are included as appendix A. This
  approach ensured that test participants were
  solving realistic tasks for which they were
  highly motivated. Test participants were asked to
  think aloud and to comment on the website while
  they were carrying out their tasks.
• Debriefing Test participants were asked to fill
  in a questionnaire while thinking aloud. We were
  particularly interested in the verbal comments
  they made while filling in the questionnaire. In
  addition, test participants were debriefed about
  their general impression of the website. The
  questionnaire and the debriefing questions are
  included in the usability test script in appendix
  A. The questionnaire was modified based on the
  international survey instrument PSSUQ (usability
  and user satisfaction questionnaire).

21
Data Collection
22
Participants

• 6 students from Psychology Department, which
  includes Bachelor and Graduate students.

23
Usability testing
In the ECNU-MU Joint Lab of Usability
Engineering, each participant was given an
instruction sheet with fourteen tasks (see
APPENDIX A). Among those tasks, task 6 (insert a
bookmark to mark your learning process), task 8
(participate in subject-based debating in chapter
1), task 9 (go to the case analysis in chapter
1),task 10 (participate in the brainstorm in
chapter 1), task 12 (check your own learning
calendar and learning process), task 13 (find out
the related content to consciousness in key
words database), task 14 (using note-taking tool
provided by the course to comment on this course)
focused on high interactive learning activities
and tools, and the remaining tasks focused on
course content and its multimedia expression,
such as task 3 (about warm-up), task 4 (about
teaching materials), task 7 ( about the guide to
the present module) and task 11 (about
supplemental materials). Participants were
encouraged to complete the tasks independently.
At the end of each task, participants were
encouraged to describe their impressions of the
task. The whole testing session was recorded and
analyzed with Morae software (Morae recorder and
manager made by TechSmith).
24
Performance Analysis
To understand the usability of this web-based
course, in the light of the efficiency and
effectiveness, we focused on the completion score
and time on tasks. Data was analyzed via Morae
manager. We marked each task with a start time,
end time, and task completion rate (0--completion
with ease 1--completion with difficult
2--completion failure), based on our observation
via video recordings by Morae software. The
"completion with difficult" rating refers to
unnecessary steps used to complete the task.
25
User perception (1)
To understand satisfaction perceptions,
participants were required to describe their
opinions for certain functions. After they
completed each task, participants commented about
their impressions of this course (See APPENDIX
A),and some tasks(task3, task6, task7, task8 and
task12) asked the participants to give a general
mark(1-5). We also designed a questionnaire (see
APPENDIX B) with 28 questions from navigation,
study support, and satisfaction aspects. The
ratings used a likert scale that was defined from
1 to 5 (1 completely disagree, 2 disagree, 3
neutral, 4 agree, 5 completely agree).
Participants completed the questionnaire in
approximately  15-20 minutes.
26
Findings
27
Time on tasks
28
Results

• The mean completion time for 14 tasks was a range
  from 23.92 seconds to 144.23.
• The tasks with the most time were task2 (enter
  chapter 3 from the first webpage) 144.23 seconds,
  task10 (randomly enter a brainstorm and free
  talk module) 107.57 seconds and task12 (go to
  the learning calendar and learning process)
  86.85.
• The tasks with the least amount of time were
  task5, task9 and task 7.

29
Completion score
30
User satisfaction perceptions

• Two parts from questionnaire and from the
  comments while they completing tasks. Three
  perspectives
• Is the content, teaching design or navigation
  feasible (task2, 3,4,5,7)?
• Is learning activities feasible (task8,9,10,11)?
• Is the learning support useful (task6,12,13,14)?

31
User perception(2)
32
User perception (3)
33

• Most welcomed parts learning guide and case
  analysis.
• Problems with task 6 and task 8.

34
Results to the Questionnaire
35
Related documents

• Testing Tasks
• Questionnaire
• Reports

36
thank you