Usability Evaluation Methods for TechnologyEnhanced Learning

1
Usability Evaluation Methodsfor
Technology-Enhanced Learning

• ETHZ (Swiss Federal Institute of Technology
  Zurich)
• Effie Law
• JSI (Institute Jozef Stefan)
• Toma Klobucar, Matic Pipan, Tanja Arh
• Borka Jerman Blaic
• ICE (University of Iceland)
• Ebba Thora Hvannberg

2
Rationales

• Severe usability problems in interactive software
  systems, despite 20 years of usability research
• Limited understanding about the educational
  effectiveness, practical efficiency and user
  satisfaction in the usage of technology-enhanced
  learning (TEL) systems
• Low usability of TEL systems undermines their
  applicability
• Conventional usability evaluation methods (UEMs)
  needs to be adapted and augmented for emerging IT
  

3
Iterative development cycle
PERVASIVE USABILITY
Evaluation Testing
Design
Requirements
USER
USABILITY
TASK
TOOL

• Early focus on users
• Empirical measurements
• Interplay between
• evaluation and design

CONTEXT
IMPORTANT ROLE OF USABILITY IN RESEARCH
DEVELOPMENT
4
General Objectives

• To bridge the gaps in the current usability
  research on investigating the scope and
  applicability of different usability evaluation
  methods in TEL systems
• To bring more science to bear on the field of
  usability, which is still young but highly
  significant, and consequently drive it towards
  maturity

5
Specific Objective

• M14.2 Usability Evaluation of the Common
  Infrastructure
• D14.12 Report of Usability Evaluation of the
  Common Infrastructure of PROLEARN

6
Methods and Results Three Empirical Works
7
Empirical Work 1 - Background

• Title User Rationality and System Learnability
  Performing Task Variants in User Tests
• Partners ETHZ, JSI
• System EducaNext (http//www.educanext.org)

8
Empirical Work 1 Outcome

• Method
• - 22 participants from ETHZ and JSI
• - Ten tasks of which two were near-identical
  (i.e. task variants)
• - Quantitative (e.g. time-on-task) and
  qualitative data (e.g. think-aloud protocols)
• Main Finding
• - Including task variants in user tests can
  provide more valid and reliable assessment of
  usability, especially learnability
• Main Publications
• - Behaviour Information Technology
• - Journal of the American Society for
  Information Science and Technology

9
Empirical Work 2 Background

• Title Evaluation of Personalized Search Engine
• Partners ETHZ, JSI
• System ELENA Human Capital Development Suite
  (http//www.hcd-online.com/HCDExp/ubp)

10
Empirical Work 2 Outcome

• Method
• - 22 participants from JSI, ETHZ, WUW, ZSI, and
  UPM
• - Seven search tasks general, goal-specific and
  interest-specific
• - Compare system-defined and user-defined
  rankings
• Main Finding
• - Extent of within- and between-user
  (in)consistency
• - Flaws in relevance measurement methods
• Main Publication
• - Conference on Human Factors in Computing
  Systems (CHI 2006), (under review)

11
Empirical Work 3 Background

• Title Classification of Usability Problems (CUP)
  Scheme
• Partners ETHZ, ICE
• System UGAL (Owl Teaching Web Version 2.0)

12
Empirical Work 3 Outcome

• Method
• - 11 students for user testing baseline
  measures
• - 8 faculty for applying CUP reliability
  measures
• - 2 developers for evaluating CUP validity
  measures
• Main Finding
• - Established reliability and validity of CUP
• - Further improvement of CUP
• Main Publication
• - Refereed book chapter User-centred Design of
  Online Learning Communities, IDEA Publishing Co.
  (under review)

13
Social Network Analysis

• Partners ETHZ, ZSI
• Aim Investigate the evolution of PROLEARN
• Expected outcomes
• Visualization of integration patterns
• Insights into computer-supported collaborative
  work
• Please complete the Questionnaire ?