Evaluation of Adaptive Web Sites

1
Evaluation of Adaptive Web Sites
by

• Elizabeth LaRue

2
Presentation Resources

• Gena, C. (2005). Evaluation of Adaptive Web
  Sites. Retrieved September 6, 2005, from
  Courseweb.
• Gena, C. (2002, January 13-16). An empirical
  evaluation of an adaptive web site. Paper
  presented at the International Conference on
  Intelligent User Interfaces, San Francisco, CA.
• Marucci, L., Paterno, F. (2002). Design and
  evaluation of an adaptive virtual guide for web
  applications. UAIS Universal Access in the
  Information Society, 1(3), 163-176.
• Cheverst, K. Davies, N. Mitchell, K. et al.
  (1-6 April 2000). Developing a context-aware
  electronic tourist guide some issues and
  experiences. CHI. electronic source
  retrieved from ACM Digital Library.

3
Objectives

• Review associated terminology
• Describe project phases
• Examine various evaluation methods

4
Terminology

• Adaptive Web Sites present and format
  information material automatically on a web page
  for the user
• Evaluation to examine carefully 1
• Empirical relying on or gained from observation
  or experiment rather than theory 1

1 Houghton Mifflin Company. (1996). Webster's ii
new riverside dictionary (Revised Edition ed.).
Boston Houghton Mifflin Company.
5
Benefits of Evaluation

• Helps design
• Insight into user behavior
• Tests usability and functionality
• Assists with user models

6
Adaptive Systems Evaluation

• Consider a layered approach separate evaluation
  into the adaptive components of the adaptive
  systems
• A. Content layer
• B. Interface layer

7
Software Lifecycle Development
Waterfall Methodology
Planning/Requirements
Design/Analysis
Code/Program
(development)
Testing
Implementation
8
Planning/Requirements
Why?
Continual EVALUATION helps to complete the
requirements specification document
Assists in Focusing on users typical behavior,
actions, needs, etc Focusing on tasks
actions, needs, etc
9
Design/Analysis
Why?
Assists in Prohibiting expensive design
mistakes Prohibiting costly redesign
10
Code/Program
(development)
Why?
Assists in Evaluating overall quality of a system
11
Before Evaluating

• Data collection query technique
• Usually used at each step in the life cycle
• Usually cheap
• Usually simple

12
Before Evaluating - Tools

• Questionnaires
• Interviews

13
Before Evaluating ToolsObservational Methods

• Verbal and think aloud (pre and post)
• Users observation
• Logging Use

14
Three Types of Evaluation

• Formal
• Analytical
• Empirical

15
Formal Type of Evaluation

• Rarely used
• Look at prediction of performance, complexity and
  learnability

16
Formal Methods

• Task Analysis
• Cognitive
• GOMS
• KLM (Keystroke Level Model)
• Socio-technical models

17
Analytical Type of Evaluation

• Used without users, must have clear evaluation
  criteria of the system
• Benefits
• Quick
• Cheap
• Reasonably effective

18
Analytical Models

• Heuristic Evaluation
• Cognitive walkthrough
• Accessibility
• Automatic Usability Testing and Web usage mining

19
Empirical Type of Evaluation

• Best for objective assessment of design
• Best for broadest range of usability problems
• Requirements to do empirical evaluation
• 1. Careful planning
• 2. Careful execution
• 3. Users must represent actual user population
• 4. Must have expense account pay users

20
Empirical Methods

• Contextual evaluation
• Focus group
• Systematic Observation
• Expert review
• Wizard of Oz prototyping
• Prototyping
• Card sorting
• Cooperative evaluation
• Participative evaluation
• Usability testing
• Accessibility
• Automatic Usability Testing and Web usage mining
• Controlled experiments
• Ethnography
• Grounded theory
• Metric to evaluate the effectiveness of adaptive
  contents

21
Specific Examples

• Gena, C. (2002, January 13-16). An empirical
  evaluation of an adaptive web site. Paper
  presented at the International Conference on
  Intelligent User Interfaces, San Francisco, CA.
• Mixed Methods Empirical Method
• survey
• parallel design
• task analysis
• questionnaire - post

22
Evaluation of adaptive commercial web site
Gena, C. (2002, January 13-16). An empirical
evaluation of an adaptive web site. Paper
presented at the International Conference on
Intelligent User Interfaces, San Francisco, CA.
23
Evaluation of adaptive commercial web site
Gena, C. (2002, January 13-16). An empirical
evaluation of an adaptive web site. Paper
presented at the International Conference on
Intelligent User Interfaces, San Francisco, CA.
24
Evaluation of adaptive commercial web site
Test 1. Measure the task completion time
2. Amount of within page navigation
3. The satisfaction of the users
Evaluation Methods Interaction simulation with
pre-test Accomplish task in non-adaptive version
and then repeat in adaptive version
Study Sample 14 subjects Aged 24-35 High Internet
knowledge and web browsers and use the Internet
during work
Gena, C. (2002, January 13-16). An empirical
evaluation of an adaptive web site. Paper
presented at the International Conference on
Intelligent User Interfaces, San Francisco, CA.
25
Specific Example 2

• Marucci, L., Paterno, F. (2002). Design and
  evaluation of an adaptive virtual guide for web
  applications. UAIS Universal Access in the
  Information Society, 1(3), 163-176.
• Mixed Methods Observational Method
• Questionnaire
• Think aloud

26
Evaluation of adaptive museum web site
http//giove.cnuce.cnr.it/servlets/StartVisitEng

• Marucci, L., Paterno, F. (2002). Design and
  evaluation of an adaptive virtual guide for web
  applications.
• UAIS Universal Access in the Information
  Society, 1(3), 163-176.

27
Evaluation of adaptive museum web site

• Marucci, L., Paterno, F. (2002). Design and
  evaluation of an adaptive virtual guide for web
  applications.
• UAIS Universal Access in the Information
  Society, 1(3), 163-176.

28
Evaluation of adaptive museum web site
Test 1. Does the virtual guide make a visit
more pleasant and instructive 2.
Learn if the type of information provided by the
guide was interesting to the user
3. Is the presentation of the material effective
or could be improved
Evaluation Methods Think aloud without tasks
minimal 30 minutes with and without adaptive
system Post-questionnaire 29 questions
Study Sample 40 subjects Aged 20-53 (16 males and
24 females) Previous web experience and half
rarely accessed a museum web site

• Marucci, L., Paterno, F. (2002). Design and
  evaluation of an adaptive virtual guide for web
  applications.
• UAIS Universal Access in the Information
  Society, 1(3), 163-176.

29
Evaluation of adaptive museum web site
Think aloud methodology

• Evaluators recorded particularly negative and
  positive reactions
• Evaluators noted specific paths followed by
  users
• Evaluators were careful to not make suggestions
  to users

Questionnaire

• 15 questions on general usability of interface,
  navigation and ease of access
• 5 questions related to each of the five types of
  information provided by the
• virtual guide
• 9 questions related to user personal
  information, i.e. kn of museum application
• domains

• Marucci, L., Paterno, F. (2002). Design and
  evaluation of an adaptive virtual guide for web
  applications.
• UAIS Universal Access in the Information
  Society, 1(3), 163-176.

30
Specific Example 2A
Evaluation of tourist web site (GUIDE)
Mixed Methods Expert walkthrough one hour Talk
aloud Post-interview
Study Sample 4 expert subjects Knowledge area of
user-centered design and computer supported
learning

• Marucci, L., Paterno, F. (2002). Design and
  evaluation of an adaptive virtual guide for web
  applications.
• UAIS Universal Access in the Information
  Society, 1(3), 163-176.

31
Evaluation of tourist web site (GUIDE)
http//www.guide.lancs.ac.uk/whatisguide.htm l
Cheverst, K. Davies, N. Mitchell, K. et al.
(1-6 April 2000). Developing a
context-aware electronic tourist guide some
issues and experiences. CHI. electronic
source retrieved from ACM Digital Library.
32
Evaluation of tourist web site (GUIDE)
Prototype Evaluation Methods Talk aloud
without tasks Time stamped log files Post
semi-structured interview
Study Sample 60 volunteer subjects Aged 10-70
Knowledge area of user-centered design and
computer supported learning
Cheverst, K. Davies, N. Mitchell, K. et al.
(1-6 April 2000). Developing a
context-aware electronic tourist guide some
issues and experiences. CHI. electronic
source retrieved from ACM Digital Library.
33
Thank You