Web performance Part2: The users perspective

1
Web performance-Part-2 The users perspective

• CS 7270
• Networked Applications Services
• Lecture-8

2
Reading

• Measuring Human Satisfaction in Data Networks
  by M. Andrews et al.
• Appeared at Infocom06
• First paper to seriously look at Human-Centered
  performance metric for Web browsing

3
Measuring Human Satisfaction in Data Networks
Matthew Andrews Jin Cao Jim McGowan Bell
Labs April 27, 2006
4
Objective -gt Subjective

• Common questions
• What is good performance?
• What do data users actually need?
• Objective measurements useful but not enough
• Need to convert objective measurements into
  subjective score
• In voice world, have notion of Mean Opinion
  Score
• Converts echo, distortion, latency into opinion
  score
• Can we create a Data Mean Opinion Score for
  data applications?

5
Data Mean Opinion Score
USER PERCEPTION
OBJECTIVE PERFORMANCE

• Can we map objective performance into user
  perception?
• Is there some minimum threshold for acceptable
  performance?
• Is there some maximum threshold beyond which
  better performance isnt noticed?

6
Methodology for Subjective Performance Evaluation
Module

• Use (wireless) link emulation software
• Enables us to run real applications over variety
  of link conditions
• Measure satisfaction with human subjects
• Subjects use typical data applications under a
  given link condition
• Subjects given several typical real-world tasks
  to obtain ecologically valid scores
• Subjects are asked to score their experiences
  using a variety of measures
• Create Data MOS function
• Use Principle Components Analysis (PCA) to
  disentangle effects of different tasks and reduce
  number of objective/subjective variables
• Generate Data MOS function that maps objective
  and subjective measures into a single score.

DATA MOS FUNCTION
CLIENT
SERVER
LINK EMULATION
SUBJECTIVE
OBJECTIVE
7
User applications

• This talk
• Web browsing (canonical data application)
• Other applications weve tested
• FTP
• Exchange email
• Instant messaging

8
Important questions . . .

• Does the data MOS function exist?
• How much variation is there person to person?
• What variables among objective measurements are
  most influential on data MOS
• Bandwidth (meanvariance)? Latency? Jitter? What
  else?
• When comparing different commercial networks
• How do we take into account network congestion?
• How do we take into account geographical location
  of measured users?

9
Important questions...

• Network effects vs website effects?
• How much is user perception determined by network
  effects?
• How much is user perception determined by website
  design?
• Frustration due to intricate page with many small
  objects
• Frustration due to poorly designed site that is
  hard to navigate
• ? Investigating very low-latency sites (google),
  highly designed and well-branded sites (Barnes
  and Noble), poorly designed sites (NJ Transit)
  and hard-to-find information on well-designed
  sites (HowStuffWorks).
• Influence of user goals
• User who is told to simply download a webpage may
  have different opinions from users needing to
  complete a more complex task.
• ? Users not simply rating delay, but delay is
  allowed to affect their ratings of quality.

10
Experimental Design
11
User tasks

• Simple page downloads
• Users quickly get to information, then may
  browse or read slowly
  (ranges from 2
  pages to 4).
• Steps toward goal are clear, user is simply
  waiting between clicks.
• Go to www.google.com. Search for Bell Labs
• Go to www.cnn.com. Click on the Politics
  section.
• Go to www.espn.com. Find the current position of
  the New York Yankees.
  (Click on MLB. Click on
  standings.)
• Measures
• opinions overall quality, directed questions,
  etc.
• competence ability to reach page, correctness
  for question 3.

12
User tasks

• Goal driven tasks
• Users dont necessarily know how to get to
  information directly
  (ranges from lt5
  pages to sometimes gtgt 10 pages).
• Many steps toward goal, not every step makes
  progress toward goal, sometimes users cant even
  find the information (although information is
  always available).
• Go to www.bn.com. Find the price of the book
  Friday by Robert A. Heinlein. Easy
• Go to www.njtransit.com. Find the timetable for
  the Morris and Essex train line. What time is the
  first outbound train from Penn Station New York?
  Difficult
• Go to www.howstuffworks.com/laser.htm. What kind
  of laser can cut through steel? Difficult
• Rate four Rutgers professors at
  www.ratemyprofessor.com Long
• Find six world records at www.guinessworldrecords.
  com Long
• Measures
• opinions overall quality, directed
  questions, etc.
• competence ability to reach page, correctness

13
Questionnaire

• For each task we ask
• Question 1 What is your opinion of the overall
  quality of this web surfing experience?
• Question 2 How easy was it for you to complete
  the task?
• Question 3 Was it easy to find information on
  the website?
• Question 4 Was the site visually appealing?
• Question 5 Did the website seem sluggish or
  responsive?
• Question 6 How quickly did the website load?
• Also measure
• Network conditions
• Did subject complete task
• Did subject answer correctly

14
Network configuration

• Bandwidth
• Link bandwidths varied between 20kbps and 1Mbps
• Bandwidth was held constant for each task
• Assignment of bandwidth to task done randomly for
  each subject
• Delay
• Propagation delay varied between 0ms and 300ms
• However, queuing delay still present!!!!

15
Results
16
Recap

• EIght web browsing tasks
• Google search
• CNN download
• ESPN download
• Search for book price on BarnesNoble
• Look up train time on NJ transit
• Find out about a laser on howstuffworks.com
• Rate Rutgers professors on ratemyprofessor.com
• Look up world records on Guinness world record
  site
• Questionnaire
• Q1 overall opinion
• Q2 Q3 ease of use
• Q4 visual appeal
• Q5 website responsiveness
• Q6 download speed

17
Results

• Run 83 subjects
• Main results
• At low bandwidth, opinion is linear with log of
  bandwidth
• At high bandwidth, opinion score saturates
• No difference observed between 0ms prop delay and
  300ms
• Three notions of bandwidth
• Link bandwidth speed of link
• Browser bandwidth how fast does browser take
  in data
• Bandwidth opinion answers to Q5 and Q6 on
  questionnaire

log(Link bandwidth)
log(Browser bandwidth)
Bandwidth opinion
log(Link bandwidth)
18
Results
Download speed
Overall opinion
Responsiveness
200 400 kbps
19
Results
Guiness world records
Google
CNN
ESPN
BN
NJ Transit
Laser
Professors
Download speed Responsive Overall opinion
20
Results
21
PCA

• Two main factors explain most variance in
  subject ratings.
• Not surprising, since we focus study (and
  therefore subjects) on two factors Delivery
  and Design.
• Overall quality is a roughly balanced
  combination of both of these factors.
• Only a portion of the log2(Bandwidth) is
  explained . . . Link Bandwidth and Bandwidth
  Opinion are different.

original factor 1 (unrotated)
Design
Overall
Delivery
link bandwidth ? delivered bandwidth ?
perceived bandwidith ? bandwidth opinion (what
we control)

(what we measure)
22
How does propagation delay affect objective
performance?

• www.cnn.com
• Response time (Time for text to appear on screen)
• Download time (Time for page to complete after
  text appears)

200kbps bandwidth vary propagation delay
Curve almost flat Propagation delay dominated by
queueing delay!!!