5 Dirty Little Secrets of Web Performance

1
5 DIRTY LITTLE SECRETS OF WEB PERFORMANCE
2

• Psssst  do you want to know a secret? How about
  five? Below I will share with you five dirty
  little secrets of web performance some may
  surprise you.
• EVERY USER GETS A DIFFERENT EXPERIENCE
• There is no such thing as an average user
  experience. No two experiences will ever be the
  same for a web page. There are a host of factors
  that impact a users experience, including their
  physical location, time of day, network
  congestion, whether they are on a wired or
  wireless connection, their device and its specs,
  personalization of the application  the list
  goes on and on. Is a user accessing an
  application with Chrome from Brisbane, Australia
  on a wired connection going to have the same
  experience as a user accessing the application
  with Safari, from Miami, FL in the US across
  their mobile providers network? I highly doubt
  it. Reporting on the average across a single
  parameter such as location or browser or
  connectivity does not take into account the large
  number of other variables that are also
  influencing the user experience. In experiments
  typically only 1 independent variable is changed
  at a time to ensure fair and valid results. With
  multiple independent variables changing,
  measuring the single impact of a variable is
  impossible. So, why are we in the web performance
  community focused on defining the average user
  experience when the list of independent variables
  is so large and in constant flux?
• 2. MOST METRICS DONT REPRESENT REAL LIFE
• Most performance testing is synthetic, which
  means that most metrics are synthetic and are not
  able to accurately quantify the true end user
  experience. Organizations rely on synthetic data
  for the majority of decision making purposes.
  When running synthetic tests, it is important to
  try and mimic real world conditions as much as
  possible in terms of locations of testing agents,
  device types, and connectivity. If the majority
  of users access an application with Chrome from
  Europe using a mobile device, testing should not
  be conducted from the United States on Internet
  Explorer using a cable modem connection.
• Real User Monitoring (RUM) is attempting to
  change the reliance on synthetic data, but is
  still in its infancy. User Timing and Resource
  Timing are not yet supported by all browsers. As
  more browsers adopt these specifications, more
  decisions will be made based on RUM data.

3
3. AVERAGES OBSCURE THE TRUTH Averages are
commonly used to report on web performance
metrics, such as the average page size or average
page load time. Im not sure why averages are
used  maybe its because it is a metric that
most people understand... but remember secret 1.
Ilya Grigorik recently wrote about the myth of
averages when it comes to page size. If the
average page size is a myth, cant the same be
said for any average? Averages make sense when
the data forms a nice bell curve, but in the web
performance space it is rare that we see such a
distribution. Often our data looks like the
image below.
4
I would venture to say there is no such thing as
an average when it comes to web performance given
the variability of the user experience. As a
result, the truth is often obscured. Averages can
be misleading based on outliers or based on the
number of test runs performed. If only three test
runs were performed, is that really indicative of
average performance? For example, say you
conducted an A/B test for a configuration change
to determine impact on page timings. Both
averages across ten runs were 4.1 seconds.
Looking just at the average one might say there
is no difference in the user experience, but if
you look at the individual test runs a different
story is told. Looking at the individual
data points it is harder to say that the
performance is the same for configuration A B.
In configuration B response times go up for 8 out
of 10 users, while with configuration A response
times are significantly higher for 2 out of 10
users. Which is better?
  Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 8 Run 9 Run 10
A 2 2 2 3 3 3 3 3 10 10
B 3 3 3 4 4 4 4 4 6 6
5
4. USER PERCEPTION MATTERS MORE THAN
NUMBERS Traditional web performance metrics have
included items such as time to first byte (TTFB)
and page load times. The problem is these metrics
dont accurately reflect how the end user
perceives page loading. The user doesnt have to
(and most likely wont) wait until all items on
the page have loaded before they begin
interacting with the page. There has been a shift
in the web performance community to focus on
metrics that have more to do with the page
rendering and interactivity, as these are more
indicative of the user experience. Understanding
how a user perceives the page load is much more
important than how long it takes for all
resources to actually load. Take a look at this
side by side comparison of two pages loading.
Which do you think loads faster?
6
Would you be surprised to hear that they both
load in the exact same amount of time? Yet
it feels like the J.Crew page loads faster (at
least I think it does). Humans have memory bias
and perceive time differently. We have a tendency
to perceive things as taking longer than they
actually do then, when recalling an event, we
think it took even longer than we originally
perceived it to. The bottom line is, if a
user perceives a page as loading slowly, thats
all that matters. 5. METRICS CAN
LIE Measurements and metrics can be manipulated
to present anything in a good or bad light. If
the average isnt showing improvement, try
reporting on the median or percentile. If start
render times arent showing a good enough gain,
look at TTFB or page load times. Not all pages
are showing an improvement? Present results for
only the home page. In web performance there are
many metrics that matter reporting on a single
metric doesnt show the whole picture. But when
some metrics show improvement and others dont,
how do you make an informed decision? When
presented with performance results, ask questions
and ask for additional metrics  especially if
you are only presented with a limited set of data
(Tip always ask for full histograms to make
judgments). While it is frustrating to have to
explain to customers why 1 page out of 10 tested
didnt show the results expected, I always prefer
to share more data than less when conducting A/B
or performance tests. In the coming weeks we will
share more details on how to find the truth in
the maze of web performance.
7
Instart Logic makes application delivery fast,
secure, and easy.
Instart Logic provides the first cloud
application delivery service in the world to
extend beyond the limited backend-only cloud
architectures of the past with an innovative,
intelligent end-to-end platform delivered as a
service.Our end-to-end platform combines
machine learning for performance and security
with a CDN for delivery. Designed for DevOps and
mobile-first applications.
Discover how it can work for your
website/application.
Request a Demo