Title: Rigorous Performance Testing - But, Wait: There’s More (Data)
1RIGOROUS PERFORMANCE TESTING - BUT, WAIT THERES
MORE (DATA)
BY GRANT ELLIS
2This is the third and final blog post on how to
best evaluate performance on the Web. If you
havent already read parts one and two, then
please do for some vital context Part 1
Rigorous Performance Testing How We Got
Here Part 2 Rigorous Performance Testing
Modern Testing Tools
3(No Transcript)
4In short, the Internet has changed substantially
over the past few decades, and particularly over
the past few years. Infrastructure has evolved,
delivery media has evolved, software has evolved,
browsers have evolved, and web sites have
evolved. These trends yielded new paradigms in
development, deployment, delivery, and testing.
There are new tools out there designed to address
all of these changes and give you great data that
truly represents user experience. So, supposing
youve read my prior blog posts, youve chosen
the best performance metrics and youve collected
a boatload of data with the testing tool of your
choice. Now what? How do you interpret those
results?
AVERAGES, MEDIANS AND PERCENTILES OH MY!
Generally speaking, performance results are
expressed as either an average or a median.
Heres the problem average can get skewed
horribly based on just a handful of severe
outliers. Median expresses only the performance
of your middle of the road users but discards
the data from your fastest and slowest users. As
a service provider, you should care about all of
your users but you should definitely have
particular concern for your slowest users. One
possible workaround to the above problems is to
use a percentile. Aggregate performance data
expressed as a percentile is (loosely) sorted in
ascending order, and the first nth percentile
figures are lower than the calculated figure. So,
for example, if the 95th percentile of a
population is 5.3 seconds, then you know that 95
of page loads were completed in 5.3 seconds or
less.
5- There are still outliers, no matter what
percentile you choose (except for 100th,
otherwise known as the max) - All data points except for the designated
percentile are essentially discarded. For
example, if you request a 95th percentile, you
have no idea what load times the 94th percentile
or the 96th percentile saw let alone the 1st,
15th, or 50th. - The outliers are, by definition, your slowest
users. Dont you care about them the most? Why
would you discard them?
BUT WAIT THERES MORE (...DATA)!
- The bottom line is that, if you went to the
effort and expense of collecting hundreds or
thousands of data points, then those data points
contain hundreds of data permutations
different - internet connection types
- devices
- browsers
- geographies
- wireless connection qualities
- computing power
- And, aside from all of the above variables, there
is the natural variability inherent to the
Internet. - Why should you express all that data as a cheap
sound bite especially a misleading one? There
is much more data. Lets figure out how to use it
all.
6ENTER THE HISTOGRAM
- If we have hundreds or thousands of data points,
it is just plain wrong to distill it all down to
a single figure. So, we should try a new
visualization. The histogram expresses how many
users experienced a particular page load time - Taller bars mean that more users saw a load time
within that interval. - Conversely, shorter bars mean that fewer users
saw a load time within that interval. - Faster load times are on the left of the
histogram, and slower load times are on the right
side. - So, the crux of the histogram is this if there
are more, taller bars on the left side of a
histogram, then your users had a fast experience. - Histograms really shine if you are comparing
multiple data sets. You can simply put both data
sets on the same chart. See this example
histogram
7The red bars are more concentrated on the left,
and the blue bars are more concentrated on the
right. It is obvious that, in this example, red
users had a faster experience overall.
8HIGH OCTANE THE CUMULATIVE DISTRIBUTION FUNCTION
(CDF)
Everyone loves histograms. They represent all the
data. They are easy for everyone to consume.
There are no misleading statistical cuts. But,
despite all these great points, histograms have a
shortcoming finite granularity. Every histogram
has discrete buckets and it is complicated to
find a good representation. Too many buckets and
you end up with just a long tail that is hard to
understand. Too few buckets, and the load times
get lumped together and become meaningless. There
is a (much) lesser-known chart that represents
all performance data with incredibly high
granularity (roughly, as sharp as your monitor!)
the cumulative distribution function (CDF). The
CDF expresses the percentage of page loads
completed after a given amount of elapsed time
9So, in this example CDF, approximately 20 of
page loads are done in 5 seconds or less.
Slightly less than 70 of page loads are done in
10 seconds or less.
10As with histograms, better CDFs have a curve to
the left. If you plot two separate data sets on
the same chart, it is easy to see which one is
truly faster
11In this comparative CDF, the speed difference is
expressed as the gap between the two lines. You
could summarize a part of this chart in one
statement by saying something like when 80 of
blue users are done loading the page, over 90 of
read users are done loading the page. More
importantly, since the red line is above and to
the left of the blue line for the entire length,
we know that red is faster for all users not
merely a subset of those users.
TIE IT ALL TOGETHER...
- Performance on the Internet is a black box for
many, and I hope that my blog series has shed
light on some dark corners. Here are some
conclusions to chew on until next time - The Internet is a jungle.
- Methodology matters more than results.
- Pick your tool wisely.
- Irrelevant metrics mislead.
- Performance should never be expressed as a single
number. - Powerful visualizations always trump aggregate
figures. - Spreadsheets are amazing. Use them.
- Positive or negative, I always want feedback.
Please email me at grant_at_instartlogic.com. - Thanks for reading!
12Want to know more? Visit our Blog