CS533%20Performance%20Evaluation

1
Projects Overview

• CS533 Performance Evaluation

2
Your Projects This Semester

• Performance Measurement Warm-ups
• A series of small tasks to get you in the mode
  for measuring and reporting
• Follow closely the material in the text
  Network Performance

• The Main Performance
• A large project following your own interests.
• Contains a series of steps to ensure success.
• These steps include
• Project Pre-Proposal
• Project Proposal
• Design Proposal
• Project Report

3
Warm-Ups
Students taking this course have usually had a
strong interest in network performance. As such,
these warm-up exercises give you a chance to
practice measurement on networks.

• There are three warm-ups to be accomplished in
  the next few weeks
• Using a network snoop device Wireshark or
  tcpdump
• Measuring network performance
• Measuring application performance

Warning these warm-ups are undefined and open
ended.

• Ground rules for all warm-up reports.
• Note the measurement and the report both
  matter.
• The report should state clearly whats been done.
  
• It should state the environment used. Wireshark
  was used to measure the round-trip ping time as a
  function of physical distance between sites.
• The report should include measurements where one
  thing depends on another there MUST BE a graph
  in your report.
• There should be an indication of the accuracy of
  your measurements. Whats the uncertain in your
  values? Are they repeatable? If someone else
  were to measure the same thing, would they get
  the same result?
• Plan on delivering your report to the class
  should be no more than 5 minutes in length. I
  will critique your results on the fly. Everyone
  will learn.
• Plan also on handing in your report it can be a
  few slides for instance or it can be on paper.

4
Warm-Ups
Does Correlation Imply Causation?
Examples There's a strong correlation between
foot size and ability to do arithmetic. (Of
course, it's not causation both measured
variables are strongly correlated with the age of
the children. Older ones have bigger feet and
better math ability.) People who drink diet soft
drinks don't lose weight. In fact, they gain
weight, a new study shows. (Best Theory When
we offer our bodies the sweet taste of diet
drinks, but give them no calories, it makes us
crave more calories.) The more firemen fighting
a fire, the bigger the fire is observed to be.
(Therefore firemen cause an increase in the size
of a fire? ) As ice cream sales increase, the
rate of drowning deaths increases sharply.
(Therefore, ice cream consumption causes
drowning.)
5
Warm-Ups
Use of Performance Nomenclature It is expected
that by the time you finish each of these
warm-ups, you will have made use of Performance
concepts. It is SOOO easy to talk only about
throughput, for instance, and ignore all the
other nice concepts. But Im expecting that you
will talk about Response Time Utilization Through
put Capacity/Bandwidth
6
Warm-Ups
Ground rules for all warm-up reports. The WPI
Acceptable Use Policy http//www.wpi.edu/AUP
clearly prohibits the use of sniffers on the
network. This rule is in the Acceptable Use
Policy to prevent denial of service conditions
resulting from 'active' sniffing, which uses
gratuitous ARP traffic to redirect packet flow,
and to preserve privacy and uphold Federal law by
not allowing students to accidentally wiretap a
communications channel. There is a way to
perform the measurement tests as described in
these projects that does not violate the AUP.
Students need to run their sniffers in
non-promiscuous mode. This allows for collection
of data in the sniffer that they would be
processing anyways, without seeing any other
traffic on the network. This solution avoids
both potential problems outlined above. Here is
how to use non-promiscuous mode for your tests.
In Wireshark, that setting is a checkbox on the
capture configuration screen. In tcpdump, that
setting is the '-p' flag.
7
Environments and Tools
In doing these warmups, were exploring three
different levels of environments. Warmup 1
focusses on very LOW level environments where
we look at individual actions for instance,
individual network packets, individual disk
accesses, individual process scheduling. We can
study only ONE action and get as much possible
information as we can. Or we can look at
multiple instances of this simple action, with a
goal of getting more information about its
performance. Warmup 2 looks at the next
higher level of environment. An aggregate
action a stream of data made up of many
packets, disk accesses over a span of time, and
so on. Warmup 3 in intended to look at a
complex environment. For instance, the disk
activities that occur to a user environment,
the network traffic coming from multiple sources,
the car and truck traffic associated with an
intersection, etc. These do not lend themselves
to simple analysis, and we get only aggregate
results. Once weve identified the environment,
then we can select the tool. Low level
environments require tools that can measure
individual actions Wireshark to watch an
individual network packet, timing of individual
left turns at an intersection. Medium level
experiments probably wont measure individual
actions, but require tools that can measure
throughput, response time, queueing time for a
series of actions. High level, complex
environments will perhaps need tools that
understand the environment. For instance, a
database in the cloud will support transactions
made up of a number of smaller actions it may
involve network, disk, and other resources.
8
Warm-Up 1
Use a software snoop device to measure network
traffic on a machine.

• What you need to know
• Chapters 1 2 in Network Performance
• Class notes from the first lecture on measurement.

• In this project, you will install a sniffer on a
  machine of your choice and use it to measure low
  level network performance.
• A sniffer is a program running on a machine that
  is able to capture packets and packet information
  on that machine. In considering what machine to
  use, be aware that you will need generally
  administrative access to that hardware. Chapter
  2 of Network Performance talks about tcpdump,
  windump, and Wireshark (formerly ethereal). Im
  a Wireshark fan myself, but it doesnt really
  matter which you use.
• Heres your assignment
• Measure some aspect of network traffic where you
  measure the same thing but change some one or two
  variables. Organize your results so they are
  easily understandable (presentation is
  everything!) including a plot or graph.
• Here are several possible examples but I
  encourage you to think up your own
• While Wireshark is running, connect to a website
  with your browser. How long does it take to
  complete the handshaking required to establish
  the connection. Now vary the distance to the
  site youre connecting to. Does this change the
  connection time? Does the connection time depend
  on other things like time of day, number of
  hops, etc. ( Remember that correlation does not
  imply correlation! What is the real cause of a
  dependency you find?)
• While Wireshark is running, ping a number of
  sites and determine the exact time it takes for
  that ping to complete (rather than the very
  general number reported by ping.) The ping time
  depends on what distance, hops, etc???

9
Warm-Up 1
Name______________________
Use a software snoop device to measure network
traffic as seen by a node.

• Does the measurement accomplish the assignment?
• Uses a sniffer of some kind.
• Measures network traffic at a low level (and not
  via some mega-very-complicated-application high
  level interface).
• Uses a simple tool (browser, ping, or traceroute,
  for instance) to drive the traffic.
• Measurement Criteria
• Is the experiment conducted such that theres
  only one independent variable at a time? And
  that one independent variable leads to a change
  in the dependent variable i.e., the dependent
  variable is really dependent?
• Are there several measurements under the same
  conditions do we have the ability to assess the
  uncertainty in the results obtained? Is there
  discussion of significant figures, standard
  deviation and variation so we know how much we
  trust the numbers?
• Is there an analysis of why the variations seen
  in the numbers might occur? Is there a sanity
  check that the information presented makes sense?
• Is there an analysis that the dependence is due
  to causation and not just correlation?
• Is there a physical model presented that explains
  why the observed behavior occurs?
• The presentation
• Is well organized.
• Theres a plot or graph presented. The visual
  information accurately represented what was
  measured.
• There was an in-class delivery and material given
  to me for further evaluation.
• Completed within the time limit.

10
Mid-Level Tools

• In Warm-Up 1, there was a low-level tool you
  used Wireshark is an excellent example it
  gives lots of very detailed information.
• Warm-Up 2 will use tools at the next higher
  level. On Windows these might be the Task
  Manager (fairly simple statistics) or the
  Performance Monitor that gives you detailed
  information. Neither of these give packet level
  details, but they provide information about
  aggregate network behavior.
• On LINUX, there are not such nice tools that come
  with the OS (at least, not that Ive found.)
  There are numerous tools out there that you can
  download. Netstat and traceroute come with
  LINUX, but the command line interface isnt much
  you can get free GUI interfaces for them. Also
  take a look at http//www.uperf.org/
  http//www.cyberciti.biz/tips/top-linux-monitoring
  -tools.html and many others.
• Note that there are two flavors of measurement
  here and either is appropriate for your
  experiment
• The measuring device is independent of the
  generator of traffic its essentially standing
  outside of the generator and gathers whats
  happening.
• The tool that generates the traffic also
  incorporates a means of measuring that traffic
• The network generation tool is usually designed
  to max out the connection between A and B. Its
  a measurement of capacity, and will find a
  bottleneck somewhere along the path. This is
  more of a network measuring tool (fine for this
  Warmup) than a system or application tool.

11
Warm-Up 2
Measure Network Performance - throughput

• What you need to know
• Chapters 4 12 in Network Performance
• Class notes from the first lecture on measurement.

In this project, you will measure the performance
of a network. This network might be a local LAN
consisting of machines on the same switch, it
could be a metropolitan network of machines near
us, or it might span the globe your
choice. Heres your assignment Use several of
the tools described on the previous slide to
measure network traffic. As in Warm-Up 1, vary
the properties of what you are measuring in order
to get a variety of measurements, and again
display and explain your results. Here is one
possible example but I encourage you to think
up your own 1. Determine the throughput
between your machine and an off-site location as
a function of the time of day. When is the best
and worst time to use your machine as a
source across the WPI network? Are your numbers
consistent from day to day? What matters in
getting good throughput? Your report should
include the data analysis as you did in Warm-Up
1, (see Ground Rules) but in addition you should
report on the relative advantages and
disadvantages of the several tools you used..
12
Warm-Up 2
Name______________________
Measuring a Mid-Level Environment

• Does the measurement accomplish the assignment?
• Uses a tool to measure traffic.
• Measures network traffic at a medium level (and
  not via some mega-very-complicated-application
  high level interface).
• Uses a simple tool to generate the traffic
  perhaps same as measuring tool.
• Measurement Criteria
• Is the experiment conducted such that theres
  only one independent variable at a time? And
  that one independent variable leads to a change
  in the dependent variable i.e., the dependent
  variable is really dependent?
• Are there several measurements under the same
  conditions do we have the ability to assess the
  uncertainty in the results obtained? Is there
  discussion of significant figures, standard
  deviation and variation so we know how much we
  trust the numbers?
• Is there an analysis of why the variations seen
  in the numbers might occur? Is there a sanity
  check that the information presented makes sense?
• Is there an analysis that the dependence is due
  to causation and not just correlation?
• Is there a physical model presented that explains
  why the observed behavior occurs?
• The presentation
• Is at the right level just the right amount of
  detail enough to explain what you did, but not
  so much you are giving TOO much data.
• Theres a plot or graph presented. The visual
  information accurately represented what was
  measured.
• There was an in-class delivery and material given
  to me for further evaluation.
• Completed within the time limit.

13
Warm-Up 3
Measure Application Performance

• What you need to know
• Chapters 14 18 in Network Performance
• Class notes on simulation.

In this project, you will determine the
performance of an application. You can do that
in one of two ways with a traffic
generator/benchmark/workload (Chpt. 16) or with a
simulator (the other chapters) Heres your
assignment Use a tool of your choosing to
emulate the behavior a system would have. This
system is more complex than the simple traffic
patterns that you would get from the network
measurements. This tool can be one you build, it
can be one youve previously used (but
enhanced to emulate a whole system) or could be
a new tool. You are free to come up with an
environment of interest to you. But as before,
measurement and reporting are essential. Please
try to DESIGN YOUR EXPERIMENT There are a series
of steps in Section 12 of the Measurement and
Statistics Slides. Use the steps from there that
are applicable. That section asks questions that
will help you break down your project into
do-able steps. Too many of us do an experiment
and then say thats what I was planning on
measuring all along. This design process is
extremely important. Your report should include
the data analysis as you did in Warm-Up 2, (see
Ground Rules) but in addition you should report
on the relative advantages and disadvantages of
the tool you used.
14
Warm-Up 3
Name______________________
Measuring an Application Environment

• Does the Design and Measurement accomplish the
  assignment?
• Shows a use of the 12-step design process.
• Uses a tool to measure traffic.
• Measures at an application level a fairly
  complex environment. Show that the benchmark you
  are using is representative of a real
  environment.
• Measurement Criteria
• Is the experiment conducted such that theres
  only one independent variable at a time? And
  that one independent variable leads to a change
  in the dependent variable i.e., the dependent
  variable is really dependent?
• Are there several measurements under the same
  conditions do we have the ability to assess the
  uncertainty in the results obtained? Is there
  discussion of significant figures, standard
  deviation and variation so we know how much we
  trust the numbers?
• Is there an analysis of why the variations seen
  in the numbers might occur? Is there a sanity
  check that the information presented makes sense?
• Is there an analysis that the dependence is due
  to causation and not just correlation?
• Is there a physical model presented that explains
  why the observed behavior occurs?
• The presentation
• The presentation is mailed to me by noon on the
  date it is due.
• Includes a discussion of the environment being
  studied.
• Is at the right level Not too much or too little
  detail.
• Theres a plot or graph presented. The visual
  information represents what was measured.
• There was an in-class delivery and material given
  to me for further evaluation.
• Presentation Completed within the time limit.

15
The Main Performance
Choosing a Project. There are of course many
great ideas for projects. Heres my overview of
these ideas.
Simulations I am delighted if you want to do a
simulation.  This can be a simulation of computer
behavior (of course), or it can be of any other
phenomenon.    There are two conditions for the
simulations I will accept a)  A simulation
based on some REAL data of a physical
phenomenon.  You can use some already-available
simulation tool for this.  BUT the tool must be
able to predict a result that you didn't measure
and you need to be able to verify this
prediction. It is not acceptable to use an
existing model with data you get from the web or
anywhere else you must measure it. EXAMPLE 
You measure a traffic intersection.  You know the
timing of all the lights, you know traffic
patterns you know for each minute of a two-hour
period how many cars are queued waiting for a
left turn, etc.  You know EVERYTHING about that
intersection.  You put that data into an existing
simulation package and tune it to macth your
intersection.  THEN you give it the light
patterns and basic properties for a second
intersection - now the model is able to
successfully predict the traffic patterns of this
second intersection.  That's how you know the
model is correct.
16
The Main Performance
Choosing a Project.
Simulations (Continued) a)  You build the
simulation model.  You need to collect some data
for this, but Im much less concerned about the
extend of that data youve doing a lot of hard
work building the model, so that makes up for the
lesser time spent measuring. EXAMPLE  On the
following pages, I give two examples of models
built by former students one build a monopoly
game it had a mechanism for moving the pieces
around the board, included buying and selling of
properties, for Chance and Community Chest,
etc. Another student built a network model
including several routers and nodes and
determined what happens as traffic increases in
this environment.
17
The Main Performance
Choosing a Project.
Measurements People almost always measure
something thats computer or network related
its so much easier to determine the performance
characteristics of such things. You get to use
all the things youve learned this semester.
Talking about the performance of weather or the
performanceof responses to a questionnaire
while theoretically possible, I suppose, are
pretty hard to do in practice. Usually in this
case your project would be a bigger example of
the kind of thing you did for your warm-ups. It
would follow the twelve-step process, and would
be of a bigger scope and in more detail than you
did for those warm-ups. Heres a list of
measurement tools that people have used in the
past Tracing and Tools Profiling Project in
Eclipse, Iometer, Perfmon, Tcpdump, Wireshark,
Iperf Simgrid ? simgrid.gforge.inria.fr,
NetworkTrafficGenerator (ntg), Network Simulator
(ns)
18
The Main Performance
Choosing a Project. Projects From Last Year
Ray Tracing Performance A study of the factors
affecting the performance of a program rendering
a three dimensional object.  Evaluating Virtual
Machine Software Performance Compares the
performance of two virtual machines as well as
native OS running a number of benchmarks on Linux
and Windows. Performance Analysis of Common
Security Tools Looks at the hardware and
network performance of two products. Slingbox
Analyzing the performance of the video streaming
of SlingBox under different network
settings. Traffic Simulation Wrote a simulation
model (from scratch) and used it to understand
automobile traffic patterns. Monopoly
Simulation Wrote a simulation model (from
scratch) for the game of monopoly. This did not
use much real data, but presented a major
challenge in modeling the various aspects of the
game.
19
The Main Performance
Choosing a Project. Projects From Last Year
Web Server Performance What factors affect the
performance of Apache Tomcat. Home WLAN
Performance Studies factors affecting the
performance of home networks. Mesh Networks
Measured and analyzed the performance of networks
of computers. Hadoop A study of the measured
performance of a big data database. Data Base
Behavior Performance factors affecting the
measured performance of data base
updates. Mobile Hard Drive Measured the
performance of a hard drive under a large number
of conditions.
20
The Main Performance
It is assumed that you will choose a topic an
aspect of performance that is of interest to you
or that you can use in your job. The project
contains several components as shown below. The
due dates for these components are given on the
main syllabus page.

• Project Pre-Proposal
• Goal To get agreement on a topic in an informal
  way.
• Deliverable Here you and I have a
  conversation/e-mail/brief write-up that contains
  a very short statement of what you want to do for
  a project.
• I Owe You Very rapid feedback that the direction
  you have chosen is OK.
• What To Look For Major issues I will check for
  include
• Scope/size - does the project you are proposing
  contain the appropriate amount of work?
• Diversity does the project contain several
  components such as measurement, simulation, etc.
  from the topics we are discussing in class.

21
The Main Performance

• Project Proposal
• Goal To allow you to write up your project and
  present it to the class.
• Deliverable You will prepare and deliver a 10
  minute class presentation. You will also
• have a written
  proposal. Its the written proposal that will
  be evaluated
• for a grade.
• Whats Included There are a number of items that
  should be here. They include
• A description of the performance project you
  will do.
• What performance components are included in
  this project.
• An explanation of how this fits into some
  larger context.
• A justification of why this is important and/or
  useful.

• Design Proposal
• Goal To give a formal progress report on your
  project. By this time, you have
• gone down a number
  of dead-ends, figured out a number of details,
  and
• now have a project
  thats quite a bit different from when you made
  your
• proposal. We
  want to hear where youre at.
• Deliverable You will prepare and deliver a 10
  minute class presentation. You will
• also have a
  written proposal. Its the written proposal that
  will be
• evaluated for a
  grade.
• Whats Included There are a number of items that
  should be here. They include
• A brief reminder of the performance project you
  are doing.
• What performance components you have used thus
  far.
• How your project has changed, and why (ways we
  have gone wrong can be very educational.)
• Difficulties ahead.

22
Design Proposal

• What you should report during this presentation
• Goal To give a status report on your project.
  We want to hear what youve done so far.
• .
• Deliverable You will prepare and deliver a 15
  (FIFTEEN) minute class presentation.
• Whats Included There are a number of items that
  should be here. They include
• A description of the performance project you
  are doing. Focus on the PERFORMANCE aspects and
  less on the network, etc. aspects. Please avoid
  ratholes.
• A justification that this is a PERFORMANCE
  project and not some warmed over project from
  another course.
• The Performance Engineering techniques/practices
  you are using in this project.
• Current progress
• a) Portion of the goal achieved so far.
• b) Portion still to be done.
• 5. A description of

23
Project Report

• Goal To give a formal report on your
  project. We want to hear what youve done
• and how youve done it.
• Deliverable You will prepare and
  deliver a 30 (thirty) minute class presentation.
  You
• will also have a
  written report. Its the written report that
  will
• be evaluated for
  a grade.
• Whats Included There are a number of items that
  should be here
• A brief reminder of the performance project you
  are doing. Functionality (how your product works)
  ONLY to the extent absolutely necessary to
  explain the performance work you've done. Try to
  avoid rat-holes please.
• Items from previous reports youve done that help
  us understand the context of your project.
• The Performance Engineering techniques/practices
  you used in completing your project. The best
  way to do this is to fit your methodology into
  the 12-step process weve discussed.
• Show the performance measurements or performance
  practices that you used. This includes models,
  tools and techniques.
• A description of how much you trust your results.
  Show IN DETAIL the statistical analysis you
  perform to show your trust.