ICOM 6115: COMPUTER SYSTEMS PERFORMANCE MEASUREMENT AND EVALUATION

1
ICOM 6115 COMPUTER SYSTEMS PERFORMANCE
MEASUREMENT AND EVALUATION

• Nayda G. Santiago
• August 18, 2006

2
Administrative Stuff

• Quiz 1
• August 23, 2006, Wednesday
• WebCT if activated
• Article Designing computer architecture research
  workloads
• Students who do not have cards to access lab
• Cesar and Idalys

3
Back to performance measurements

• How to measure software performance on a system?

4
Experiment

• In order to analyze the performance of an
  application, one has to conduct one or more
  experiments to find out about its possible
  performance problems.

5
General Structure of Measurement

• Typical execution of an application program

Source repository
Output Data
Repository Control system
Input Data
Application Process
Source Code
Compilation System
Executor/ Launcher
Executable
6
Typical execution of an application program

• Typical execution of an application program
• The source code of the application is either
  available directly or through a repository
  control system like RCS or CVS which also allows
  to store and access multiple versions versions of
  the application's code at the same time in a
  so-called source repository.
• A compilation system is used to produce an
  executable.
• The compilation system typically consists of
  several phases including preprocessors, parsers,
  optimizers, assemblers, linkers, and runtime
  libraries. Compilation systems for high-level
  parallel languages like HPF often generate output
  in another language like Fortran or C in a first
  step and then use compiler systems for these
  languages to produce an executable.

7
Typical execution of an application program

• A launcher / executor takes the executable and
  suitable input data and executes it as one ore
  more application processes on a computer system.
• Launching an application can be done
  interactively (e.g. through a UNIX shell program
  or a program manager) or through a batch system
  under the control of a command script. Typically,
  the program generates some output data containing
  the results of the computation.

8
Instrumentation

• In order to determine the performance of an
  application, the application has to be modified
  in order to insert code which collects
  information about the performance of the program.
  This process is called instrumentation. This can
  be done in several ways
• A source code instrumentor modifies the original
  source code to add the necessary measurement
  instructions. It is typically implemented as a
  source-to-source translation tool which runs as a
  preprocessor before the actual compilation or it
  is handled by the compilation system itself.
• An object code instrumentor modifies object code
  or executables to accomplish the same task.
• An dynamic instrumentor also modifies executables
  but does this during the execution of the
  program. It has the great advantage that
  instrumentation can be changed / adapted during
  the execution.

9
Measurement Experiment Setup
10
Selecting Performance Metrics
response time n. An unbounded, random variable
representing the elapses between the time of
sending a message and the time when the error
diagnostic is received. S. Kelly-Bootle, The
Devils DP Dictionary
Time
Speed
Rate
Request
Resource
Correct
Done
Probability
System
Not Correct
Errori
Reliability
Time between
Not Done
Eventk
Duration
Availability
Time between
11
Selecting Performance Metrics

• Mean is what usually matters
• But variance for some (ex response time)
• Individual vs. Global
• May be at odds
• Increase individual may decrease global
• Ex response time at the cost of throughput
• Increase global may not be most fair
• Ex throughput of cross traffic
• Performance optimizations of bottleneck have most
  impact
• Example Response time of Web request
• Client processing 1s, Latency 500ms, Server
  processing 10s ? Total is 11.5 s
• Improve client 50? ? 11 s
• Improve server 50? ? 6.5 s

12
Selecting Performance Metrics

• May be more than one set of metrics
• Resources Queue size, CPU Utilization, Memory
  Use
• Criteria for selecting subset, choose
• Low variability need fewer repetitions
• Non redundancy dont use 2 if 1 will do
• ex queue size and delay may provide identical
  information
• Completeness should capture tradeoffs
• ex one disk may be faster but may return more
  errors so add reliability measure

13
Commonly Used Performance Metrics

• Response Time
• Turn around time
• Reaction time
• Stretch factor
• Throughput
• Operations/second
• Capacity
• Efficiency
• Utilization
• Reliability
• Uptime
• MTTF

14
Response Time

• Interval between users request and system
  response

• But simplistic since requests and responses are
  not instantaneous
• Users type and system formats

15
Response Time
System Starts Response
System Starts Execution
User Finishes Request
User Starts Request
System Finishes Response
Time
Reaction Time
Think Time
Response Time 1
user starts next request
Response Time 2

• Can have two measures of response time
• Both ok, but 2 preferred if execution long
• Think time can determine system load

16
Response Time

• Turnaround time time between submission of a
  job and completion of output
• For batch job systems
• Reaction time - Time between submission of a
  request and beginning of execution
• Usually need to measure inside system since
  nothing externally visible
• Stretch factor ratio of response time at load
  to response time at minimal load
• Most systems have higher response time as load
  increases

17
Throughput

• Rate at which requests can be serviced by system
  (requests per unit time)
• Batch jobs per second
• Interactive requests per second
• CPUs
• Millions of Instructions Per Second (MIPS)
• Millions of Floating-Point Ops per Sec (MFLOPS)
• Networks pkts per second or bits per second
• Transactions processing Transactions Per Second
  (TPS)

18
Throughput

• Throughput increases as load increases, to a point

• Nominal capacity is ideal (ex 10 Mbps)
• Usable capacity is achievable (ex 9.8 Mbps)
• Knee is where response time goes up rapidly for
  small increase in throughput

19
Efficiency

• Ratio of maximum achievable throughput (ex 9.8
  Mbps) to nominal capacity (ex 10 Mbps) ? 98
• For multiprocessor, ration of n-processor to that
  of one-processor (in MIPS or MFLOPS)

20
Utilization

• Typically, fraction of time resource is busy
  serving requests
• Time not being used is idle time
• System managers often want to balance resources
  to have same utilization
• Ex equal load on CPUs
• But may not be possible. Ex CPU when I/O is
  bottleneck
• May not be time
• Processors busy / total makes sense
• Memory fraction used / total makes sense

21
Miscellaneous Metrics

• Reliability
• Probability of errors or mean time between errors
  (error-free seconds)
• Availability
• Fraction of time system is available to service
  requests (fraction not available is downtime)
• Mean Time To Failure (MTTF) is mean uptime
• Useful, since availability high (downtime small)
  may still be frequent and no good for long
  request
• Cost/Performance ratio
• Total cost / Throughput, for comparing 2 systems
• Ex For Transaction Processing system may want
  Dollars / TPS

22
Utility Classification

• HB Higher is better (ex throughput)
• LB - Lower is better (ex response time)
• NB Nominal is best (ex utilization)

23
Setting Performance Requirements

• The system should be both processing and memory
  efficient. It should not create excessive
  overhead
• There should be an extremely low probability that
  the network will duplicate a packet, deliver it
  to a wrong destination, or change the data
• Whats wrong?

24
Setting Performance Requirements

• General Problems
• Nonspecific no numbers. Only qualitative words
  (rare, low, high, extremely small)
• Nonmeasureable no way to measure and verify
  system meets requirements
• Nonacceptable numbers based on what sounds
  good, but once setup system not good enough
• Nonrealizable numbers based on what sounds
  good, but once started are too high
• Nonthorough no attempt made to specify all
  outcomes