Performance Evaluation

1
Performance Evaluation

• Operating Systems
• Fall 2002

2
Performance evaluation

• There are several approaches for implementing the
  same OS functionality
• Different scheduling algorithms
• Different memory management schemes
• Performance evaluation deals with the question
  how to compare wellness of different approaches
• Metrics, methods for evaluating metrics

3
Performance Metrics

• What is the performance metric for the sorting
  algorithms?

• Is something wrong with the following statement
• The complexity of my OS is O(n)?

• This statement is inherently flawed
• The reason OS is a reactive program

4
Performance metrics

• Response time
• Throughput
• Utilization
• Other metrics
• Mean Time Between Failures (MTBF)
• Supportable load

5
Response time

• The time interval between a users request and
  the system response
• Response time, reaction time, turnaround time,
  etc.
• Wellness criterion Being fast is good
• For the user waiting less
• For the system free to do other things

6
Throughput

• Number of jobs done per time unit
• Applications being run, files transferred, etc.
• Throughput and response time are interdependent
• Good response time usually comes on expense of
  reducing throughput

7
Throughput vs. Response Time
3 jobs with times T1, T22T1, T33T1
What is wrong with this picture?
8
Throughput vs. Response Time
The correct picture
Context switch
9
Utilization

• Percentage of time the system is busy doing
  servicing clients
• Important for expensive shared system
• Less important (if at all)
• for single user systems, for real time systems
• Utilization and response time are interrelated
• At very high utilization, response time grows
  exponentially

10
Performance evaluation methods

• Mathematical analysis
• Based on a rigorous mathematical model
• Simulation
• Simulate the system operation (usually only small
  parts thereof)
• Measurement
• Implement the system in full and measure its
  performance directly

11
Analysis Pros and Cons

• Provides the best insight into the effects of
  different parameters and their interaction
• Is it better to configure the system with one
  fast disk or with two slow disks?
• Can be done before the system is built and takes
  a short time
• Rarely accurate
• Depends on host of simplifying assumptions

12
Simulation Pros and Cons

• Flexibility full control of
• Simulation model, parameters,
• Level of detail
• Disk average seek time vs. acceleration and
  stabilization of the head
• Can be done before the system is built
• Simulation of a full system is infeasible
• Simulation of the system parts does not take
  everything into account

13
Measurements Pros and Cons

• The most convincing
• Effects of varying parameter values cannot (if at
  all) be easily isolated
• Often confused with random changes in the
  environment
• High cost
• Implement the system in full, buy hardware

14
The bottom line

• Simulation is the most widely used technique
• Combination of techniques
• Never trust the results produced by the single
  method
• Validate with another one
• E.g., simulation analysis, simulation
  measurements, etc.

15
Workload

• Workload is the sequence of things to do
• Sequence of jobs submitted to the system
• Arrival time, resources needed
• File system Sequence of I/O operations
• Number of bytes to access
• Workload is the input of the reactive system
• The system performance depends on the workload

16
Workload analysis

• Workload modeling
• Use past measurements to create a model
• E.g., fit them into a distribution
• Analysis, simulation, measurement
• Recorded workload
• Use past workload directly to drive evaluation
• Simulation, measurement

17
Statistical characterization

• Every workload item is sampled at random from a
  distribution
• Workload is characterized by the distribution
• E.g., take all possible job times and fit their
  to a distribution
• Typically, a lot of low values and a few high
  values
• There might be enough high values to make a
  difference

18
Exponential (Poisson) Distribution

• Memoryless
• Regardless of how long you have waited, you can
  expect to wait for an additional a seconds

19
Fat-tailed distribution

• The real life workloads frequently do not fit the
  exponential distribution
• Fat-tailed distributions

20
Pareto Distribution

• Mean is unbounded
• The more you wait, the more additional time you
  should expect to wait

21
Exponential vs. Pareto

• The mean additional time to wait is determined by
  the shape of the tail
• The fatter tail, the more additional time to wait
• For exp. the tail shape is the same regardless
  of how much we have waited alreadygt
• The mean additional time stays the same
• For Pareto The more we wait, the fatter tail
  becomes
• The more we wait, the more additional time we
  will wait

22
Exp. vs. Pareto Focus on tail
23
Queuing Systems
queue
Disk A
queue
Disk B
new jobs
finished jobs
CPU
queue

• Computing system can be viewed as a network of
  queues and servers

24
The role of randomness

• Arrival (departure) are random processes
• Deviations from the average are possible
• The deviation probabilities depend on the
  inter-arrival time distribution
• Randomness makes you wait in queue
• Each job takes exactly 100ms to complete
• If jobs arrive each 100ms exactly, utilization is
  100
• But what if both these values are on average?

25
Queuing analysis
server
queue
departing jobs
arriving jobs
26
Littles Law
27
How response time depends on utilization?

• Write the average number of jobs as a function of
  arrival and service rates
• Queuing analysis
• Substitute it to the Littles law

28
M/M/1 queue analysis
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
Response time (utilization)
34
Summary

• What are the three main performance evaluation
  metrics?
• What are the three main performance evaluation
  techniques?
• What is the most important thing for performance
  evaluation?
• Which workload models do you know?
• What does make you to wait in queue?
• How response time depends on utilization?

35
To read more

• Notes
• Stallings, Appendix A
• Raj Jain, The Art of Computer Performance Analysis

36
Next

• Processes