Where to start

1
(No Transcript)
2
Where to start

• moodle.epfl.ch
• Enrol for this course at moodlehttp//moodle.epf
  l.ch/course/view.php?id228
• Enrolment key PerfEval (Notice the Capitals)
• Or http//perfeval.epfl.ch

3
Learning Lecture Homeworks Miniproject

• Lectures Thu 810, Fri 1415 to 1600 (exceptions
  see web site)
• Labs Fri 1615 to 1800 (exceptions see web site)
• 5 homeworks, defined on web page
• Scientific method, simulations, statistics, lab
  measurements, forecasting contest
• to be completed for the due date indicated on
  web page
• Graded
• Mini-project
• Define on half a page your own performance
  evaluation problem or issue (based on your own
  semester, masters of phd project)
• Can be (for example)
• Simulation study
• Analysis
• Measurement study
• Your proposal (one paragraph) due April 3rd if
  no idea, tell us before March 27
• Final results presented in class May 29

4
Grading

• One oral exam E
• Average of 5 homeworks H
• Grade ( E H M)/3
• I will ask you one question about your homework
  or mini-project during the oral exam

5
Lecture Notes

• Document home pagehttp//perfeval.epfl.ch
• Lecture Notes
• http//perfeval.epfl.ch/lectureNotes.htm

6
How to Use the Lecture Notes

• Main points discussed at lecture
• Read the rest
• Solve inline questions and review questions
• Still work in progress, pls give me feedback

7
References

• Most are cited in lecture note Find them and
  more on the web site http//perfeval.epfl.ch/doc
  uments.htm

8
Outline of the Lecture

• Part I The Basics Labs
• 1. Methodology (today) L1
• 2. Confidence Intervals (next lectures)
• 3. Simulation (next lectures) L2
• 4. Model fitting
• 5. Queuing L3
• Part II Selected Topics
• Tests
• Load Generation L4
• Predictions L5
• Palm Calculus
• Fluid Modelling
• Patterns

9
Methodology

• easy but important

10
ToC

• What is performance evaluation about ?
• Metrics, Load and System
• Hidden Factors
• Patterns
• Be Scientific

11
What is Performance Evaluation ?

• Quantify the service offered by a system of
  computers or communication equipment
• Examples
• Response time for a user of web site
• Compare compilers
• Power consumption of a web server farm

12
Know your goals

• A1 and A3 are comparisons, A2 is an absolute
  statement
• E2 is an engineering rule

13
What is Performance Evaluation ?
14
2. Metric, Load and System

• Define a metric examples
• Response time
• Power consumption
• Throughput
• Define operational conditions under which metric
  is measured ( Viewpoint , see Chapter 11)

15
Load

• You need to define the load under which your
  system operates
• Make the difference between
• Intensity of the load (e.g. nb jobs per second)
• Nature of the load
• Statistical details that may matter e.g. job
  sizes are heavy tailed or not
• Benchmarks are artificial load generators we
  will play with one of them

16
Is load identification required ?
17
Compare Windows vs Linux
18
Syscall Benchmark
19
Memory Access Time
20
Ghostscript
21
3. Hidden Factors

• Factor an element that may impact the
  performance
• (desired factors) intensity of load, number of
  servers
• (nuisance factors) time of the day, presence of
  denial of service attack

22
TCP Throughput Increases with Mobility
23
TCP Throughput Decreases with Mobility
24
Why were we fooled ?

• Hidden factor had a more important role than the
  factor we were interested in
• We interpreted correlation as causality
• Need to be aware of all factors and incorporate
  in the analysis
• Or randomize experiment to reduce impact of
  hidden factors

25
Simpsons Paradox

• A well known phenomenon -- Special case of Hidden
  Factor paradox when metric is success rate and
  factors are discrete

26
Berkeley Sex Case 1973 (source wikipedia)
27
Take Home Message

• Pitfall number 1 is the presence of hidden factor
• Any study is susceptible to it
• Easy for opponents to find

28
4. Patterns

• There are common traits to be found in different
  situations
• Bottlenecks
• Other (Chapter 9)

29
(No Transcript)
30
Bottlenecks are Your Friends
31
Behind a Bottleneck May Hide Another Bottleneck
32
Other Patterns
33
5. Be Scientific

• Joe measures performance of his Wireless Shop
• what would you conclude ?

34
Scientific Method

• Joe buys 2 more Access Points
• improvement ?

Before
After
35
Scientific Method

• A conclusion can only proven to be wrong
• Do not draw conclusions unless the experiment was
  designed to test the statement
• Measurement 1 suggested that the wireless network
  was congested, but the experiment was not
  designed to test this statement
• Joe should design an experiment to validateH1
  the wireless network is the bottleneck
• for example measure the number of collisions /
  packet loss
• result collision 1 conclusion H1 is not
  valid
• hypothesis H2 the server is saturated
• experiment measure memory utilization result ¼
  100

36
Performance After Doubling Server Memory
37
Example from Nitin Vaidya, Mobicom 2000 Tutorial,
slides 298-299
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
Use of Scientific Method

• Recognize a fact
• Pose a hypothesis
• Verify the hypothesis on simulations /
  measurements designed to test it

• TCP throughput may increase with mobility
• (1) Duration of link failure period is impacted
  by speed(2) It has a negative impact on TCP
  throughput
• Do more simulationsmeasure distrib of link
  failure period
• Verify (1) and (2)
• How ?

42
Is ATM-ABR better than ATM-UBR ?
43
Take Home Message

• You should not conclude from an experiment
  without trying to invalidate the conclusion
• (Popper, 1934) you should alternate between the
  roles of
• Proponent
• Adversary

44
Summary (1/2)
45
Summary (2/2)
46
Announcements

• http//research.nokia.com/locations/lausanne/stude
  ntcfp

47
Scheduling Issues

• Schedule conflicts on March 6 and 13 due to
  Highschool student visits
• Proposed changes
• Friday March 6 lecture at 1615-1800 INM
  201 lab at 1415-1600 INF 3
• Friday March 13 lecture at 1615-1800 INM
  201 no lab
• Monday March 16 lab at ?? TBA