Software Performance Engineering SPE

1
Software Performance Engineering (SPE)

• CMPT 318Fall 2006SFU Surrey

2
Introduction

• Software must be designed before it is
  implemented
• So far, the performance techniques weve seen can
  only be applied to implemented software
• E.g. timing and profiling only work on
  implemented software

3
Introduction

• Many applications have performance goals
• Websites
• Embedded systems
• Real-time software
• Waiting until the software is implemented could
  be too late to do anything about performance
• Thus, for some projects it is important that
  performance be built-in from the start

4
Examples of Performance Failures

• The NASA Flight Operations Satellite was delayed
  8-months due to slow scheduling software
• A live Victorias Secret webcast was so popular
  (due to Super Bowl advertising) that 5 of
  viewers couldnt see the webcast

5
Examples of Performance Failures

• A Fortune 100 company tried to create a new
  distributed order management system based on a
  number of legacy systems, but found that there
  were performance problems that no amount of extra
  hardware or system tuning could overcome

6
Why Performance Problems Occur

• No consideration given to performance in
  requirements/design phase
• Just buy faster hardware and more computers.
• Knee-jerk fix-it later mentality
• Premature optimization is the root of all evil.
  -- Donald Knuth
• They key word is premature --- the quote does
  not discourage optimization, just premature
  optimization
• In performance critical systems, optimization
  should be taken into consideration from the start

7
Three Performance Myths

• It is not possible to do anything about
  performance until you have something to execute.
• Performance models can be used before a system is
  implemented to estimate its performance and find
  bottlenecks

8
Three Performance Myths

• Managing performance takes too much time.
• Only performance-critical system need to do
  extensive performance management.
• If the risk of project failure due to low
  performance is high, then only then is
  performance management needed.

9
Three Performance Myths

• Performance models are complex and expensive to
  construct.
• Simple and inexpensive performance models can be
  constructed.
• Not necessary to create a detailed performance
  simulation.

10
Performance Issues

• The approach we will look at is called Software
  Performance Engineering (SPE)
• SPE is concerned with two major performance
  issues
• Responsiveness/throughput
• Scaling

11
Responsiveness

• Response time is the time required to respond to
  a request
• Could be a single transaction
• Or the time to complete a longer task
• Throughput of the system
• Typically defined with respect to user

12
Responsiveness

• Objective response time
• Subjective response time
• Does the user feel that the system is responsive?
• E.g. outdoor ATM user in Hawaii may tolerate a
  30-second response time, but not an outdoor ATM
  user in winter in Toronto
• E.g. putting input forms on-screen first may make
  users feel the system is more responsive, even if
  the true response time is unchanged

13
Scalability

• Scalability is how the system performs when
  demand for it increases

Response Time
knee
Throughput
14
Scalability

• Before the knee linear scaling
• After the knee exponential scaling
• If the knee is before your target load
  requirements, you have a performance problem

15
SPE Process

• SPE is a methodology
• Thus it comes with various rules, instructions,
  and patterns for users of it to follow
• Plus tools and advice you can purchase

16
SPE Modeling Strategies

• SPE supports three kinds of models
• Simple
• Best- and Worst- Case
• Adapt to precision

17
Simple Model

• Start with the simplest possible model that
  identifies performance problems
• Good for rapid feedback in early design phases

18
Best- and Worst- Case

• Use best- and worst-case estimates of resource
  requirements to establish bounds on expected
  performance
• Useful when there is much uncertainty
• If worst-case estimates are satisfactory, then
  you can move on

19
Adapt to Precision

• Match model detail to knowledge of software
  processing
• As a project progresses, you can add more detail
  to your performance models

20
SPE Process

• Assess performance risk
• Identify critical use cases
• Select key performance scenarios
• Establish performance objectives
• Construct performance model(s)
• Add software/hardware resource requirements
• Evaluate model
• If the results are feasible, modify the software,
  re-create scenarios, and goto step 5 otherwise,
  revise performance objectives

21
Example ATM Machine

• Suppose you are creating the software for an ATM
  machine
• The ATM accepts a bank card, requests a PIN, and
  the lets the user withdraw cash, deposit cash, or
  see their balance the customer may perform
  multiple transactions per session.
• The ATM communicates with a host computer to
  verify the PIN and transaction

22
Example ATM Machine

• Step 1 Assess performance risk
• The risk is small since only one customer uses it
  at a time
• (the performance risk of the host computer is
  much higher but we are not worrying about the
  host here)

23
Example ATM Machine

• Step 2 Identify critical use cases
• Three uses cases
• Operator transaction (e.g. change currency file)
• Command function (e.g. reset ATM)
• User transaction
• Clearly, the user transaction is the critical one
  for performance

24
Example ATM Machine

• Step 3 select key performance scenarios
• An error-free customer transaction is the key
  scenario
• In a single transaction, the user could perform
  multiple actions
• Need to get (or estimate) probabilities that
  various actions will be performed

25
Example ATM Machine

• Step 4 establish performance objectives
• How long is a reasonable ATM session?
• SPE prefers to consider the entire end-to-end
  transaction, as opposed to the response time of
  individual functions
• Lets say that 30s is a reasonable time

26
Example ATM Machine
getCardInfo
getPIN
UML activity graph
n
process transaction
terminate transaction
27
Example ATM Machine
getTransaction
UML activity graph
28
Example ATM Machine

• Step 5 hardware/software requirements
• Software resources include
• Number of screens
• Number of host interactions
• Number of log entries
• Delays due to things like the cash dispenser and
  receipt printer
• For each process on the execution graph, required
  amount of each of these resources is listed

29
Example ATM Machine

• Hardware resources includes
• CPUs
• Disks
• Displays
• Network cards
• Now we can create a software-by-hardware matrix,
  and estimate the time each part of a transaction
  will take
• This can then be used to solve the model, i.e.
  determine if the model predicts high-enough
  performance
• This particular model is very simple --- but then
  the performance risk is small, and so doesnt
  need an elaborate model