A Framework for Workload Generation

1
A Framework for Workload Generation

• Archana Ganapathi
• Armando Fox, Dave Patterson

2
A Case for Workload Generation

• No uniform methodology for workload generation
• Need tools to predict scaling issues during
  develop/deploy phase
• Obstacles for industry to share data

3
State of the art Workload Generators

• SURGE Scalable URL Ref generator, captures file
  sz/req sz distrib, relative popularity, think
  times
• SPECweb Caters to web servers http requests
  only, hard to configure, only captures 200 OK
  response, distribution is different from traces
  especially at high numbers, doesnt handle
  dynamic content
• TPC-W Online bookstore, Webserving/browsing/shopp
  ing cart etc, high set up overhead
• SLAMD Java-based, tests network-based apps
  (specifically LDAP directory servers), Also used
  for Web servers and Web-based apps, relational
  databases, and mail servers
• Harpoon A Flow-level Traffic Generator, mimics
  internet traffic, generate representative
  background traffic for app/protocol testing
• Optixia hardware-based, IP Performance Test
  platform, create and transmit any type of Layer
  2-3 traffic patterns at up to line rate over a
  network
• Hammer hardware-based VoIP and PSTN telephone
  call generation

4
State of the art Workload Generators - Comparison

• Hardware vs. software based
• Hammer, Optixia vs SURGE, SLAMd
• Tunability vs Automation
• SPECweb, TPC-W, Harpoon vs Optixia, SLAMd
• Realistic vs Synthetic
• SURGE, SLAMd, Harpoon vs TPC-W
• Generic vs App-Specific
• SLAMd, Harpoon vs TPC-W, Hammer
• Open-loop vs Closed-loop
• Partly-open loop is most realistic for web
  services

5
Goals for our Framework

• Generic to accommodate existing workload
  generators
• Re-configurable to allow black-box testing and
  targeted testing
• Address privacy concerns

6
Block Diagram
Quantity
Quality
Request Type
Response Awareness

• granularities
• num users/req
• distribution
• burstiness
• other metrics

• per user/request
• math models
• traces

• std protocols
• http/ftp..
• examples
• traces
• hard-coded

• msg header
• sender
• type
• msg body
• objects
• pattern match

App-level
Code-gen
RAMP
Target System
Source Code
Workload Generator

• modules
• branches
• computation units

• time to generate request
• time to parse response

Coverage Statistics
Performance Metrics
7
Understanding Workload

• Workload has static and dynamic features
• Static features - Properties inherent in system
• File size
• Response type
• Dynamic features Properties based on user
  behavior/system runtime effect
• Response time/inter-arrival rate
• Request type distribution

8
Formally speaking

• Workload set of equivalence classes
• Wstatic Wdynamic
• Equivalence class transactions,
  distributions etc.
• Wstatic cluster centroidi, cluster radiusi
  where 1 i N
• N num equivalence classes
• Metrics set of feature vectors
• Cluster set of related metrics given
  pair-wise distance and clustering algorithm
• Wdynamic NxN transition probability matrix
• Dependent on real traces and Wstatic

9
Putting it all together
Traces
Wstatic
Wdynamic
Metrics
clustering
parse traces and scale Wstatic
Firewall
System Under Test
Workload Generator
Workload Model (open, closed, ajax etc.)
10
Validation

• Create models using real traces
• Scale up workload by generating synthetic model
• Compare behavior of system under trace-based and
  synthetic workloads

11
Addressing Privacy

• Industry can generate workload model and provide
  us with a digest of info
• Anonymized clusters
• Number and distribution of equivalence classes
• Dont need to know what each equivalence class is
  (and types of transactions)
• No user traces are revealed

12
Other things to consider

• How to generically characterize resource demands
  on workload
• Manifestation of workload on system
• Normal operation
• Saturation point
• Temporal variation (time of day/week)

13
Comments/Feedback
14
Comparison of Generators