Automated and fine grained profiling of distributed transactions

1
Automated and fine grained profiling of
distributed transactions

• B Nagaprabhanjan (03329010)
• Under the guidance of
• Prof. Varsha Apte

Project sponsored by Intel corporation
2
Motivation
A typical distributed system
Web Server
App Server 1
Client
Internet
LAN

• Requirements
• Satisfy user performance requirements.
• Should utilize the systems well.

Capacity Planning
App Server 2
Performance Analysis
3
Techniques

• Measurement based

• Modeling based

Resource Consumption Details
Exit
?
?
T
S2
S1
?
Message Flows
?
S3
4
Inputs
Resource Consumption profile
Web Server
App Server 1
20 ms
Client
Internet
10 ms
40 ms
LAN
105 ms
35 ms
App Server 2
5
Obtaining resource profiles manually
Profile App server1
Profile Web server
Client
Generate Load
Profile Servers
Httperf WAPT Silk Performer LoadRunner
Profile App server2
2
1
ps netstat vmstat top
Collect client statistics
3
Collect server statistics
Correlate display
4
6
Problems
Error Prone
Client
Generate Load
Profile Servers
2
1
Collect client statistics
Tedious
3
Collect server statistics
Correlate display
Accounting
Hence there is a need to automate this process
7
Autoprofiler
A tool, we developed, to automate the process of
resource consumption profiling
8
Autoprofiler
Servers
Client
Generate Load
Profile Servers
2
1
Autoprofiler
Collect client statistics
3
Collect server statistics
Correlate display
9
Autoprofiler - Architecture
S
Servers
S
M
Client
Generate Load
S
Profile Servers
S
2
1
Autoprofiler

• Master Slave Architecture
• Slaves profiles the servers.
• Master co-ordinates the process.

Collect client statistics
3
Collect server statistics
Correlate display
10
Autoprofiler Functional Details
S
Trigger Profiling
Read Transaction Information
3
1
2
4
Profile resource usage
M
Client
S
Generate Load
5
6
Collect profiles
Process data and display the per
transaction resource profiles.
11
Transaction Specification (Master)
1
What is required?

• Web Interface for the transaction
• URL
• Port
• URI information
• Nodes Information
• Physical servers that are accessed by the
  transaction (IP or Domain Name).
• Software servers that serve the transaction.

12
Input in XML format (Master)
1
Nodes information
Web Interface
ltNodesInfogt ltjavaNodegt ltNodegt
10.129.112.175 lt/Nodegt ltProcessgt java
lt/Processgt ltComponentInfogt
ltComponentgt ltNamegt
com.sun.ecperf.webbeans.OrderSesBean
lt/Namegt ltInterfacegtgetOrderStatuslt/
Interfacegt lt/Componentgt
lt/ComponentInfogt lt/javaNodegt
lt/NodesInfogt ltNodesInfogt
ltnativeNodegt ltNodegtmetcalfe.it.iitb.ac
.inlt/Nodegt ltProcessgtpostmasterlt/Proces
sgt lt/nativeNodegt lt/NodesInfogt lt/Trans
Infogt lt/servicesgt
lt?xml version"1.0" encoding"UTF-8"?gt lt!DOCTYPE
services SYSTEM "../Definitions/services.dtd"gt ltse
rvicesgt ltTransInfogt ltNamegtOrder
Statuslt/Namegt ltWebInterfacegt
lturlgt10.129.112.175lt/urlgt
ltportgt8080lt/portgt lturiInfoFilegtorderstatus
.urislt/uriInfoFilegt lt/WebInterfacegt
13
Request Generation (Master)
2
Detecting Steady state

• Uses the technique of moving average.
• Calculate the average of the values in the
  moving window of size w (w 5).
• If the normalized difference between the two
  averages is less than a given e, we conclude
  that the server has warmed-up.

Avg 1
r1
r2
r3
r4
r5
r6
r7
r8
Avg 2
14
Trigger profiling (Master)
3

• After detecting the steady state, master sends
  signals to slaves to start profiling.

15
Profiling with OS tools (Slave)
4
ps
Start profiling
Slave
vmstat
Send profiling data
netstat
Interacts with Operating system tools to get
resource profiles.
16
Collect profile data (Master)
5

• After generating the load, master signals slaves
  to send the profile data.

17
Output (Master)
6
Sample output as a graph
18
Need for fine grained profiling
Help construct better queueing models.

• Operating System level tools
• Linux Trace Toolkit (LTT)

Our focus was on Java based application
environments.
All Operating system tools including LTT fail to
profile Java based applications
19
Need for Java based profilers
Application
We want the application to be profiled
(ECPerf)
Application server
(JBoss)
JVM
OS tools profile JVM
OS
Hence we need Java based profilers
20
JVMPI

• Profiler agent can catch various events such as
  method entry and exit, object allocation, class
  load and unload.

21
Existing profilers
Drawbacks
Commonly available profilers

• Very large log files generated when deployed
  with J2EE servers.
• Huge measurement overhead because of enabling
  all events.
• Lack of dynamic filtering support
• Not suitable for automation

EJP YJP JProbe JProfiler hprof
Hence we developed our own profiler LiteJava
Profiler (LJP)
22
LJP - Features
Partial profiling support Can enable and disable
profiling when needed.
Remotely controllable Suitable for automation
Garbage Collection Reports information about
garbage collection
Dynamic Filtering Can set filters without
restarting the application server
No File I/O Avoids unnecessary overhead
23
Integrating LJP with Autoprofiler
ps
Start profiling
Slave
vmstat
Send profiling data
netstat
Component info
Results
LJP
JVM
24
Overall Picture
Native node
Slave
OS tools
Client
Master
Java node
Slave
OS tools
LJP
JVM
25
Experiments
ECPerf Application
JBoss Application Server
Client
PostgreSQL Database Server

• Get the status of order
• Get the status of customer
• Cancel order

26
Input
Nodes information
ltNodesInfogt ltjavaNodegt ltNodegt
10.129.112.175 lt/Nodegt ltProcessgt java
lt/Processgt ltComponentInfogt
ltComponentgt ltNamegt
com.sun.ecperf.webbeans.OrderSesBean
lt/Namegt ltInterfacegtgetOrderStatuslt/
Interfacegt lt/Componentgt
lt/ComponentInfogt lt/javaNodegt
lt/NodesInfogt ltNodesInfogt
ltnativeNodegt ltNodegtmetcalfe.it.iitb.ac
.inlt/Nodegt ltProcessgtpostmasterlt/Proces
sgt lt/nativeNodegt lt/NodesInfogt lt/Trans
Infogt lt/servicesgt
Web Interface
lt?xml version"1.0" encoding"UTF-8"?gt lt!DOCTYPE
services SYSTEM "../Definitions/services.dtd"gt ltse
rvicesgt ltTransInfogt ltNamegtOrder
Statuslt/Namegt ltWebInterfacegt
lturlgt10.129.112.175lt/urlgt
ltportgt8080lt/portgt lturiInfoFilegtorderstatus
.urislt/uriInfoFilegt lt/WebInterfacegt
27
Results
Transaction Response times
Server Profiles
28
Results (Contd)
The entire set of data items(8 set of measures of
7 transactions on 2 servers) were generated in
less than 15 minutes while it took hours to get
the same set of values manually.
29
Conclusion

• Ease of experimentation
• Eases the process of measurements and hence
  reduces the chances of errors.
• Enhances productivity
• Produces results in less time.
• A tool with a new paradigm
• For performance modeling and prediction, not
  just for performance testing.
• Can reveal valuable insights

30
Summary of Work Done

• Extensive survey of load generators, resource
  profiling tools, and survey of existing Java
  profilers
• Developed the Autoprofiler framework
• Developed the LiteJava Profiler
• Found that LTT is not suitable for our purpose.
• Learnt EJB concepts and deployed ECPerf on JBoss
  server
• Discovered by intensive experimentation that
  JVMPI was buggy till JDK1.3

Submitted paper to ICDCIT 2005
31
Future Work

• Extending the tool
• Adding innumerable number of features to make it
  more realistic.
• Automating the capacity Analysis
• Making the tool more intelligent.
• Can be made to work in a production
  (uncontrolled) environment.

32
Thank You!
33
LJP - Efficiency
Canonical Method Names
Method IDs
Information retrieval at run time is done in
constant time.
Method IDs
Profile Structure
Method IDs
True/False