J2EE: Black Box in the Oracle World

1
J2EE Black Box in the Oracle World
Don Bergal, Confio Software August 17, 2006
2
Objective Answer a Question

• Why does a DBA need visibility upstream in the
  application in order to manage the database?

3
An alternate title for today

• How to be a better DBA by watching something
  other than the Oracle database

4
Agenda

• J2EE Black Box in an Oracle World
• a) The Black Box Application Problem
• b) The DBA Hot Seat
• c) System Statistics versus Wait Time
  Diagnostics
• d) A Better J2EE End-To-End Performance
  Solution
• e) Best Practices Response Time Diagnostics
  Methodology
• f) Q A

5

• The Black Box Application Problem

6
J2EE Black Box in Oracle World
How can you manage here?
If you cannot see here!
Web store
Web customer
Black Box Applications and End Users
Billing
Internal user
Batch job
Financial System
7
See the applications that impact database
performance
Web store
Web customer
Billing
Internal user
Batch job
Financial System
Database
App Servers
Web Users
8
Result Disconnect Between IT Groups?
DBAs
Databases

• Database Management
• Database Development
• SQL Development
• SQL Tuning Optimization Tools
• Ratio Based Monitoring Tuning

No Visibility into the Database Tier
No Visibility into the J2EE Tier
9
Effect Black Boxes leave DBA defenseless

• Silo-based, Single System tools only shows
  database system perspective (even though they are
  blamed for problem in any system). Cant tell
  when to hand-off problem to another group
  (Dev/SysAdmin/etc)

Black Box
Black Box
Blamed for all layers, but can only see the
database
10

• The DBA Hot Seat

11
Finger Pointing Happens When You Cant See the
Problem
Whats wrong with your !_at_ database?
Its a database issue!
Java/Web Developer
DBA
Management
Guilty until proven innocent
DBA Hot Seat
Network Engineers
Software Vendor
Its a database issue!
Its a configuration issue!
12
Do Customers See the Problem Before You?
Warning!
SLA Violation
User Wait Time
Not my code
Java Developer
DBA
System Admin
Not my Network, Servers, Load Balancer, Firewall
What performance problems ?!?
13
Hot Seat comes from UNCLEAR Ownership
Its an Application issue!
Its a Database issue!
Java App Owner
DBA
Need evidence to show where the problem exists,
avoid Finger Pointing to the database
14
Performance Suspects Line Up
D B A
System Admin
Developer
Networking
15
Lack of Visibility is Source of Problem
Typical Multi-Tiered J2EE System Leaves the DBA
Blind
Service Levels Measured Here
Web page request Apache WebLogic WebSphere IIS
App Layer Oracle eBusiness Oracle 9iAS
Peoplesoft Siebel JD Edwards SAP Custom apps
Blame assigned Here
J2EE layer
Oracle database
Web Server
16
Why Does the DBA Need J2EE Visibility?
You cant control what you cant see You cant
show/demonstrate its not a DB Issue You cant
manage your database if you are blind to the
source of the requests
17

• Traditional Performance Tools versus Wait-Time
  Diagnostics

18
Current Tools Never Cross the Silo Boundaries
Database
J2EE Server
Web Server
CPU
CPU
CPU
Sessions
Threads
Processes
Memory
Heap Size
Reads

• Tune inside each Silo to improve end-end
  performance
• System Health or ratio-based approach to
  performance
• Statistics UtilizationEvent Counts
• No direct relationship to End-User experience or
  SLA
• Weak linkage between tiers limits collaboration

19
Standard J2EE App Performance Monitoring How
Does This Help Identify User Response?
Idle threads Number of requests Throughput Waiting
Requests Memory (Heap) Usage
20
J2EE Profilers High Load, Test Data Only
21
OS Level System Monitoring (rstatd/perfmon)
22
Network System Monitoring
23
Traditional Tools vs. Wait-Time Methodology

• Use Wait Time to correlate user requests to SQLs
  and performance impact

• Ratio-Base traditional tools show Oracle system
  health but no link to end user service

24

• A Better J2EE End-To-End Performance Solution

25
Requirements for Oracle J2EE Monitoring

• Use Oracle Wait-Time Based Monitoring
• Visibility across Java and Oracle layers
• Lightweight JVM Method Tracker
• Agnostic JVM server support
• Fast installation and data collection techniques
• Cost Effective
• EASE OF USE MAKE IT OBVIOUS!

26
Wait-Time Based Performance Analysis

• Tune by determining where client request time is
  spent
• Determine where Oracle resources are spent during
  the execution of SQL execution
• Emerging best-practice for database tuning
• You cant tell how long something took by
  counting how many times it happened. If youre
  only measuring event counts, then youre not
  measuring what the users care about.
  Optimizing Oracle Performance, OReilly Press
• Oracle is building up Wait-based tuning
  capabilities with each database release10g has
  over 800 wait-time based performance counters

27
Oracle Wait Interface Book
Free book at www.confio.com
28
A Wait-Time J2EE Performance Methodology 3 Key
Requirements

• Follow every transaction individually
• Identify every step along the path
• Measure the wait-time at each step
• Result Minimize time to resolution and division
  of work by clearly identifying which step is
  performing poorly.

29
Alternative Monitoring Architectures
30
Byte Code Instrumentation (BCI)
? Best of breed Java monitoring technique ?
Production Monitoring not synthetic
transactions ? Instrumentation inserted into
selected compiled Java methods by choosing
specific class files ? Measure start/stop to
execute each Method ? Calibration Adjust load
to lt 1 load by controlling level of
instrumentation ? Auto Throttling, Low risk
31

• Best Practices Wait Time Diagnostics Methodology
• (Day in the life on the performance team)

32
4 Steps for J2EE to Oracle Visibility

• Capture Every User Request (end-to-end
  transaction)
• Capture Every Step Inside Each Tier
• Measure Response Time of Each Step
• Identify Possible Causes Based on Wait-Times

Since every step is visible, you can always see
the problem
33
4 Steps for J2EE to Oracle Visibility

• Capture Every User Request (end-to-end
  transaction)
• Capture Every Step Inside Each Tier
• Measure Response Time of Each Step
• Identify Possible Causes Based on Wait-Times

Since every step is visible, you can always see
the problem
34
1. Capture Every User Request
Identify the slowest performing end-to-end
Requests

• Slowest performing can be
• Longest average request-response time
• Largest accumulated Wait-Time in application
  (across all sessions)

35
1. Capture Every User Request
PlaceOrder.jsp Request has most serious delays
36
2. Capture Every Step Inside Each Layer
Identify the slowest steps within the slowest
Request
37
Key Requirement Correlate Application Response
to SQL Response Times
Question Which SQL statements are causing Wait
Time for my application?
38
2. Capture Every Step, Identify Slowest
UpdateOrder Total (SQL) is slowest in
PlaceOrder.jsp (Most Wait-Time accumulated)
39
3. Measure Response Time of Each Step
Focus on steps that consume the most time
40
Detailed Response Time for the Critical Step

• Update Order Total
• Undo Segment Extension Wait-Event is
  bottleneck

41
4. Possible Causes Based on Wait Times
Diagnose the step with the greatest Wait Time
42
4. Possible Causes Based on Wait Times
Execution path and time spent on each Hot Spot
43
Identify Chain of Calls

• URL Request -gt Method Call gt SQL
• Allocate Wait Time to each Method or SQL Call
• Profiler data Without the System Impact

44
Recall 3 Key Requirements to Provide the Analysis
1. Identify individual Request
2. Identify Every step Java method or Wait-Event
3. Measure Wait time for Every Step
45
Customer Example

• Retail transaction system
• Online media fulfillment
• Customer requests running slowly, unknown reason
• Ignite for Oracle and Ignite for Java used to
  examine situation

46
Wait Time for Authorize Transaction Identifies
Problem Steps
47
Associates Request Wait-Time with Single Procedure

• Same query multiple times in single procedure
• Same row retrieved 7 times
• Change to select once, cut Wait-Time to less than
  half

48
Summary Look For

• Continuous monitoring 24/7 not session/period
  specific
• Production data not synthetic test
  transactions
• Correlate Application Oracle where most of
  the time is spent
• Individual transactions, not system-wide
  statistics

49
Visibility for the DBA

• DBA has a need to see the J2EE Application
• Solve database problems by understanding the
  other systems
• Look for Wait Time thats where the action is
• Achieve collaboration between Developers,
  Architects and DBAs
• DBA can be the leader, not the one to take the
  blame!

50
About Confio Software

• Wait-Time based performance tools for continuous,
  low impact monitoring
• Ignite for Oracle Deep, detailed visibility
  into SQLs and Wait-Events
• Ignite for Java End-to-End Application through
  Oracle visibility, correlates Java and SQL
  Wait-Time
• Ignite for SQL Server Coming soon, customer
  testing
• Free Trial at www.confio.com

51
About the Author

• Don Bergal, Confio Software
• Responsible for product management, among other
  things
• 18 years software/telecom industry experience
  donbergal_at_confio.com

52
Q A