E139 Why Troubleshooting ? Avoid Problems Upfront !

1
E139Why Troubleshooting ? Avoid Problems Upfront
!

• Markus Ohly
• Sybase European CSS
• Markus.Ohly_at_sybase.com

2
Agenda (1/2)

• Introduction
• Have the Right Software in Place
• Have an eye on your Jaguar
• Write Components that ease your life
• Benchmarking
• Memory Usage
• Connection Caching

3
Agenda (2/2)

• Character Sets, Globalization
• C components and 3rd party libraries
• Fault Tolerance and Service Continuity
• Limitative Configuration Options
• Threading
• (BOOT)CLASSPATH

4
Agenda ...

• Introduction
• Have the Right Software in Place
• Have an eye on your Jaguar
• Write Components that ease your life
• Benchmarking
• Memory Usage
• Connection Caching

5
Introduction

• This presentation is not a classical
  troubleshooting session, it rather explains the
  known cliffs and how to avoid them.
• This will help you to avoid a lot of trouble
  before you get into it and will in turn avoid
• projects to stall
• production services to interrupt
• technicians and management to be upset and
• customers to shy away

6
Agenda ...

• Introduction
• Have the Right Software in Place
• Have an eye on your Jaguar
• Write Components that ease your life
• Benchmarking
• Memory Usage
• Connection Caching

7
Have the Right Software in Place

• Choose the right Edition of EAServer
• Developer Edition
• Small Business Edition
• Advanced Edition
• Enterprise Edition

8
Have the Right Software in Place

• Choose the right Edition of EAServer
• Earlier Versions had different limitations

9
Having the Right Software in Place

• Sybase continuously improves EAServer and
  releases updated versions. The latest are
• EAServer 3.5 3.5 C5
• EAServer 3.6.1 3.6.1 ESD 4
• EAServer 4 4.1.1
• This applies as well to third party software, eg.
• Java Virtual Machine
• Operating System and Patches
• DBMS and Drivers

10
Having YOUR Right Software in Place

• Repository Versioning
• Repository Versioning has been introduced with
  EAServer 4.0.
• It allows you to do changes to components and
  server configuration in a clean, explicit,
  traceable and undoable way.

11
Agenda ...

• Introduction
• Have the Right Software in Place
• Have an eye on your Jaguar
• Write Components that ease your life
• Benchmarking
• Memory Usage
• Connection Caching

12
Have an eye on your Jaguar

• Having an eye on your Jaguar will allow you to
  notice the very firsts signs of unnormal
  functioning
• This gives you time to react, intervene and keep
  trains on time
• Suggestions
• Reading log files with JaguarFileViewer
• Statistics about Components and Connections
• Third Party Tools
• Operating System Level Tools

13
Have an eye on your Jaguar

• JaguarFileViewer allows remote access to log
  files
• LOG_BEGIN 100
• 1. fileviewer.initialize(Jaguar)
• 2. int ids fileviewer.getFileIds()
• 3. String txt fileviewer.text(id, LOG_BEGIN,
  length)
• 4. fileviewer.destroy()

14
Have an eye on your Jaguar

• Jaguar Monitoring
• Number of components
• Number of connections

15
Have an eye on your Jaguar

• Jaguar Monitoring
• MONITOR_COMPONENT_ACTIVE
• MONITOR_COMPONENT_POOLED
• MONITOR_SESSION_IIOP
• MONITOR_SESSION_HTTP
• MONITOR_CONNCACHE_OPENED
• MONITOR_CONNCACHE_CLOSED
• MONITOR_CONNCACHE_FORCED

16
Have an eye on your Jaguar

• Jaguar Monitoring
• MONITOR_PEAK_MAXIMUM
• MONITOR_LAST_MAXIMUM

17
Have an eye on your Jaguar

• Your Operating System provides many useful tools,
  eg
• vmstat, mpstat
• NT Performance Monitor
• Third Party Tools do exist
• SNMP and JMDK interfaces exist since 4.0

18
Agenda ...

• Introduction
• Have the Right Software in Place
• Have an eye on your Jaguar
• Write Components that ease your life
• Benchmarking
• Memory Usage
• Connection Caching

19
Write Components that ease your Life

• Write Components that ease your life when things
  do not work as expected
• Connections to Databases can fail
• Running Commands on Databases may abort
• Avoid the neccessity to restart jaguar, allow
  reinitialization

20
Components that ease your Life (ctd)

• Error and Warning Messages
• Should be precise and useful for users,
  administrators, and engineers
• Choose to write warnings and errors into a
  separate log file
• Mind the cost of I/O
• Mind the cost of String concatenation
• Think about a configurable verbosity level

21
Components that ease your Life (ctd)

• The J2EE Reference Implementation includes an
  EJB Verifier that checks the beans if they
  respect the conditions set in the specification
• Components should have built-in methods to
  perform simple testing, eg. Resource References,
  EJB Links, Environment settings
• Nested try-catch will make it very difficult to
  find the root place where the error was first
  raised.

22
Components that ease your Life (ctd)

• Environment and Configuration Reports
• Admin/Debug Interface that allows to inspect the
  running system
• CORBA components only If the methods test() /
  debug() / control() are defined in a separate
  interface, you can get a generic stub which
  facilitates coding a lot.

23
Agenda ...

• Introduction
• Having the Right Software in Place
• Having an eye on your Jaguar
• Write Components that ease your life
• Benchmarking
• Memory Usage, Pool Management
• Connection Caching

24
Benchmarking Functionality

• More commonly referred to as Testing
• Programmatic verification of the functionality
  with given data and known results
• Either manual or automatic
• Performed under laboratory conditions

25
Benchmarking Functionality

• Embedded Test Functions inside the released
  product to check the correct settings for J2EE
  Applications after deployment, eg.
• Resource Links
• EJB Links
• Environment settings
• Connection Caches

26
Benchmarking Performance

• Record the average expected execution time of
  known functionality with known data
• idle
• under load
• Repeat this in production, you will then be able
  to see where your system is behind
• Com.sybase.jaguar.component.trace true

27
Agenda ...

• Introduction
• Have the Right Software in Place
• Have an eye on your Jaguar
• Write Components that ease your life
• Benchmarking
• Memory Usage
• Connection Caching

28
Memory Usage

• Caching of Objects
• Component Instance Pool Management
• Variable Scope method variables preferred over
  instance variables
• Home Interfaces
• Stateful vs. Stateless

29
Memory Usage

• Stateful vs. Stateless
• A stateful component has a 11 relation with its
  client
• A stateless component can be used by many clients
  in a sequence, thus less instances are needed.
• In case that the client fails to call remove()
  for a stateful instance, the instance will be
  kept until a timeout fires
• Instance Timeouts Server Default and
  per-Component Setting

30
Memory Usage

• Caching of Data
• Data Caches
• Keep results of queries in memory that are
  frequently needed but change rarely
• Optimally Changes are done only via Beans
• Files to be read from disk
• ResultSet
• Tools can help a lot JVM Profiling, OptimizeIt,
  Jprobe, Hpjmeter, jmeter

31
Memory Usage

• Runtime System Managed Memory
• Sometimes referred to as Garbage Collection
• Set variables holding references objects to NULL,
  especially in pooled Components such as Beans and
  Connections
• This requirement needs to be added to a Project
  Coding Style Guide

32
Memory Usage

• Operating System Configuration
• Windows NT processes can address up to 3 GB
  (NT/2000 Server Enterprise Edition)
• Thread Stack Size
• 1M by default
• The number of threads times the thread stack
  size must fit into the maximal process space size

33
Agenda ...

• Introduction
• Have the Right Software in Place
• Have an eye on your Jaguar
• Write Components that ease your life
• Benchmarking
• Memory Usage
• Connection Caching

34
Connection Caching

• The number of connections in the connection cache
  is a balance either the concurrency is managed
  in EAServer (few connections) or in the database
  (many connections, force connections)
• The Jaguar Connection Cache Manager allows
  CORBA-Components do specify explicitly how to
  behave if the configured number of connections is
  taken
• With JCMForce, only as many connections as
  configured on the database server can be obtained
  at this point the trouble is at two points

35
Connection Caching

• The options are implicit for EJB that have
  Resource References
• You could as well set the Sanity Checking query
  to a more elaborate query than select 1 in
  order to cover planned service outage.

36
Connection Caching

• You should foresee DB admin means to overcome db
  stall situations, such as Full segments or full
  transaction logs
• In ASE, sp_thresholdaction can be used to
  configure automatic action for full segments,
  especially the log segment.

37
Connection Caching

• ORB Properties for Connection Caching
• The ORB Connection Caching depends upon the
  setting of two options
• socketReuseLimit -- number of method invocations
  after which the socket is closed and a new one is
  created for the next invocation
• IdleConnectionTimeout -- time after which an
  unused (idle) connection will be discarded.
• Tradeoff between load balancing and performance

38
Agenda ...

• Character Sets, Globalization
• C components and 3rd party libraries
• Fault Tolerance and Service Continuity
• Limitative Configuration Options
• Threading
• (BOOT)CLASSPATH

39
Character Sets

• Traditional character sets do only cover part of
  the characters used in todays publications
• Because e-commerce applications are open to a
  global customer base, the application must be
  capable to deal with different encodings at the
  same time.
• Java holds characters and Strings as Unicode
  Characters -- conversion are done during reading
  and writing !
• A decompiler can help you to find which uni-codes
  have been embedded into a class file by the
  compiler
• LC_CTYPE, LC_ALL

40
Globalization

• Customers all over the world do live in different
  Timezones
• Even worse, this changes regularly Daylight
  Savings Time
• Names for the very same location can vary in
  different languages
• Roma Rome Rom,
• Den Haag La Haye The Hague

41
Agenda ...

• Character Sets, Globalization
• C components and 3rd party libraries
• Fault Tolerance and Service Continuity
• Limitative Configuration Options
• Threading
• (BOOT)CLASSPATH

42
C Components

• A serious failure in C or C code (SIGxxx, GPF)
  will cause the whole EAServer process to
  terminate or to be blocked, and all clients will
  be affected.
• In EAS 4.0, a stateless C component may be run
  in a separate process automatically
• If the component is stateful, you will need to
  write your own stub implementation (in Java or C,
  again)

43
Agenda ...

• Character Sets, Globalization
• C components and 3rd party libraries
• Fault Tolerance and Service Continuity
• Limitative Configuration Options
• Threading
• (BOOT)CLASSPATH

44
Fault Tolerance and Service Continuity

• In the case of failure
• Data must still be available
• Transactions should be restarted
• Provide continuous access to resources to clients
• Jaguar provides High Availability and Service
  Continuity per Failover and Clusterting

45
Fault Tolerance and Service Continuity

• For the client the failover should be as
  transparent as possible
• However, hiding the failover fully is not
  possible in as situations (and sometimes too
  costly)
• The client application must therefore be capable
  to repeat the last couple of steps with our
  without notification to the user
• Make sure that the users input is not discarded !

46
Fault Tolerance and Service Continuity

• Databases provide sophisticated failover options
• Clustering on machine and database-level
• Replication to a Hot Standby (plus OpenSwitch)
• other Sessions with more details.
• E148 Achieving 24x7 Availability
• E150 Building Highly Available Sybase Servers

47
Fault Tolerance and Service Continuity

• Eliminate all single points of failure by adding
  redundancy
• Multiple Machines
• Multiple Application Servers
• Multiple Name Servers

48
Fault Tolerance and Service Continuity

• Clients have a list of available name servers
• iiop//host19000iiop//host29000
• Specify RetryCount and RetryDelay
• RetryCount number of times to check server
• RetryDelay how long to wait between retries
• Properties props new Properties()
• props.put(com.sybase.CORBA.RetryCount, 10 )
• props.put(com.sybase.CORBA.RetryDelay, 5)

49
Fault Tolerance and Service Continuity

• Clients accessing Multiple Name Servers

Client
Application Server
Name Server
Client
Application Server
Name Server
50
Fault Tolerance and Service Continuity

• When one application server fails, another will
  take over

Routing Agent
Routing Agent
Web Server
Application Server
Browsers
Application Server
Web Server
51
Fault Tolerance and Service Continuity

• Autofailover for Components
• In-memory storage
• Persistent storage
• WebServer redirector plugins

52
Agenda ...

• Character Sets, Globalization
• C components and 3rd party libraries
• Fault Tolerance and Service Continuity
• Limitative Configuration Options
• Threading
• (BOOT)CLASSPATH

53
Configuration Options

• Limitative Configuration Options
• Number of HTTP Connections
• Number of IIOP Connections
• Number of Connections in a Cache
• Number of file descriptors
• Memory segment size
• Java VM Maximum Heap Size

54
Configuration Options

• When you request more than the configured number
  of objects
• An error may occur at a very low level inside the
  application that is not reported to the user
  interface
• The error may be intermittent
• The caller may be suspended

55
Threading

• Not all Component Types support Threading
• EJBs forbidden by the specification EJB Issues
• EJBs are non re-entrant and apartment threaded
• Cannot implement Runnable or subclass Thread
• PB Components are generally not thread safe, so
  they cannot be used as Service Components
• Spawning uncontrolled threads will cause EAServer
  to abort (tml_key_getdata()), using the Thread
  Manager or Service Components is the preferred way

56
(BOOT)CLASSPATH and Class Loading

• Changing either CLASSPATH or BOOTCLASSPATH
  requires to restart the server to become
  effective
• The length of (BOOT)CLASSPATH environment
  variable is limited
• The more archives, the longer it takes to load
  classes
• There might be multiple different versions of a
  Java class referenced by (BOOT)CLASSPATH
• See ClassSearch Utility http//www.sybase.com/deta
  il/1,6904,1017251,00.html

57
(BOOT)CLASSPATH and Class Loading

• EAServer does have Class Loaders to load class
  images for components and web resources
• There are several levels of ClassLoaders
• Server, Application, WebApplication, Package,
  Component
• Classes need to be loaded by the same
  classloader, otherwise ClassCastException and
  IncompatibleClassChangeError will be thrown

58
Why Troubleshooting ?

• If you want to know more, a few more
  presentations at TechWave 2002 cover topics
  related to software quality and reliability
• E144 EAServer Tips and Tricks
• E145 EAServer Sizing Estimation
• E148 Achieving 24x7 Availability
• E150 Building Highly Available Sybase Servers

59
E139Why Troubleshooting ? Avoid Problems Upfront
!

• Markus Ohly
• Sybase European CSS
• Markus.Ohly_at_sybase.com