DM233 Troubleshooting your OpenSwitch 12'0 Environment

1
DM233Troubleshooting your OpenSwitch 12.0
Environment
Samir Nigam Principal PSE Enterprise Solutions
Division Samir.Nigam_at_sybase.com
2
Agenda

• Introduction
• Installation and Configuration
• Administration Commands
• Troubleshooting
• Error Messages
• Questions?

3
Agenda

• Introduction
• Installation and Configuration
• Administration Commands
• Troubleshooting
• Error Messages
• Questions?

4
IntroductionWhat is OpenSwitch

• Open Server Gateway Solution
• Provides functionality to manually or
  automatically manage client connections.
• Can manage any number of remote servers
• Routing and load balancing capabilities
• Includes error detection and recovery features
• Integrates into any HA environment

5
IntroductionFeatures

• Transparent connection management
• Failure detection and recovery
• Transaction aware
• Server pooling and routing
• Load balancing and chaining
• Notification Events

6
IntroductionFeatures (cont.)

• Customizable HA Coordination
• Connection Caching
• Connection Suspend/Resume
• Resource Governing
• Client Messaging
• Dynamic Configuration

7
Agenda

• Introduction
• Installation and Configuration
• Administration Commands
• Troubleshooting
• Error Messages
• Questions?

8
Installation and ConfigurationSetup

• Sybase Studio Installer installs OpenSwitch and
  all its dependencies
• OpenSwitch 12.0 Components
• Jaguar Runtime (MP Open Server libraries)
• Very similar to Open Client/Server installation
• OpenSwitch Binaries
• OpenSwitch Server binary
• CM library and headers
• Sample Configuration file

9
Installation and ConfigurationSetup (cont.)

• Only supported Jaguar libraries are those shipped
  with OpenSwitch.
• Do not mix this with your any EA Server
  installation.
• Environment variables
• SYBASE
• OPENSWITCH
• LD_LIBRARY_PATH/SHLIB_PATH/LIBPATH
• PATH

10
Installation and ConfigurationConfig File
(OpenSwitch.cfg)

• Used by OpenSwitch for initialization. This file
  is required.
• Broken into sections similar to a Windows .INI
  file.
• Contains sections
• CONFIG - General configuration variables
• SERVER - Initial state of each server managed.
• POOL - Defines a group of servers.
• LIMIT_RESOURCE - Describes resource governing.

11
Agenda

• Introduction
• Installation and Configuration
• Administration Commands
• Troubleshooting
• Error Messages
• Questions?

12
Administration Registered Procedures

• Categories
• Users monitoring
• User connections management
• Configuration Management
• Miscellaneous
• Methods of Execution
• Connected directly using isql with
  administrative user/password
• Executing as an RPC through an ASE server.

RPC Request
OpenSwitch Server
Admin (isql)
Application
13
AdministrationUsers Monitoring

• OpenSwitch provides several procedures for
  monitoring the state of user connections.

14
AdministrationUsers Connections Management

• These procedures should not be used to manage the
  connection that is issuing the request!

15
AdministrationConfiguration Management

• OpenSwitch attributes such as pools can be
  modified at run-time via built-in registered
  procedures.

16
AdministrationConfiguration Management

• The following procedures can be used to control
  the status of pools and servers status.

17
AdministrationMiscellaneous

• The following procedures may be used to display
  general help or debugging information.

18
AdministrationUseful Debugging Flags

• The rp_debug registered procedure may be used to
  enable/ disable debugging output to the
  OpenSwitch log.
• Can use DEBUG parameter in OpenSwitch config file
  to do the same.

Log
rp_debug l, on
Admin
19
AdministrationOpen Server Trace Flags

• SRV_TRACE option in OpenSwitch config file can be
  set to enable OpenServer trace flags (only at
  startup).

• More details on these flags in Open Server/C
  Reference Manual.

20
AdministrationClient Library Trace Flags

• CTX_TRACE option in OpenSwitch config file can be
  set to enable Context-level trace flags (only at
  startup). These flags are enabled for ALL
  outgoing ct-lib connections.

• More details on these flags in Open Client/C
  Reference Manual.

21
AdministrationOpen Client Trace Flags

• CON_TRACE option in OpenSwitch config file can be
  set to enable Connection trace flags (only at
  startup). These trace flags are enabled on ALL
  out-going ASE server connections.

• More details on these flags in Open Client/C
  Reference Manual.

22
Agenda

• Introduction
• Installation and Configuration
• Administration Commands
• Troubleshooting
• Error Messages
• Questions?

23
TroubleshootingSystem Diagnostics

• System per-process limits
• File Descriptors
• Memory
• Hardware Issues
• CPU
• Network
• System Monitoring Tools
• Windows NT
• Solaris
• AIX
• HP/UX

24
TroubleshootingSystem Diagnostics

• File Descriptors
• 2 Per Client (plus additional 1 for each client
  during failover).
• Ex, max. x clients support need is 2x 10
  (OpenSwitchs own needs) and add y for if at
  fail-over y clients are connected to this
  OpenSwitch server. Total would be 2x10y.
• Set using ulimit or limit command. Check man for
  more.
• If possible, request your System admin. to set
  this in the kernel. On Solaris, it is done by
  /etc/system setting
• Make your soft and hard limits for maximum
  available file descriptors high enough to support
  above needs.
• Windows NT has no such limitation.

25
TroubleshootingSystem Diagnostics

• Memory
• POSIX Thread (created for each client)
• Thread stack size 64K on most platforms. Set
  using STACKSIZE option in OpenSwitch
  configuration file.
• Additional memory allocation
• Client connection memory pool
• Around 54K/connection
• Set data and vmemory resource of ulimit or limit
  to unlimited for best scenario. It will allow
  OpenSwitch process to grow (if need be) to the
  size of a machines memory (RAMswap).
• Memory requirement 118K per connection

26
TroubleshootingSystem Diagnostics

• Hardware Issues
• CPU
• OpenSwitch will scale uniformly to all CPUs.
• Open Server threads are scheduled by Operating
  System. All threads have same priority.
• Network
• Check your bridges, hubs, routers for any
  possible network congestion with this new
  configuration.
• Since all clients will connect to this OpenSwitch
  host m/c which connects to ASE host(s) in the
  back, It may cause network delays.

27
TroubleshootingSystem Diagnostics

• System Monitoring Tools
• Windows NT 4.0
• Task Manager
• Aggregated summary of cpu, memory, and process
  statistics.
• Performance Monitor
• More granular look at the Processor, Memory,
  Server, and Process.
• Logs and graphs the statistics to identify
  possible bottlenecks.
• Solaris 2.6
• top Lists summary of host machine at specified
  interval.
• perfmeter Displays cpu usage, system load,
  network packets, swap activity.
• netstat Displays network usage summary.
• truss Traces system calls and signals

28
TroubleshootingSystem Diagnostics

• System Monitoring Tools
• Solaris 2.6 (cont.)
• vmstat Reports virtual memory statistics.
• mpstat Reports per-processor statistics.
• sysdef Lists all devices, modules, and values of
  select kernel tunable parameters.
• prtdiag Display system configuration and
  diagnostic information.
• psrinfo Displays information about available
  cpus on the host.
• showrev Show machine and software revision
  information.
• HP/UX 11.00
• swlist Displays information about software
  products.
• sam System Adminitration Manager.
• glance Process Monitoring and System call
  tracing.

29
TroubleshootingSystem Diagnostics

• System Monitoring Tools
• HP/UX 11.00(cont.)
• route Displays information about connectivity.
• netstat Report network resource summary.
• AIX 4.3.2
• lslpp Displays information about software
  products.
• smit System Management Interface Tool to setup
  host, network, file systems, and kernel
  parameters.
• trace, trcrpt System level tracing. Needs some
  familiarity to use it.
• svmon Captures and analyzes a snapshot of
  virtual memory.
• netpmon Monitors activity and reports statistics
  on network I/O and network related CPU usage.
• vmstat Reports virtual memory statistics.
• netstat Report network resource summary.

30
TroubleshootingSystem Diagnostics

• System Monitoring Tools
• EXTRA Process commands on Solaris
  (/usr/proc/bin)
• pfilesLists all open files in a Process. Useful
  to count files open at runtime.
• pmap Prints the address space map of each
  process. Useful to calculate the size of the
  OpenSwitch process and growth with each new
  connection.
• pstack Lists each threads execution stack.
  Possible use in identifying threads causing a
  particular problem.
• pldd Lists all dynamically loaded modules.

31
TroubleshootingConnectivity Issues
32
TroubleshootingConnectivity Issues
33
TroubleshootingConnectivity Issues
34
TroubleshootingStability Issues
35
TroubleshootingFail-over Issues
Automatic Fail-over
Note Next Server is next available server in a
pool as determined by POOL mode.
36
TroubleshootingFail-over Issues
Automatic Fail-over
37
TroubleshootingFail-over Issues
Manual Fail-over
38
TroubleshootingThread States
Threads (SPID) states during Normal Processing
39
TroubleshootingThread States
Threads (SPID) states during Attentions Processing
40
Agenda

• Introduction
• Installation and Configuration
• Administration Commands
• Troubleshooting
• Error Messages
• Questions?

41
Error Messages

• Error Handlers
• DEBUGe enables all errors to be logged to LOG
  file.
• All errors with severity gt 10 are logged by
  default. All messages are passed to the Client.
• Open Switch Error Prefixes
• CSLIB
• SRVLIB
• CTLIB
• Server
• unknown
• Suppress error messages.
• SUPPRESS_CHARSETS Do not send charsets change
  message.
• SUPPRESS_LANG Do not send language change
  setting message.
• SUPPRESS_DBCTX Do not send database context
  change message.

42
Error Messages

• Error Message Structure
• OpenSwitch does not modify any error message
  properties such as severity.
• Error Message Structure

43
Error Messages
44
Error Messages
45
Error Messages
46
Error Messages
47
Agenda

• Introduction
• Installation and Configuration
• Administration Commands
• Troubleshooting
• Error Messages
• Questions?