PeopleTools Application Server

1
PeopleTools Application Server BEA Tuxedo
Internals

• david.kurtz_at_go-faster.co.uk
• Go-Faster Consultancy Ltd.
• http//www.go-faster.co.uk

2
Who am I?

• Independent Consultant
• Performance Tuning
• PeopleSoft / Oracle

3
Rules of Engagement

• Interrupt me
• if you think I have got something wrong
• if you have a question
• if you cant hear me

4
BEA Tuxedo

• Historical Overview
• Tuxedo Internal Architecture
• PeopleSoft 3-tier client implementation
• How does PeopleSoft Internet Architecture (PIA)
  fit in
• Configuration issues
• Monitoring

5
Brief Historical Background

• In the beginning
• monolithic

6
Client Server

• Two-tier client-server

7
3-Tier Client Server

• 1st Generation

8
3-Tier Client Server

• 2nd Generation - Tuxedo

9
Where did Tuxedo come from?

• Bell Labs - 1993
• Transactions for
• UniX
• Extended for
• Distributed
• Operations

10
What is Tuxedo?

• It is the knots on the ends of the string!

11
PeopleSoft 2-Tier Client
12
PeopleSoft 3-Tier Client
Presentation Logic
Server Side
Client Side
13
PeopleSoft 2-Tier Client

• Client Connects Directly to the Database
• 1 database connection per window
• inactivity timeout

DATABASE
14
PeopleSoft 2-Tier Client

• Client Connects Directly to the Database
• 1 database connection per window
• inactivity timeout
• What happens in 3-Tier?

DATABASE
15
PS/Tuxedo Internal Structure

• 1st Application Server Process Started
• Bulletin Board Liaison (BBL)
• Reads PSTUXCFG configuration file
• Shared Memory Segment
• Bulletin Board (BB)

DATABASE
BB
PSTUXCFG
BBL
16
PS/Tuxedo Internal Structure

• Workstation Listener (WSL)
• Listens for connection request from clients on a
  specified port
• Client initially contacts the WSL

WSL
DATABASE
BB
BBL
17
PS/Tuxedo Internal Structure

• Workstation Listener (WSL)
• Client needs to know where to find the WSL
• Machine and port specified in Configuration
  Manager

WSL
BB
BBL
18
PS/Tuxedo Internal Structure

• Workstation Handler (WSH)
• handles traffic from client after connection
• Client is assigned to a port on the WSH
• WSH port range can be constrained to match a
  firewall
• Additional WSH can be spawned if necessary
• Compress Messages

WSL
WSH
DATABASE
BB
BBL
19
PS/Tuxedo Internal Structure
WSL
WSH
DATABASE
BB
BBL
20
How does it work?
WSHQ
WSL
WSH
DATABASE
BB
BBL
21
How does PIA work?

• Java Client PIA use Jolt
• Java Class Library
• Jolt Listener maps Java functions to Tuxedo
  Messages

22
Jolt Internal Structure
JSHQ
JSL
JREPSRV
WSL
WSH
DATABASE
BB
JREPOSITORY
BBL
23
New structure for PIA (Apache Webserver)
Sessionid xy1kvm8010
http 1.1 addresscookie (stateless)
Java
Cookie
JVM
Servlet Engine
Servlet thread IS stateful
Servlet Container
Servlet
Webserver
Servlet Thread
Servlet Thread
HTTPD
Client
Servlet Thread
Client
mod_jserv
Port 8010
Tuxedo Application Server
JSL
JSH
JSH
JSH
24
What is the difference between the web servers?

• Apache
• Separate Servlet Engine
• Separate encryption module (Raven)
• Weblogic
• Contains an integral servlet container

25
Progressive Connection Concentration
35000 users

• Typical picture
• 35,000 users
• 1000 concurrent connections
• 1000 users active in the last n minutes
• where n is the Java JSH timeout
• therefore need 1000 threads
• but maybe only 4 JVMs
• 100 JSH
• assumes 10 clients per JSH
• 10-20 PSAPPSRV processes
• each PSAPPSRV connects to database
• 1 database

1000 connections 4 JVMs 1000 Java Threads
100 JSHs
10-20 PSAPPSRVs
1 database
26
Configuration
psappsrv.val
psadmin (configuration dialogue)
psappsrv.cfg
psappsrv.ubx
ubbgen (PS Utility)
psappsrv.ubb
tmloadcf (tuxedo process)
PSTUXCFG
BBL (domain startup)
27
Configuration Files

• psappsrv.cfg
• contains configurable parameters
• psappsrv.val
• validation file for configurable values
• psappsrv.ubx
• configuration template for Tuxedo domain
• psappsrv.ubb
• configuration file for Tuxedo domain
• PSTUXCFG
• compiled configuration file
• read by BBL at domain startup

28
psappsrv.ubx -v- psappsrv.ubb

• psappsrv.ubx psappsrv.cfg
• PeopleSoft file
• make changes to these files
• difficult to read

• psappsrv.ubb
• Tuxedo file
• never make changes to this file
• easier to read
• this file is compiled by tmloadcf to generate
  domain configuration file PSTUXCFG
• variables fully resolved

29
psappsrv.ubb

• Five (or Six) Sections
• Resources
• resource limits -gt BB sizing
• Machine
• Resource limits. Environment
• (Network
• Connection between Machines)
• Groups
• Logical Split (BASE, APPSRV, JREPGRP, JSLGRP)
• Servers
• Number of servers, Command line
• Services
• Timeout, Load, Priority

30
Application Server Configuration Issue

• Move quick PSAPPSRV services into a second server
  (PSQCKSRV) (y/n)?
• ltPT7.58 this question is misleading
• services are additionally advertised on PSQCKSRV
  as well as PSAPPSRV
• so they are advertised on more than one queue
• gtPT7.59 and PT8.1
• services are de-advertised from PSAPPSRV
• quick services not used in PIA, only IC services
• so not a problem

31
PeopleTools lt7.58

• Quick Service

WSL
WSH
DATABASE
BB
BBL
32
Inside the Bulletin Board

• Without Load Balancing (default)
• Table of servers process where each service is
  advertised
• Read and populated as a stack
• WSL looks for first free server where request can
  be enqueued
• This can cause problems

PSAPPSRV
PSQCKSRV
PSQCKSRV
PSQCKSRV
PSAPPSRV
PSAPPSRV
PSAPPSRV
33
Load Balancing

• Load Balancing is about choosing the queue upon
  which to place the message
• by default all requests have the same load

34
Load Balancing Trace

• Add -r to application server command line
• CLOPT"-r -e APPQ.stderr -A -- -C psappsrv.cfg -D
  GP81O81 -S PSAPPSRV
• -r enables trace to file stderr (not channel 2)
• -e overrides the output filename
• process trace file with Tuxedo utility txrpt
• obtain average service time
• make load proportional to average service time

35
Load Balancing Trace

• Output from TXRPT
• SVCNAME 18p-19p TOTALS
• Num/Avg Num/Avg
• --------------- -------- -------
• SqlRequest 16/0.10 16/0.10
• MgrClear 5/0.04 5/0.04
• RamList 1/0.43 1/0.43
• SamGetParmsSvc 1/0.50 1/0.50
• --------------- ------- -------
• TOTALS 23/0.12 23/0.12

36
Load Balancing

• load execution time (seconds) x 100

599
116
37
Multiple Queues on Busy System

• You can have many server processes on a single
  queue
• BEA recommend no more that 10 server processes
  per queue
• If you have queue contention it is likely to
  impede the WSH/JSH processes enqueuing requests.
  You are unlikely to observe queuing with PQ.
• So, if you want more than 10 PSAPPSRV processes
  you should use two APPQ queues
• In, which case you should to enable load
  balancing and you must calculate the load.
• Less significant with PIA because services
  different.

38
Multiple Queues on Busy System
WSL
WSH
DATABASE
BB
BBL
39
Multiple Queues on Busy System

• psadmin will not correctly calculate
• MAXACCESSERS
• MAXSERVICES
• tmadmin-gtbbs
• will report maximum number of services
• count of all services advertised on each server
  process
• and if you have multiple work station listeners
• MAXWSCLIENTS

40
Service Priority

• Priority is about queue jumping
• servers de-queue the request
• 9 out of 10 de-queued on the basis of priority
• 1 out 10 is taken from the front of the queue

QCKQ
41
Service Priority

• All PeopleSoft Services have same priority
• specified in SERVICES section in psappsrv.ubb
• therefore there is no queue jumping
• no reason to believe that it would be advantageous

42
Tuning Tip

• Financials
• On-line journal edit post - remote call
• Service initiates a Cobol process
• Process can take minutes
• RemoteCall services can block APPQ
• Move RemoteCall service to a separate queue

43
Multi-Node Domain

• 1 domain on many nodes

DBBL
BRIDGE
TLISTEN
DATABASE
TLISTEN
44
Multi-Node Domain

• Distinct Bulletin Board Liaison process
• Master Node
• TLISTEN process
• Listens for incoming requests
• BRIDGE
• Sends Receives messages between nodes
• in a load balancing environment
• configurable bridge load add to remote queues
• limiting number of messages sent across the
  bridge

45
Multi-Node Domain

• DO ATTEMPT THIS WITHOUT BEA CONSULTANCY
• At UBS
• PT 7.5x
• susceptible to network errors
• integrated into HP service guard
• ultimately this was removed from production
  because of transient network errors between the
  nodes causing the domain to partition

46
So what is going on?

• tmadmin
• Tuxedo command line interface
• also available within psadmin

47
tmadmin commands

• pq - printqueues
• amount of work queued and waiting
• not always accurate

48
tmadmin commands

• psr - printservers
• reports activity of individual server processes

49
tmadmin commands

• psc - printservices
• each service on each server

50
tmadmin commands

• Lots of other tmadmin commands
• documented in the Tuxedo documentation
• Use them to
• dynamically reconfigure the domain
• dynamically resize the domain
• start/shutdown extra servers
• monitoring scripts
• tmadmin -r for read only mode

51
Java Administrative Applet
52
Conclusion

• So now you know
• BEA Tuxedo/Jolt is a messaging protocol
• How the messages are moved around
• How PeopleSoft has used it
• How the PIA has been constructed
• Options for Tuning and Sizing
• Load Balancing
• Multi-Node domains
• Monitoring Scripts
• BEA Documentation
• is on your CD
• http//e-docs.bea.com

53
More Information about PT8 Application Server

• PeopleSoft 2000 Conference Americas
• 8006 PeopleTools 8.1 Application Server
• 8010 Inside the Application Server
• Buzz Voelker
• PeopleTools Application Server Team

54
Final Caveat

• PeopleSoft will not support any of the changes
  that I have described in this presentation
• Everything described in this presentation has
  been done, in production, at UBS
• If you want to use a multi-node domain you are
  strongly recommended to arrange
  support/consultancy direct from BEA.

55
Questions?
56
PeopleTools Application Server BEA Tuxedo
Internals

• david.kurtz_at_go-faster.co.uk
• Go-Faster Consultancy Ltd.
• http//www.go-faster.co.uk