Ecotools Case Study : Database Monitoring at BNP Paribas

1
Ecotools Case Study Database Monitoring at BNP
Paribas

• Dennis Adams
• BNP Paribas

2
Whats in a name Paribas Capital Markets

• Now part of BNP Paribas
• Investment Banking
• Euro-Denominated Bonds.
• Equities, Bonds and Derivatives Trading Systems
  in most major financial centres
• Infrastructure in territories and central London
  Data Centre.
• Finance and Reporting in Paris.

3
Databases and OS Types in Production
Tokyo, Singapore, Hong Kong, Paris, London,
New York
4
Our Responsibility Data Management Group

• Managing DBMS Servers
• Based in London, remote Support to territories
  (out-of-hours callouts !)
• Liaise with London Operations 24h/day
• Other Teams
• Systems Management - hardware OS
• Networks - LAN WAN
• Application Teams - Application Support

5
Objectives when purchasing Ecotools

• Monitoring of Sybase DBMS
• detail down to SQL statement
• Ingres as an additional requirement
• Reliable
• WAN based
• Event Alerting
• Links to TNG for Central Alerting
• Extract trend data for capacity planning

6
How we use Ecotools
GUI
Monitored Agent Machines
Alerts
Command Line Control Files for Scenarios
Ecotools Repository
7
How we use Ecotools

• Single Central Solaris Console (V6.2.1)
• Continuous running 24 X 7
• Minimal use of the Ecotools GUI
• Scenarios controlled from ECOCLI
• start/stop UNIX shell scripts.
• Alerts picked up by tail of ecotools log
• summary on daily in-house web page

8
Configuring Domains

• Domain Logical grouping of Servers
• Unit of discovery within Ecotools
• Monitor All Servers together

9
Configuring Domains

• Group by Business Unit / DBMS Type / Territory ?
• Lots of small Domains ?
• speed up discovery
• Few no of large Domains ?
• easier to get to individual machines
• COMPROMISE DOMAINS OF MAX 25
• ORGANISED BY APPLICATION TYPE (ALMOST)

10
Creating Scenarios

• Scenario Basic Collection Task
• One or more agents (cache hit, log size)
• One or more machines (All Servers)
• Time Interval (10 Minutes)
• Either using the GUI or from .CTL file
• CREATE USING GUI, SAVE AS A CTL TEMPLATE...
  SED/AWK
• START FROM UNIX SHELL

11
Managing Scenario Control Files

• Lots of Small Control files ?
• One UNIX impf process per scenario
• a single machine can have 15 requests
• 80 15 impf processes on console (!!!)
• ps -ef more
• Few no of Large Control files ?
• Unix VI ctl file when new machine added.
• Lots of machines per ctl file makes restart a
  problem

12
Managing Control Files

• EVERY MACHINE HAS ITS OWN CONTROL FILES (MAXIMUM
  OF 3)
• Use SED string substitution to created from
  standard Templates
• Ecotools Startup logic...
• For file in .ctl
• do
• ecocli
• eco run file
• done

13
GUI Stability Issues

• GUI hangs when Alerts arrive.
• Determined by ECOCLI_BLINKALARM
• ECOCLI_ALARMINTERVAL 1200 default
  2CliLogInterval
• The GUI Crashes !
• Corrupts master imdb - lose all data
• Better in version 6.2.1 (latest patches)
• SWITCH OFF ECOCLI_BLINKALARM, USE UNIX SCRIPT TO
  TAIL LOG FILE

14
(No Transcript)
15
Managing Ecotools Views

• View Repository for performance data
• consists of indexed/ flat file directory
• Need X months data - capacity planning
• Can keep views going for several weeks
• but imdb 12MB, .datalog 1.8GB.
• CREATE NEW VIEW EVERY MONTH
• SET NO OF DATA POINTS1500 IN CONTROL FILES (OR
  LOSE DATA)

16
Trend Data for Capacity Planning

• Several Months data required
• Start a new GUI for every single view ??
• SOLUTION extract data to CSV files for reading
  into database
• There is no supported utility to do this.
• SHELL SCRIPTS - GREP / SED / AWK
• UNSUPPORTED HACK !

17
Unicenter/TNG Integration

• In-house implementation calling KSH script from
  Ecotools Scenario Language
• Can be implemented as MgrAction or AgtAction
• MgrAction run on Solaris Console
• GMRPY_CDBSTOP errors - not scalable
• AgtAction run on Agent Machine
• NOT YET PROVEN

18
Summary Ecotools Experience so far

• Evaluation of version 4.n in 1997
• Monitoring of Euro changeover in 1998
• Y2K tracking and Reporting 1999
• Version 6.1 chosen as Y2K version
• Soon replaced by 6.2.1 with patches.
• Currently Monitoring nearly 80 Production Systems
  from London.

19
Looking at Ecotools V.7

• Looks impressive on paper
• SQL server storing collected data
• Genuine 3-layer architecture
• Better User interface
• Concerns
• Support for Legacy Operating Systems
• Dec UNIX versions
• Ability to create own shell script agents.
• Earlier Versions of Sybase

20
Ecotools Future at BNP Paribas

• Evaluate V7 for monitoring NT/Sybase Production
  Systems.
• Need UNIX shell agents for NT console
• Objective move to centralised NT console -
  eventually
• Keyword STABILITY

21
Success Stories

• Sybase Settlement System - Tokyo
• Reserved log space low
• caused by backup server failing and transaction
  logs not being properly dumped. Restarted backup
  server
• Potential system hang averted
• Delayed Settlement We get Fined

22
Success Stories

• Ingres Equity Derivative Trading - New York
• Large Table Approaching Ingres Architectural
  Limit of 2GB
• Planned outage to remodify table into multiple
  locations to allow it to grow
• Equity Trading Outage averted
• Keep the Traders Trading

23
Success Stories

• Ingres Forex Trading - Paris
• Ingres Error Log reported potential UNIX
  Filesystem Error one evening
• Alerted Paris Unix Team, who decided to ignore
  the warning
• System Crashed overnight
• But at least we could say I told you so

24
Final Proof

• Accepted within DMG Group
• Integrated with our way of working
• We are now a more Pro-Active Team
• Looks good on the CV, provided you call it...
• E-COTOOLS
• Lunch Time ?