JIMS the JMXbased Infrastructure Monitoring System

1
JIMS - the JMX-based Infrastructure Monitoring
System
Distributed Systems Research Group Institute of
Computer Science, University of Science and
Technology (UST/AGH) in Kraków Kazimierz Balos,
Slawomir Zielinski, Leszek Bizon, Michal
Rozenau, Marek Wiecek, Pawel Zawada, Marek Smet,
Tomasz Sekman, Marcus Hardt, Ariel
Garcia, Krzysztof Zielinski
2
JIMS Introduction (1)

• Why another monitoring system?
• existing monitoring systems in EDG and CrossGrid
  (MDS2) too tied to particular operating system
• requirement for detailed and dynamically changing
  information about each worker node, its CPUs,
  memory, etc. in the GRID
• Existing monitoring system - MDS 2
• monitoring parameters as hierarchical LDAP tree
• installed only on CE and SE
• makes available only general information
  concerning whole grid or site
• number of all CPU-s
• jobs in queues
• current number of free WN-s
• cluster configuration
• static information due to read-oriented LDAP
  storage
• makes available data delivered by specific
  providers, but not more frequently than 1 second
  (typically 300-600 s)

3
JIMS Introduction (2)

• JIMS
• exposes parameters as Web Services
• may be installed on all WN-s, CE, SE
• grid-oriented architecture including software
  installation and upgrade
• autoconfiguration (Dynamic Active and Passive
  Discovery)
• level of site autoconfiguration of WN-s
  registering in Soap Gateways
• level of grid registering of Soap Gateways in a
  set of chosen CEs
• adding new functionality doesnt require system
  restart
• own instrumentation layer (SystemInformation,
  SNMPv2c, PBS/SGE modules)
• data may be gathered at higher rates (below 1
  s)
• hierarchical architecture
• possibly not-intrusive uses connection-pooling
  from CE to WNs, doesnt affect monitored system
  if not active
• written in Java - portable
• Open Source (JMX Java Management Extensions,
  AXIS, Net SNMP)

4
JIMS Technology, JMX

• What is JMX? Technology developed by Sun
  Microsystems for managing
• and monitoring resources devices, applications,
  and service-driven
• networks.
• JMX RI Reference implementation of JMX provided
  by Sun Microsystems,
• free for development and production. Provides
  object-oriented view of
• monitored system.
• JMX Remote Provides connector technology
  enabling remote resources monitoring and
• management.
• More information sites of Sun Microsystems
  www.sun.com

5
JMX Architecture

• Key components MBeanServer, MBean

6
JIMS Architecture Overview
7
JIMS - Terminology

• Monitored station station (WN or CE Worker
  Node/Computing Element) monitored by JIMS.
  Parameters are collected in following domains
• - units of time spent by CPU on performing task
  in USER mode, SYSTEM mode and IDLE mode
  (information about WN and each CPU)
• - MEMORY utilization, BUFFERS, CACHE, SWAP
• Monitoring agent JIMS process on Monitored
  station responsible for gathering information
  and exposing it to SOAP gateway
• SOAP gateway one of services of Monitored
  station. Acts as a proxy
• between monitored stations and client
  applications, translating requests
• from SOAP to JMX/RMI
• Global gateway registry for SOAP gateways
  provides a list of registered SOAP gateways in
  the GRID

8
JIMS Dynamic Discovery Services

• Dynamic Discovery Service
• dynamic discovery of Worker Nodes in cluster
  using multicast to well known address and port
• heart beat mechanism (certain number of retries
  typically 3 before removing WN from the
  registry)
• operating in SOAP Gateway
• Global Discovery Service
• SOAP Gateways registration
• timestamps
• fail-over facility uses redundant GDS Nodes
  (typically 3)
• addresses of GDS Nodes in every SOAP Gateway

9
JIMS Monitoring Modules

• JIMS is based on plug-in technology consisting of
  modules deployable at run-time

10
Sources of Monitored Information
11
System Information Module Details (1)
12
System Information Module Details (2)
13
System Information Module Details (3)
14
Network Metrics Capabilities

• 1. measureICMPLatency
• - normal ping command
• 2. measureUDPLatency
• - measures average time assuming equal time
  for sending and receiving packet
• - can measure packets of different sizes
  (also for sending and receiving)
• 3. measureThroughput
• - uses UDP protocol
• - takes into account
• packet overhead
• UDP packet fragmentation

15
JIMS Administrative Tools
16
JIMS Command Line Interface

• cg-jims-client, cg-jims-cli
• 1. administrative tool for testing SOAP
  connection to SOAP gateways
• 2. displays statistics of whole cluster
  including
• chosen parameter values and detailed CPU
  statistics
• ICMP latency to chosen node
• UDP latency to chosen node
• throughput measured using UDP protocol
• 3. as example of JIMS API simple Java
  application using SoapGateway
• interface

17
JIMS Command Line Interface
cg-jims-client
ymbalos_at_zeus23 bin ./cg-jims-client
zeus24.cyf-kr.edu.pl 149.156.9.15 JIMS client v
1.5.25 Timestamp for Ut ts1081997570476 149.15
6.9.44 Ut11163238 ICMP lat 0.212 ms,
UDP lat 0.35 ms, Throughput 2.81E7
bit/s 149.156.9.23 Ut11884675 ICMP lat
0.125 ms, UDP lat 0.30 ms, Throughput
2.81E7 bit/s 149.156.9.26 Ut7141152
ICMP lat 0.132 ms, UDP lat 0.30 ms,
Throughput 2.40E7 bit/s 149.156.9.39
Ut7621342 ICMP lat 0.161 ms, UDP
lat 0.35 ms, Throughput 2.81E7
bit/s 149.156.9.20 Ut171871991 ICMP lat
0.129 ms, UDP lat 19.25 ms, Throughput
340686.86 bit/s 149.156.9.16 Ut10230438
ICMP lat 0.126 ms, UDP lat 0.30 ms,
Throughput 2.81E7 bit/s 149.156.9.18
Ut120823614 ICMP lat 0.141 ms, UDP
lat 0.30 ms, Throughput 2.81E7
bit/s 149.156.9.24 Ut8538910 ICMP lat
0.130 ms, UDP lat 0.35 ms, Throughput
2.81E7 bit/s 149.156.9.41 Ut9142146
ICMP lat 0.134 ms, UDP lat 0.35 ms,
Throughput 2.40E7 bit/s 149.156.9.45
Ut4189579 ICMP lat 0.119 ms, UDP
lat 0.35 ms, Throughput 3.37
bit/s 149.156.9.22 Ut6392505 ICMP lat
0.132 ms, UDP lat 0.35 ms, Throughput
2.81E7 bit/s 149.156.9.27 Ut48540937
ICMP lat 0.126 ms, UDP lat 0.30 ms,
Throughput 3.37E7 bit/s 149.156.9.29
Ut26034105 ICMP lat 0.127 ms, UDP
lat 0.35 ms, Throughput 2.40E7
bit/s 149.156.9.21 Ut171414372 ICMP lat
0.178 ms, UDP lat 27.35 ms, Throughput
340686.86 bit/s 149.156.9.15 Ut8007434
ICMP lat 0.050 ms, UDP lat 0.35 ms,
Throughput 2.81E7 bit/s 149.156.9.42
Ut10726351 ICMP lat 0.116 ms, UDP
lat 0.35 ms, Throughput 2.81E7
bit/s 149.156.9.40 Ut11989796 ICMP lat
0.165 ms, UDP lat 0.35 ms, Throughput
2.81E7 bit/s 149.156.9.17 Ut174338520
ICMP lat 0.134 ms, UDP lat 24.80 ms,
Throughput 547532.46 bit/s 149.156.9.19
Ut4789571 ICMP lat 0.127 ms, UDP
lat 0.25 ms, Throughput 2.81E7
bit/s 149.156.9.25 Ut12402766 ICMP lat
0.136 ms, UDP lat 0.30 ms, Throughput
3.37E7 bit/s 149.156.9.43 Ut6506462
ICMP lat 0.109 ms, UDP lat 0.35 ms,
Throughput 2.81E7 bit/s
18
JIMS Interactive CLI
cg-jims-cli
ymbalos_at_zeus23 bin cg-jims-cli JIMS CLI v.
1.5.23, type "help" for help JIMSgtc
ce010.fzk.de Connection prepared for
ce010.fzk.de JIMSgtget Uptime MonitoringclassSys
temInformation Uptime 01 141.52.160.16
507759.53 02 141.52.160.30 1053672.4 03
141.52.160.33 8817474.0 04 141.52.160.34
8817459.0 05 141.52.160.35 8817399.0 06
141.52.160.36 8817214.0 JIMSgtget
L15m MonitoringclassSystemInformation
L15m 01 141.52.160.16 0.0 02
141.52.160.30 1.18 03 141.52.160.33 0.0 04
141.52.160.34 0.0 05 141.52.160.35 0.0 06
141.52.160.36 0.0 JIMSgtcpu TIME/CPU0,1,... USER
NICE IDLE SYSTEM hs/s 01 141.52.160.16 0
0 0 0 0 0 202101 101 0 0
0 02 141.52.160.30 15 3 12 0 0 0
185 98 87 2 0 2 03 141.52.160.33
1 0 0 0 1 0 0 0 0 0 407102
102 102 101 0 0 0 0 0 04
141.52.160.34 0 0 0 0 0 0 0 0
0 0 407101 102 102 102 1 1 0 0
0 05 141.52.160.35 0 0 0 0 0 9
1 0 3 0 396 98 101 98 101 3 2
0 0 0 06 141.52.160.36 1 0 1 0
0 0 0 0 0 0 407101 100 101 101
0 0 1 0 0 JIMSgtquit
19
JIMS GUI - Visualization Tools

• JIMS manager
• - runs on Linux, UNIX or Windows
• - uses SOAP protocol
• - Open Source (JMX, AXIS)
• SNMP manager
• - runs on Linux, UNIX or Windows
• - uses SNMP protocol
• - Open Source (NetSNMP)
• WEB interface
• - doesnt require installing any software
• - uses HTTP protocol
• - Open Source (JMX, JMX Remote API)

20
JIMS Manager
JIMS Manager
get attribute or invoke a method
SOAP
RMI
SOAP Gateway
21
JIMS Manager
22
JIMS Manager
23
JIMS Manager
24
JIMS Manager
25
JIMS Manager
26
SNMP Manager
SNMP Manager
get attribute
SNMP
27
SNMP Manager
28
JIMS WEB Interface

• Uses built-in JMX WEB engine (HTML Adaptors)
  reliability
• Easy for development, maintenance,
  administration, demonstration and tutorial
  purposes
• Hierachical data structure
• Module domains
• Accessible by any web browser on notebooks,
  PDA-s, etc.
• Used during development phase

29
JIMS WEB Interface
Web Browser
get or set attribute, invoke a method
HTTP
30
JIMS WEB Interface SystemInformation (1)
31
JIMS WEB Interface SystemInformation (2)
32
JIMS WEB Interface SNMP (1)
33
JIMS WEB Interface SNMP (2)
34
JIMS WEB Interface NetworkMetrics (1)
35
JIMS WEB Interface NetworkMetrics (2)
36
JIMS WEB Interface ICMP Measurement
37
JIMS WEB Interface Measurement Result
38
JIMS WEB Interface UDP Measurement
39
JIMS WEB Interface Measurement Result
40
JIMS WEB Interface Throughput Measurement
41
JIMS WEB Interface Measurement Result
42
JIMS Deployment in Development Testbed

• 1. JIMS has been successfully deployed in
  CrossGrid Testbed
• providing online monitoring data required by
  Benchmarking,
• Network Prediction and Network Monitoring
  (SANTA-G)
• CrossGrid Packages
• 2. JMX-based architecture allows discovery
  monitoring
• Java 5.0 applications

43
JIMS Deployment in CrossGrid Testbed
44
Summary

• Transparency of managed resources which doesnt
  imply necessity of changing existing management
  applications
• Dynamic resources discovery is not reserved only
  for local area networks, but can be fully
  operational in wide area networks
• Not limited to grid infrastructure monitoring -
  allows Java applications monitoring due to
  compatibility with built-in JMX in Java 1.5 VM
• Scalable and modular architecture exposing
  provided functionality as Web Services

45
References, Web Links, Contact

• JIMS the JMX-based Infrastructure Monitoring
  System is available at
• JIMS web page
• http//wp3.crossgrid.org/pages/JIMS.html
• CVS, SOURCES and RPMs
• http//savannah.fzk.de/cgi-bin/viewcvs.cgi/crossgr
  id/crossgrid/wp3/wp3_3-moninfr/wp3_3_3-jims/
• http//savannah.fzk.de/distribution/crossgrid/auto
  built/i386-rh7.3-gcc3.2.2/wp3/SOURCES/
• http//savannah.fzk.de/distribution/crossgrid/auto
  built/i386-rh7.3-gcc3.2.2/wp3/RPMS/
• Contact and comments
• Kazimierz Balos kbalos_at_agh.edu.pl