R-GMA

1
R-GMA

• Producers and Archiver
• Paul Mealor

2
Overview
gsiwuftp
edg-netmon-archiver
edg-gridftplog2rgma
Log
Archiver API
P
Producer API
MySQL DB
Here be servlets
edg-netmon-rgma-info
P
Producer API
P
A
C
DBP
edg-pinger-timeping
edg-netmon2rgma
Producer API
FIFO
3
edg-netmon2rgma

• Maintains producers for the tools
• Tools scripts are rerun for each measurement
  batch
• Program running APIs must be persistent
• Files
• /opt/edg/sbin/edg-netmon2rgma.pl
• /opt/edg/var/edg-netmon2rgma/edg-netmon2rgma-fifo
• /etc/rc.d/init.d/edg-netmon2rgmad
• Configuration
• RGMA client configuration

4
Tool/netmon2rgma interaction
PingER
Normal log
Web interface

• addmetric pinger rtt 10 NMIdSourcepc18.hep.ucl.ac
  .uk toolpinger

edg-netmon2rgma
If producer doesnt exists
If bufferSize lt count 4 / 3

• CircularBufferProducer_new(NetworkRTT,
• WHERE NMIdSourcepc18.hep.ucl.ac.uk AND
  toolpinger)
• CircularBufferProducer_setRemoteBufferSize(produc
  er,102)

5
Tool/netmon2rgma interaction
PingER
Normal log
Web interface

• measurement pinger rtt pc18.hep.ucl.ac.uk
  edgnm.gridpp.rl.ac.uk pinger 100 57

edg-netmon2rgma

• CircularBufferProducer_insert(producer,
• INSERT INTO NetworkRTT
• (NMIdSource, NMIdDestination, tool, packetSize,
  value)
• VALUES
• (pc18.hep.ucl.ac.uk, edgnm.gridpp.rl.ac.uk,
  pinger, 100, 57))

6
edg-netmon2rgma

• Possibly should rewrite in Java
• Problem in C (Perl) API results in memory leak
  and heavy performance hit
• Current solution is to restart the daemon with a
  cron job
• /etc/cron.d/edg-netmon2rgma-restart
• But java lacks the text-handling of Perl
• Would have to have java on the NM

7
Tools modifications

• Generally 2 new functions
• initialise_rgma
• Writes initialisation directives to the FIFO
• i.e. fixed columns, metric names, expected number
  of entries
• Netmon2rgma creates producers, changes buffer
  sizes when necessary
• produce_it
• Writes the results of the measurements
• No extra configuration
• Assumes /opt/edg/var/edg-netmon2rgma/edg-netmon2rg
  ma-fifo

8
edg-netmon-rgma-info

• Publishes NetworkSE, NetworkCE tables
• Republishes every 12 hours (i.e. keepalive)
• Files
• /opt/edg/sbin/edg-netmon-rgma-info.pl
• /opt/edg/etc/edg-netmon-rgma-info.conf
• Local NM, associated SEs and CEs
• /etc/rc.d/init.d/edg-netmon-rgma-infod
• RGMA client configuration

9
edg-ftlog2rgma

• Publishes file-transfer logs (i.e. GridFTP)
• Files
• /opt/edg/sbin/edg-gridftplog2rgma.pl
• /etc/rc.d/init.d/edg-gridftplog2rgmad
• Reads from GridFTP log
• /var/log/gsiwuftpd.log
• Publishes new entries when they are added
• RGMA client configuration

10
edg-netmon-archiver

• Program
• /opt/edg/share/edg-netmon-archiver.jar
• /opt/edg/sbin/edg-netmon-archiver
• runs the Java code
• /opt/edg/bin/edg-netmon-archiver-setup
• Configures MySQL database requires mysql root
  password
• What to do in LCFG?
• /opt/edg/etc/edg-netmon-archiver.conf
• List of tables to archiver plus predicates for
  each
• /etc/rc.d/init.d/edg-netmon-archiverd
• Also RGMA client configuration

11
edg-netmon-archiver

• Ironically this might be the best package to
  write in C or Perl
• Archiver API is all but unaffected by C API
  problem
• But might run on R-GMA servlet machine
• Interaction with edg-netmon-info
• Archives every row that edg-netmon-info produces
• Perhaps need to automatically remove duplicates
  (via mysql)

12
Package progress

• Tools, edg-netmon2rgma, edg-netmon-rgma-info
• 99
• Latest tweaks still need to be uploaded
• edg-ftlog2rgma
• 75
• rc script, autobuild process
• edg-netmon-archiver
• 95
• need to upload, tag, check build process (ant)

13
Testing

• Tools on pc18.hep.ucl.ac.uk
• edg-netmon2rgmad on pc18.hep.ucl.ac.uk
• User servlets on pc23.hep.ucl.ac.uk
• Registry and Schema servlets on
  pc23.hep.ucl.ac.uk
• Something always goes wrong when I try to connect
  to the WP3 Registry at infocat.gridpp.rl.ac.uk
• gt1200 RTT/loss measurements archived by Tuesday
• gt130 TCP throughput measurements to localhost )

14
GridFTP logging in MDS

• Paul Mealor

15
Status

• Information provider
• Will be in EDG 2 on SE
• Publishes the most recent information for each
  host pair
• MDS being phased out in EDG 2
• GIIS structure replaced by R-GMA
• Will be all R-GMA later

16
Details

• Publishes achieved throughput
• Filesize, time etc as parameters
• Publishes most recent transfer between each host
  pair
• Specifies source and host pair, plus, if known,
  source and host NM
• NM names taken from config file as with other
  netmon info providers
• Uses same schema as other netmon
• But doesnt use the full tree structure

17
Schema

• dn NMMeasureIdsource/dest,optrootdn
• objectClass NetworkMeasurement
• NMMeasureId source/dest
• SourceHost source
• SourceNE sourceNE
• DestHost dest
• DestNE destNE
• MetricName gridftpthroughput
• MetricValue throughput
• MetricUnit bit/s
• MonitorTool GridFTP
• MetricTime timestamp
• Parameter time etime
• Parameter size size
• Parameter buffersize buffer
• Parameter streams streams
• Parameter stripes stripes
• createTimestamp timestamp
• modifyTimestamp timestamp

18
END