iRODS performance test and SRB system at KEK

1
iRODS performance test and SRB system at KEK

• Yoshimi Iida _at_ KEK
• Building data grids with iRODS
• 27 May 2008

2
Outline

• Performance measurement
• Transfer test between CC-IN2P3 and KEK
• Scaling test for ICAT
• Concurrent test for ICAT
• Compare with iRODS and SRB
• SRB/iRODS system at KEK

3
Transfer between CC-IN2P3 and KEK
CC-IN2P3, FR
KEK, JP
NY, USA

• 1GB data transfer between CC-IN2P3 and KEK
• Compare with iRODS and bbcp
• The network route go through USA
• The RTT is about 285ms

4
System used

• iRODS system at CC-IN2P3
• ICAT enabled iRODS server on Solaris 10
• Thumpers Sun x4500 (AMD processors)
• Oracle 10g is on a cluster of dedicated machines
• Linux file resource from local disk system
• iRODS 0.9
• iRODS system at KEK
• iRODS server on RHEL3
• Intel Xeon 3.0GHz 4
• Linux file resource from local disk system
• iRODS 0.9

5
From KEK to CC-IN2P3

• 1GB data transfer during 24 hours
• window size 4MB
• number of parallel streams 16
• bbcp often fail to connect

6
From CC-IN2P3 to KEK

• 1GB data transfer during 12 hours
• window size 4MB
• number of parallel streams 16
• iput is better than bbcp

7
Scaling test

• Data
• Using the same directory at client machine
• 1000 files of 1000 bytes each
• Measurement
• Ingesting directory and listing the collection
• Performance measured for every directory
  operation
• 1 collection and 1000 files

8
System used

• ICAT based on Oracle system at CC-IN2P3
• ICAT enabled iRODS server on Solaris 10
• Thumpers Sun x4500 (AMD processors)
• Oracle 10g is on a cluster of dedicated machines
• iRODS resource and client on SL4
• Dual AMD Opteron Processor 848
• Linux file resource from local disk system
• ICAT based on PostgreSQL system at KEK
• ICAT enabled iRODS server on RHEL3
• Dual Intel Xeon 2.8GHz
• PostgreSQL 8.2.5 is running on the same machine
• iRODS resource and client on RHEL3
• Dual Intel Xeon 2.8GHz
• Linux file resource from local disk system

9
Ingesting up to 1 million files
Running the other process
10
Nested collection test

• Data
• Registering same directory at client machine
• 100 files of 100 bytes each
• Measurement
• Ingesting directory and listing the collection
• Making nested collection every 10 collection
• Performance measured for every directory
  operation
• 1 collection and 100 files

11
Nested collection test
nest-121 OCI_ERROR
coll-1220 USER_PATH_EXCEEDS_MAX
12
Concurrent test

• Data
• Ingesting same directory at client machine
• 1000 files of 1000 bytes each
• Measurement
• Runing multi process at the same time
• read operation ils and iget
• write operation iput and ireg
• mixed operation iput, ireg, ils and iget
• Performance measured for every directory
  operation
• DB setting
• Set the number of maximum connections to 200

13
Concurrent test for reading metadata
Error connectToRhost failed
14
Concurrent test for writing metadata

• iput Oracle ICAT
• Because of limit of inodes, we cannot put any
  more files

15
Concurrent tests for mix - PostgreSQL
16
Concurrent test for mix - Oracle

• iput Oracle ICAT
• Because of limit of inodes, we cannot put any
  more files

17
Compare with iRODS and SRB

• For the scaling test
• Data
• Using the same directory at client machine
• 1000 files of 1000 bytes each
• Measurement
• Ingesting directory and listing the collection
• Performance measured for every directory
  operation
• For the nested collection test
• Data
• Registering same directory at client machine
• 100 files of 100 bytes each
• Measurement
• Ingesting directory and listing the collection
• Making nested collection every 10 collection
• Performance measured for every directory operation

18
System used

• iRODS system at KEK
• ICAT enabled iRODS server on RHEL3
• Dual Intel Xeon 2.8GHz
• PostgreSQL 8.2.5 is running on the same machine
• iRODS 1.0
• iRODS resource and client on RHEL3
• Dual Intel Xeon 2.8GHz
• Linux file resource from local disk system
• SRB system at KEK
• MCAT enabled SRB server on RHEL3
• Dual Intel Xeon 2.8GHz
• PostgreSQL 8.2.5 is running on the same machine
• SRB 3.5.0
• SRB resource and client on RHEL3
• Dual Intel Xeon 2.8GHz
• Linux file resource from local disk system

19
Scaling test - iRODS and SRB
20
nested collection iRODS and SRB
coll-1220 USER_PATH_EXCEEDS_MAX
nest-47 Error Problem running command
21
SRB system for Belle at KEK
LCG user
LCG user
GridFTP
KEK-DMZ
SRB
SRB-DSI
SRB server
Enhanced GridFTP service
Belle Net dedicated inside
Pluggable Extension
NFS
3.5PB

• Both protocols are authorized by GSI
• Setting up on GridFTP server
• grid-mapfile for SRB user
• SRB configuration file for server and resource
• Register LCG user DN on MCAT

MCAT
HSM
SRB client
LSF
Computing Farm Still not integration with Grid
Belle analysis user
22
Plan for iRODS system at KEK

• Data transfer for J-PARC project
• Generate huge amount of imaging data at Tokai
• About 1PB data in a year in total
• Store the data at Tokai storage once, then copy
  to KEK and distribute for collaborators
• Storage at Tokai are recycled
• Bandwidth between 2sites
• will be 10Gbps

23
Acknowledgements

• Special thanks to Jean-Yves Nief from CC-IN2P3
  for his help to setup iRODS system and supporting
  at CC-IN2P3
• Thanks to Adil Hasan from RAL for his help to
  these tests

24
Back up
25
Bandwidth Lyon-KEK

• iperf with some options
• -w 4M TCP window size Bytes
• -P 16 the number of parallel threads
• -i 5 periodic bandwidth reports sec

26
Summery of iRODS Performance

• Transfer from KEK to CC-IN2P3 is not stable but
  iput is better than simple transfer software
  (bbcp)
• iRODS can manage 1 million files stability
• In case of very nested collection, PostgreSQL
  ICAT takes long time to register data
• Better performance than SRB
• Oracle ICAT can handle more than 300 clients at
  the same time, but it takes time according to
  increase the number of clients