Integrating HDF5 with SRB

1
Integrating HDF5 with SRB

• Peter Cao
• The HDF Group
• xcao_at_hdfgroup.org
• November 29, 2006

Sponsored by CIP/NLADR, NFS PACI Project in
Support of NCSA-SDSC Collaboration
2
Imagine
100 Frames x 10 GB
10 GB
3
Outline

• Introduction to the project
• HDF5-SRB model
• Server and client application
• Demo if time permits

4
Motivation
SRB HDF5
Distributed data system Indexing and searching Access control, etc Large and diverse data High performance I/O Subsetting, etc
High performance distributed data system
5
Goal
A client/server system for object-level access to
HDF5 files stored at SRB server

• Use SRB as middleware to transfer data between
  the server and client
• Use Object-level access for interactive and
  efficient access to parts of the file

6
HDF5 Data Access on SRB

• Normal ways to access SRB
• Get the whole file large files (100GB)
• Use POSIX low level calls low performance
• New way
• Implement proxy operations to access objects or
  parts of objects in one request

7
Example Whole File Access
I need to see the eye of Hurricane Bob!
client
server
Transfer large file slow!
8
Example Object Level Access
I need to see the eye of Hurricane Bob!
client
server
1 request, small transfer fast!
9
HDF5/SRB Model
client
server
HDF5 Library
HDF5 Object (File, Group, Dataset, Subset,
Attribute)
HDF5 Object (File, Group, Dataset, Subset,
Attribute)
SRB Server
HDF5-SRB Module (pack/unpack messages)
HDF5-SRB Module (pack/unpack messages)
MCAT
10
New Data Objects and APIs

• A new set data objects
• H5File, H5Group, H5Dataset, H5Datatype, etc
• Encapsulated client requests and server results
• Enhanced SRB APIs
• Pack/Unpack routines (exchange data between byte
  stream and structure) to handle complicated
  struct string, pointers, pointers to arrays,
  arrays of pointers, etc
• New srbGenProxyFunct (general Proxy Function)
  handles other types of request besides HDF5

11
Running Server/Client

• A SRB server that supports HDF5
• HDF5 library and other external libraries (SZIP,
  ZLIB)
• A SRB version 3.4.0 or later from
  http//www.sdsc.edu/srb/
• Follow instruction on how to run SRB server from
  UG packed with SRB source release or online at
  http//www.hdfgroup.org/hdf-srb-html/
• Any client application that implements HDF5-SRB
  Objects
• No HDF5 library is required on the client
• Example client application HDFView 2.3 or above

12
Example
13
HDFView property file

• .hdfview2.3
• srbaccount0.domaindemo
• srbaccount0.home/A/home/srbAdmin.demo
• srbaccount0.hostkagiso.hdfgroup.uiuc.edu
• srbaccount0.passwordsrbDemo05
• srbaccount0.port5544
• srbaccount0.resourcedemoResc
• srbaccount0.usersrbAdmin
• srbaccount1.domaindemo
• srbaccount1.home/A/home/srbAdmin.demo
• srbaccount1.hoststanley.ncsa.uiuc.edu
• srbaccount1.passwordsrbdemo05
• srbaccount1.port5544
• srbaccount1.resourcedemoResc
• srbaccount1.usersrbAdmin