HDF5 Tools Update

1
HDF5 Tools Update

• Peter Cao - The HDF Group
• xcao_at_hdfgroup.org
• November 6, 2007

This report is based upon work supported in part
by a Cooperative Agreement with NASA under NASA
NNX06AC83A. Any opinions, findings, and
conclusions or recommendations expressed in this
material are those of the author(s) and do not
necessarily reflect the views of the National
Aeronautics and Space Administration.
2
Outline
Overview of current tools New tools in
HDF5 HDF-Java 2.4 release

3
HDF5 Command Line Tools

• Readers
• h5dump, h5diff, h5ls
• new tools h5check, h5stat

• Writers
• h5repack, h5repart, h5import, h5jam/h5unjam
• new tool h5copy, h5mkgrp

• Converters
• h4toh5, h5toh4, gif2h5, h52gif

4
Where to start

• Tools located at bin/ directory
• Use -h option for help
• Online help http//www.hdfgroup.org/hdf5tools.html
  
• Email to THG help desk help_at_hdfgroup.org

5
h5dump

• Dumps file content to stdout in
• ASCII text
• XML
• Binary

4
6
h5dump -H SDS.h5
HDF5 "SDS.h5" GROUP "/" GROUP
"Floats" DATASET "FloatArray"
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE ( 4, 3 ) / ( 4, 3 )
DATASET "IntArray"
DATATYPE H5T_STD_I32LE DATASPACE
SIMPLE ( 5, 6 ) / ( 5, 6 )
7
h5dump -d /Floats/FloatArray SDS.h5

• HDF5 "SDS.h5"
• DATASET "/Floats/FloatArray"
• DATATYPE H5T_IEEE_F32LE
• DATASPACE SIMPLE ( 4, 3 ) / ( 4, 3 )
• DATA
• (0,0) 0.01, 0.02, 0.03,
• (1,0) 0.1, 0.2, 0.3,
• (2,0) 1, 2, 3,
• (3,0) 10, 20, 30

8
h5dump -x SDS.h5
9
h5dump Binary Output

• -b F, --binaryMEMORYFILELEBE

h5dump -d /IntArray -o out_le.bin -b LE SDS.h5
0000000 00000000 00000001 00000002 00000003
00000004 00000005 0000030 0000000a 0000000b
0000000c 0000000d 0000000e 0000000f 0000060
00000014 00000015 00000016 00000017 00000018
00000019 0000110 0000001e 0000001f 00000020
00000021 00000022 00000023 0000140 00000028
00000029 0000002a 0000002b 0000002c 0000002d
10
h5diff
Show Differences
Between
two objects or two files
11
h5diff SDS.h5 SDS2.h5
Dataset lt/IntArraygt and lt/IntArraygt 5
differences found
12
Report differences
h5diff SDS.h5 SDS2.h5 -r /IntArray

• Dataset lt/IntArraygt and lt/IntArraygt
• position IntArray IntArray difference
• --------------------------------------------------
  ----
• 0 0 0 10 10
• 1 0 10 100 90
• 2 0 20 200 180
• 3 0 30 300 270
• 4 0 40 400 360
• 5 differences found

13
h5repack

• Copies a file to a new file with
• Compression filter
• Data layout
• Other user options

14
h5repack new filters

• -f GZIPSZIPSHUFFLETNBITSOFFNONE
• For example
• h5repack -i SDS.h5 -o out.h5 -f GZIP6

15
h5repack data layout
-l CHUNK COMPA CONTI For
example h5repack -i SDS.h5 -o out.h5 -l CHUNK2x3
16
h5repack latest version of file format
The testing file was created with HDF5 1.6.5. The
root group of the file contains 100 groups. Each
group at the root contains 9 sub groups. Each sub
group contains one compound dataset. The compound
dataset has 40 members of 32 bit integer
17
h5repart

• Repartitions a file into a family of files
• For example
• h5repart -m 200m int16kx16k.h5 part200md.h5

part200m0.h5
200 MB
977 MB
200 MB
part200m1.h5
200 MB
part200m2.h5
200 MB
part200m3.h5
177 MB
part200m4.h5
18
h5import

• Imports binary/ASCII data into an HDF5 file
• h5import infile -c config_file infile -c
  config_file2 ... -outfile outfile
• For example
• h5import float5x4x2.txt -c First_set.conf -o
  First_set.h5

GROUP "/" GROUP "work" DATASET
"First-set" DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE ( 5, 2, 4 ) / ( 8, 8,
H5S_UNLIMITED ) DATA
(0,0,0) 1.01, 1.02, 1.03, 1.04,
(0,1,0) 1.11, 1.12, 1.13, 1.14,
(1,0,0) 1.21, 1.22, 1.23, 1.24,
(1,1,0) 1.31, 1.32, 1.33, 1.34,
(2,0,0) 1.41, 1.42, 1.43, 1.44,

PATH work/First-set INPUT-CLASS
TEXTFP RANK 3
DIMENSION-SIZES 5 2 4 OUTPUT-CLASS
FP OUTPUT-SIZE 64
OUTPUT-ARCHITECTURE IEEE
OUTPUT-BYTE-ORDER LE
CHUNKED-DIMENSION-SIZES 2 2 2
MAXIMUM-DIMENSIONS 8 8 -1
19
h5jam/h5unjam

• h5jam
• h5jam -u test_ub.txt -i test_ub.h5
• h5unjam
• h5unjam -i test_ub.h5 -o out_ub.txt -o out_ub.h5

HDF5
User Block
unjam
Jam
HDF5
User Block
20
h5ls

• Lists selected information about file objects in
  the specified format
• For Example,
• h5ls -r SDS2.h5

/Floats Group /Floats/DoubleArray
Dataset 10, 5 /Floats/FloatArray
Dataset 4, 3 /Floats/subs
Group /IntArray Dataset 5, 6
21
gif2h5 / h52gif

• gif2h5 converts a GIF file into HDF5
• gif2h5 apollo17_earth.gif apollo17_earth.h5
• h52gif converts an HDF5 file into GIF
• h52gif apollo17_earth.h5 apollo17_earth2.gif -i
  /apollo17_earth.gif/Image0 -p "/apollo17_earth.gif
  /Global Palette"

22
h5toh4 / h4toh5

• h5toh4 -- Converts an HDF5 file to an HDF4 file
• h4toh5 -- Converts an HDF4 file to an HDF5 file

http//www.hdfgroup.org/h4toh5/
23
New tools
h5copy
h5stat
h5mkgrp
h5check
24
h5copy

• Copies an object within a file/cross files

/
/
Floats
IntArray
FloatArray
FloatArray
25
h5copy -f shallow
/
floats
-f shallow
/
64-bit
f32
floats
integers
/
i1
64-bit
i2
f32
floats
f2
f1
64-bit
f32
f2
f1
26
h5copy -f soft
/
/
f1
dset_SL /f1
-f soft
f1
dset_SL /f1
/
dset_SL /f1
27
h5copy -f ref
/
d1
d2
/
dset_ref
-f ref
d2
d1
/
dset_ref
dset_ref
28
h5mkgrp

• Creates groups

h5mkgrp test.h5 /info /data /data/test1
/data/test2
/
info
data
test1
test2
29
h5stat

• Prints statistics about HDF5 file
• Header size
• Raw data size
• Attributes, datatypes, and etc.
• Helps
• To troubleshoot overhead in HDF5 files
• To choose specific objects properties and
  storage strategies

30
h5check

• A validation tool that verifies if an HDF5 file
  is encoded according to the HDF5 File Format
  Specification

31
Why h5check?

• Verify if the file is compliant with the File
  Format
• As a verification tool required by the
  application of HDF5 File Format to be an ANSI
  standard
• Serves as a watch dog that the HDF5 library
  implementation is compliant with the File Format

32
Example a compliant file

• h5check example1.h5
• VALIDATING example1.h5
• FOUND super block signature
• VALIDATING the super block at 0...
• VALIDATING the object header at 928...
• VALIDATING the btree at 384...
• FOUND btree signature.
• VALIDATING the local heap at 96...
• FOUND local heap signature.
• Result File is in compliance.

33
Example a non-compliant file

• h5check invalid2.h5
• FOUND super block signature
• VALIDATING the super block at 0...
• VALIDATING the object header at 928...
• VALIDATING the btree at 384...
• FOUND btree signature.
• VALIDATING the SNOD at 1248...
• FOUND SNOD signature.
• VALIDATING the object header at 976...
• check_sym(at 1248) Errors from
  check_obj_header()
• decode_validate_messages() Failure in
  type-gtdecode().
• H5O_sdspace_decode() Bad version number in
  simple dataspace message.
• VALIDATING the local heap at 96...
• FOUND local heap signature.
• Main() Errors from check_obj_header().
• decode_validate_messages() Failure in
  type-gtdecode().
• H5O_attr_decode() Can't decode attribute
  dataspace.
• H5O_sdspace_decode() Bad version number in
  simple dataspace message.

34
Implementation Status

• Basic file format components

1.6 - DONE
1.8 - IN PROGRESS
35
HDF-Java 2.4
HDF-Java 2.4 Released !!

• http//www.hdfgroup.org/hdf-java-html
• ftp//ftp.hdfgroup.org/HDF5/hdf-java

36
HDF-Java 2.4 Release

• Build with HDF4 2r2 and HDF5 1.6.6
• New platforms
• Mac intel
• Linux 64-bit AMD
• Solaris 64-bit
• 36 new enhancements and 44 bugs fixed
• Fixed memory leak
• Test suite
• Enhanced documentation

37
Memory leak in old version

• while (true)
• H5File f new H5File
• (fname, H5File.READ)
• f.open()
• f.close()