QIO SciDAC file format

1
QIO SciDAC file format

• Bálint Joó (bjoo_at_jlab.org)
• Jefferson Lab, Newport News, VA
• given at
• ILDG 8 Virtual Meeting
• May 11, 2006

2
Motivation

• Defined ILDG Configuration Format success
• Now considering format for propagators
• What will you want a format for, the day after
  tomorrow?
• SciDAC has unified approach for scalar/lattice
  quantities.
• Defined sequence of LIME records
• Private fields to describe datatype, hold
  checksum etc.

3
SciDAC QIO File Structure

• File Header Record followed by logical data
  records.
• Record has at least
• private XML
• user XML
• Data records also have
• binary data
• parallel checksums.

Private File XML Data
User File XML Data
Logical Record 1
Logical Record 2
4
Use of the Private Records

• File Private XML Holds
• Format Version
• no of spacetime dimensions
• volume format (eg single file, one file per node
  etc)
• Private Record XML holds
• Version and date-time stamp
• precision
• whether data is scalar (same on all nodes) or a
  lattice
• no of spins, colours, no of items and size of
  each item

5
Use of Private Records (continued)

• Complete description is held for every record
• File self describing
• Byte order issue solved by always writing big
  endian
• Checksums are global XORs of the data
• Not 100 foolproof
• But two different ones are done
• The likelyhood of both checksums remaining
  unaffected in the presence of data corruption is
  small.

6
Software

• Software exists to read and write these files
• Freely available, relies on
• UNIX like File I/O calls
• QMP (in parallel mode only)
• Should be straighforward to port (eg works on
  QCDOC)
• QIO also reads/writes ILDG configurations
• Private records 'faked' from ILDG headers.
• But all bolt on hacks like this are yucky!

7
Conclusion

• QIO structure extremely flexible
• Can write configs, props, vectors, both wilson
  and staggered.
• Can even write correlation functions as global
  data
• Takes long term needs for flexibility into
  account rather than just the pressing needs of
  today.
• Approach worth considering if ILDG wants to be
  more than a global gauge configuration library.