BinX A Tool for Binary File Access

1
BinX A Tool for Binary File Access

• eDIKT project team
• Ted Wen tedwen_at_nesc.ac.uk
• Robert Carroll robertc_at_nesc.ac.uk

2
What is BinX?

• Binary in XML
• Annotation language
• Using XML
• Descriptive
• Low-level
• Software components
• BinX library
• Generic utilities
• API

3
How and Why BinX is used
Special Application Program
Application Program
Application Program
BinX Library
ltdatasetgt lt/datasetgt
Application Program
4
The BinX Language

• Annotating a binary data stream
• Mark up data types
• Mark up sequences
• Mark up arrays
• Complex structures

5
Primitive Data Types

• Mark up data types

FF 7F 7F FF FF FF 00 00 C8 42 42 C8 00
00 1 2 3 4

• ltshort-16 byteOrderlittleEndiangt
  32767lt/short-16gt
• ltinteger-32 byteOrderbigEndiangt
  2147483647lt/integer-32gt
• ltfloat-32 byteOrderlittleEndiangt100.0lt/float-32
  gt
• ltfloat-32 byteOrderbigEndiangt100.0lt/float-32gt

6
Abstract struct types

• Mark up a sequence

ltstructgt ltunsignedShort-16 /gt
ltunsignedShort-16 /gt ltbyte-8 /gt
ltbyte-8 /gt ltbyte-8 /gt lt/structgt
Screen descriptor in GIF Screen width
unsigned short Screen height unsigned
short Packed field a byte Background
colour index byte Pixel aspect ratio byte
7
Abstract array types

• Mark up an array

ltarrayFixedgt ltinteger-32 /gt ltdim
indexTo99gt ltdim indexTo9
/gt lt/dimgt lt/ arrayFixed gt
A 2-dimensional array containing
10-by-100,32-bit integers
8
Embedded abstract types

• Complex structures

ltstructgt ltshort-16 /gt ltarrayFixedgt ltbyte-8
/gt ltdim indexTo7 /gt lt/arrayFixedgt ltstructgt
ltinteger-32 /gt ltfloat-32 /gt ltdouble-64
/gt lt/structgt lt/structgt
9
User-defined metadata

• Label the data types and structures

ltstruct varNameData Samplegt ltshort-16
varNameID /gt ltarrayFixed varNameList of 10
complex numbersgt ltstruct varNameComplexgt
ltfloat-32 varNameReal /gt ltfloat-32
varNameImaginary /gt lt/structgt ltdim
indexTo9 /gt lt/arrayFixedgt lt/structgt
10
Reusable type definitions

• Define macros for reuse

ltdefinitionsgt ltdefineType typeNameFourCCgt lt
arrayFixedgt ltcharacter-8 /gt ltdim count4
/gt lt/arrayFixedgt lt/defineTypegt lt/definitionsgt
ltstruct varNameWave_Headergt ltuseType
typeNameFourCC varNameKeyword
/gt ltinteger-32 varNameChunk_Size /gt lt/structgt
11
Linking to binary data

• Reference the binary data file

ltdefinitionsgt ltdefineType typeNameHeadergt
lt/defineTypegt ltdefineType typeNameFormat_Chunk
gt lt/defineTypegt ltdefineType
typeNameData_Chunkgt lt/defineTypegt lt/definitio
nsgt ltdataset srcmyfile.wavgt ltuseType
typeName"Header" /gt ltuseType typeName"Format_Ch
unk" /gt ltuseType typeName"Data_Chunk"
/gt lt/datasetgt
12
A BinX document

• ltbinx byteOrderbigEndiangt
• ltdefinitionsgt
• ltdefineType typeNamemyTypgt
• ltarrayFixedgt
• ltcharacter-8/gt
• ltdim indexTo9/gt
• lt/arrayFixedgt
• lt/defineTypegt
• lt/definitionsgt
• ltdataset srcmyfile.bingt
• ltuseType typeNamemyTyp/gt
• ltinteger-32 varNameX /gt
• lt/datasetgt
• lt/binxgt

Root element
Data class section
Abstract data type
Data instance section
13
DataBinX

• DataBinX BinX with Data

ltdataset srcmyfile.bingt
ltstructgt ltshort-16 /gt ltlong-64 /gt ltdouble-64
/gt lt/structgt ltarrayFixedgt ltinteger-3
2 /gt ltdim count2 /gt lt/arrayFixedgt lt/da
tasetgt
ltdatasetgt ltstructgt ltshort-16gt100lt/shor
t-16gt ltlong-64gt1000lt/long-64gt
ltdouble-64gt5.257lt/double-64gt lt/structgt
ltarrayFixedgt ltdimgt
ltinteger-32gt1lt/integer-32gt lt/dimgt
ltdimgt ltinteger-32gt2lt/integer-32gt
lt/dimgt lt/arrayFixedgt lt/datasetgt
14
The BinX Library

• Core library
• Utilities
• Applications

15
BinX Components

• The library has core functionality to support
  generic utilities and applications

Applications
BinX core functionality Parse/Gen BinX doc
Read/write binary data Parse/Gen DataBinX
Utilities
BinX Library Core
Generic tools DataBinx pack/unpack
Extractor, Viewer BinX editor
Applications Domain-specific
16
BinX application models

• Data catalogue model
• Data manipulation model
• Data query model
• Data service model
• Data transportation model

17
Data catalogue model
BinX 1
Abstract

• Primary storage
• Binary data files
• Metadata
• Syntactic annotation
• Semantic annotation
• Classification
• Domain specific
• Cross-reference
• XLink

BinX 1.2
METADATA
BinX 1.1
BinX 1.2.1
BinX 1.2.2
BinX 1.2.3
Detailed
0101010101
0101010101
0101010101
0101010101
BINARY
18
Data manipulation model

• Extraction
• Subset of a dataset
• Combination
• Merge several datasets
• Transformation
• Conversion of data types
• Change of sequence order
• Transposition of array dimensions
• Transparency
• Automatic change of byte order

19
Data query model

• In-dataset query
• XPath against virtual XML
• Cross-dataset query
• Link into multiple datasets
• Defining result format
• XQuery-based return fragment
• Output interface
• SAX events

XLink
XPath
Transform
DataBinX SAX Events
XQuery SAX Events
VOTable SAX Events
APP VOTable
APP DataBinx
APP Custom
20
Data service model

• Publishing logical datasets in BinX

0101010101
Dataset from multiple data sources
DB
BinX
0101010101
0101010101
0101010101
0101010101
BinX
BinX
Grid
Dataset from several binary files
Dataset from one binary file
Client
21
Data transportation model

• DataBinX as interlingua

XSLT
BinX Util
ZIP tool
Send Receive
XML document
DataBinX
ZIP (MIME)
XSLT
BinX Util
ZIP tool
22
Application in Astronomy

• Case Study
• Data Conversion
• Between FITS and VOTable

23
Application in astronomy

• FITS and VOTable conversion

DataBinX Utility
BinX library Core
SIMPLE T END 01010101
lt?xml version. ltVOTABLEgt lt/VOTABLEgt
24
FITS file
0
79
Primary HDU
Header
Data
Extension
Header
Data
25
VOTable

• ltVOTABLEgt
• ltRESOURCEgt
• ltPARAM nameObs valueBob/gt
• ltTABLE nameStarsgt
• ltFIELD nameStar-name datatypechar
  arraysize10 /gt
• ltFIELD nameRA datatypefloat /gt
• ltFIELD nameDec datatypefloat /gt
• ltFIELD nameCounts datatypeint
  arraysize2x3x /gt
• ltDATAgt
• ltTABLEDATAgt
• ltTRgt
• ltTDgtProcyonlt/TDgtltTDgt114.827lt/TDgtltTDgt
  5.227lt/TDgt
• ltTDgt4 5 3 4 3 2 1 2 3 3 5 6lt/TDgt
• lt/TRgt
• lt/TABLEDATAgt
• lt/DATAgt
• lt/TABLEgt
• lt/RESOURCEgt
• lt/VOTABLEgt

26
FITS ?DataBinX ?VOTable

• FITS to VOTable conversion

DataBinX Utility
FITS
XSLT transformer
DataBinX
Schema BinX
Preprocessor
VOTable
XSLT
27
VOTable?DataBinX?FITS

• VOTable to FITS conversion

Schema BinX
VOTable
DataBinX Utility
DataBinX
XSLT transformer
Binary Data
Post processor
FITS Header
XSLT
FITS
28
BinX Software

• Software library in C
• Documentation
• Utilities and Samples

29
Future releases

• XPath-based data query
• DFDL support
• Output through SAX events
• Output as XQuery return
• Database interfacing
• Java wrapper for utilities

30
Support

• Information and software download
• http//www.edikt.org/binx
• Questions
• support_at_edikt.org
• Requirements and suggestions
• tedwen_at_edikt.org
• robertc_at_edikt.org