Data Format Description Language DFDL

1
Data Format Description Language (DFDL)

• Martin Westhead
• EPCC, University of Edinburgh
• M.Westhead_at_epcc.ed.ac.uk

2
DFDL Overview

• Motivation and benefits
• Approach
• Separation of structure and semantics
• Scope
• Status
• Proposed GGF WG
• Getting involved
• (DFDL or DaFoDL pronounced daffodil)

3
Motivation

• There will never be a standard data format
• E.g. XML verbose, tree-based, explicit
  structure
• Legacy formats
• Application specific formats
• One size will never fit all
• But could we provide a language for describing
  formats
• Transparency of physical representation
• Automatic format conversion
• Unambiguous description of data

4
Theres more

• Explicit structure enables
• Standard transformation to/from XML
  representation
• Could allow application to read/write XML
• But provide underlying efficient binary
  representation
• Data stream/file becomes database
• Point to parts of the structure
• Extract parts of the structure
• Modify parts of the structure
• Integrate parts of different structures

5
And more

• Generic tools possible
• Browsing
• Conversion and transformation
• Annotation of data
• E.g. identify bits that depict hurricane in an
  image
• Enables general semantic labels, many ontologies
  could be developed e.g.
• S.I. units, SQL types, Time
• Community specific labels, starClass
  whiteDwarf
• Application specific labels, nodeColour green
• Could lead to a standard transformation language

6
Not fairy tales

• Based on implemented work
• BinX http//www.epcc.ed.ac.uk/gridserve/WP5/Binx/
• BFD part of the Scientific Annotation Middleware
  project (http//www.scidac.org/SAM/)
• Generalized and extended a little
• Formal semantics
• Foundation for extensibility

7
Approach

• Separate out structure and semantics
• General structural language
• Repetition
• Pointers
• References to data
• New structures can be built (compositionality)
• Semantics
• Hard to express sowe dont
• General labeling
• Label semantics define elsewhere (ontologies)
• Labels can be added (extensibility)

8
Structure arbitrary labels
9
Structure example labels
10
Structural language

• Formal semnatics
• Structured binary sequence
• Defines hierarchical structure over underlying
  sequence of binary values
• Language for describing hierarchical structure
• Repetition
• Explicit number repeats
• Termination characters
• Data reference
• Conditionals
• Data size
• Pointers
• Scope
• As general as possible but
• Must be concise and implementable
• Draft language definition on web page
  (www.epcc.ed.ac.uk/dfdl)

11
Semantic labels

• Many ontologies possible
• Initial scope probably
• Basic types (floating point, integer, character)
• Simple structures (structs, arrays, tables)
• Obvious extensions
• SQL types
• XML Schema types
• Key WG goal
• Define form and requirements of new ontologies

12
WG goals

• Formal language for DFDL data structure
• Standard representation of this language in XML
• Requirements for DFDL ontology
• Basic types ontology
• Basic structures ontology

13
Status

• Proposed GGF WG
• Chairs
• Martin Westhead (EPCC)
• Alan Chappell (PNNL)
• Guy Rixon (Astrogrid)
• Key participants
• Bruce Barkstrom (NASA)
• Regan Moore (SDSC)
• Possible session at GGF8 June, Seattle

14
Getting involved

• Webpages
• http//www.epcc.ed.ac.uk/dfdl
• Mailing list (dfdl-discuss_at_nesc.ac.uk)
• Closed but archived on web pages
• Mail me to join
• My address
• M.Westhead_at_epcc.ed.ac.uk