DFDL WG Session 1 Summary of Status

1
DFDL WG Session 1Summary of Status

• WG Co-Chairs
• Mike Beckerle, Ascential Software
• Alan Chappell, PNNL
• Martin Westhead
• (Two note takers please?)
• Tuesday, 2004-09-21 17h300200 Brussels
  (BE.CEST) in Room D005.
• 11h30 New York, (US.EDT, UTC-4)
• 08h30 San Francisco (US.PDT, UTC-7)

2
Abstract

• Progress and Status Review - This session will
  review the progress since GGF11 which is
  substantial, and also provide enough context for
  those interested in getting involved who haven't
  been following along.

3
ADF BoF

• Sep 22 11h00-12h30ACCESS TO DATA IN FILES-WG
  CHARTER DISCUSSION BOF (DATA) Calendar QB
  (200p)DetailsTitle ADF - Access to Data in
  FilesSession Leaders Neil Chue Hong, Brian M
  CollinsAgenda- Summary of General Focus,
  Milestones and Objectives- Discussion of
  Relationships specifically with DFDL and DAIS as
  well as other GGF Groups- Charter Update

4
DFDL WG Session 2

• Room D008
• Thursday, 2004-09-23 11h000200 Brussels
  (BE.CEST)
• 05h00 New York
  (US.EDT, UTC-4)
• 02h00 San
  Francisco (US.PDT, UTC-7)

5
Since GGF11 (June 2004)

• Latest working examples
• document/jdm20040720.zip
• DFDL Represenation Properties Basic Set
• document/ggf-dfdl-rep-properties-proposal-basic-00
  1.doc

6
DFDL-WG Session 1 Summary of Status

• Agenda
• Review of purposes/goals
• XML / XSD impact Data Model
• Examples w/latest updates
• as evolved now different from primer document
• General discussion (40 mins)
• Overview of other sessions (5 mins)

7
Data Interchange Formats

• Prescriptive Put your data in this format!
• XML textual
• Binary ASN.1, XDR, NetCDF, HDF, EBML,
• Descriptive What format is your data in?
• Commercial products
• ASN1 Encoding Control Notation
• ITU-T X.692
• DFDL

8
Why Descriptive?

• Allows us to achieve two goals simultaneously
• Interoperability
• Modern and Legacy data formats
• Performance!
• Density
• Fewest bytes to represent data without resorting
  to compression
• Optimized I/O
• Seekable random access
• Memory mapped, aligned
• Without sacrificing general access

9
Why the GGF for DFDL?

• Grids are about big-data and big-computation
  problems
• Simplistic solutions like use XML wont cut it!
• Grids are about universal data interchange

10
General Features

• Basic ASCII/Binary Read capabilities
• Inclusion of static info, e.g. units
• Validated input (from XML Schema)
• Defaulted input for missing values
• Reference use of a previously read value in
  subsequent expressions
• Choice use of a previously read value to select
  among format variations
• Multi-layer description of an intermediate
  representation not exposed in the final result
• Multiple input streams
• Basic Math in DFDL expressions
• Basic Math to generate new representations/value
  s
• Push-back capability
• New type/transform specification

11
Desires

• Support very efficient parsers
• Hints about lengths so data can be accessed
  randomly
• Avoid requiring data copies
• But allow caching (e.g. based on hints)
• Allow black-box transforms (e.g. unzip)
• Support round-tripping
• Link transforms and inverses
• Infer inverses for lossless transforms
• Keep simple cases simple
• Default transforms and parameters

12
Related Standards Efforts

• Prescriptive systems
• W3C binary XML (http//www.w3.org/XML/Binary/)
• Formed, but discussion group has no items.
• CORRECTION - there is activity now.
• Descriptive systems
• ASN1 Encoding Control Notation
• ITU-T X.692

13
XML Synergy

• Use XSD to describe the logical data
• Use annotations within the XSD to describe the
  representation of it.

14
XSD Types

• Elements
• A.k.a. fields
• Sequence groups, All groups
• All unordered group
• Choice
• A.k.a. union, redefine,
• Vectors
• Use element with minOccurs, maxOccurs.
• Nillability
• A.k.a. Nullable values

15
XML/XSD basic types
16
Example 1 XML

• ltwgt5lt/wgt
• ltxgt7839372lt/xgt
• ltygt8.6E-200lt/ygt
• ltzgt-7.1E8lt/zgt

17
Example 1 XSD

• ltxssequencegt
• ltxselement name"w" typeint"/gt
• ltxselement name"x" typeint"/gt
• ltxselement name"y" typedouble"/gt
• ltxselement name"z" typefloat"/gt
• lt/xssequencegt

18
Example 1 DFDL - binary

• 0000 0005 0077 9e8c
• 169a 54dd 0a1b 4a3f
• ce29 46f6

19
Example 1 DFDL - binary
20
Example 1 DFDL - textual

• 5, 7839372, 8.6E-200, -7.1E8

21
Example 1 DFDL - textual

• ltxscomplexType name"example1"gt
• ltxsannotationgt
• ltxsappinfogt
• ltrepTypegttextlt/repTypegt
• ltcharsetgtUTF-8lt/charsetgt
• ltdecimalSeparatorgt.lt/decimalSeparatorgt
• ltseparatorgt,lt/separatorgt
• lt/xsappinfogt
• lt/xsannotationgt
• ltxssequencegt
• ltxselement name"w" type"int"/gt
• ltxselement name"x" type"int"/gt
• ltxselement name"y" type"double"/gt
• ltxselement name"z" type"float"/gt
• lt/xssequencegt
• lt/xscomplexTypegt

22
More Examples

• Currently working on a set of examples which
  illustrate each of the DFDL constructs

23
Examples (so far)

• BasicMathInExpressions.xsd
• BasicMathInRepresentations.xsd
• Choice.xsd
• DefaultValuePropagation.xsd
• DefaultedInput.xsd
• IncludeTransform.xsd
• IncludeType.xsd NewDFDLType.xsd
• MultiLayer.xsd
• MultiStreams.xsd
• NewDFDLTransform.xsd
• NewExternalDFDLTransform.xsd
• NewMixedDFDLTransform.xsd
• Reference.xsd
• StaticInformation.xsd
• ValidatedInput.xsd

24
Current Status

• Fairly clear idea of required features and data
  types
• Large library of simple, real world, and
  pathological examples
• Converging on an annotated XML Schema syntax for
  DFDL

25
Status

• We think we have
• A good handle on requirements/use cases
• Common idea of functionality/ overall design
• A not quite complete syntax based on XML Schema
• A set of unit-test examples to help in
  finalizing the syntax

26
Opportunities

• Good time to help with syntax
• Once that exists
• Standardized language bindings/APIs
• DFDL implementations
• DFDL-based ADF Data Virtualization Services
• Extensions for specific domains/uses

27
Next Steps

• Session 2
• Charter-revisions discussion
• Review of specific examples
• IBM Websphere Business Integrator DFDL
  presentation

28
END