DDI 1'0 is here''''what now

1
DDI 1.0 is here....what now?

• Jostein Ryssevik
• NESSTAR/FASTER
• NSD

2
DDI achievements

• Developed a comprehensive metadata standard
  expressed in a modern language (XML).
• Acceptance fast take up in the community of data
  archives and libraries world-wide.
• Contributed to the modernisation of data
  archiving.
• Community building revitalised the co-operation
  and sharing of know-how and technologies among
  the archives and libraries
• Strengthening of the ties to the data producers
• Initiated software development.

3
So, what now....?

• Can we lean back and be satisfied with what we
  have achieved and let the process unfold?
• No, if we are to profit from what already is
  done, we will have to develop it further.
  Stopping means being left behind.
• If we see the DDI as an important building block
  of the emerging data Web, we should forget all
  pleasant thoughts about an early retirement -
  the process has just begun!

4
Limits to change....?

• On the one hand we know that the DDI must be
  developed further in order to meet our internal
  needs as well as the external requirements.
• On the other hand, every change has a cost. As
  soon as archives start to invest time and money
  in developing DDI-resources, a natural resistance
  against change kicks in.
• Dilemma How fundamental and how rapid changes
  can we impose, without hurting the take-up
  process?

5
Shortcomings...

• Too rigid and too loose at the same time
• Too rigid in the sense that there are no
  extensibility mechanism - every change requires a
  new DTD.
• Too loose in the sense that most elements are
  voluntary.
• Ambiguities related to how different structures
  are to be used adds to this looseness

6
Shortcomings...

• A general lack of controlled vocabularies
• the effect will be that even within organisation
  terminology will not be consistent, causing
  problems for any resource discover process.
• Two solutions
• add more controlled vocabularies
• add a mechanism to indicate what vocabulary that
  has been used and a way to make these
  vocabularies publicly available on the Web

7
Shortcomings...

• Machine readable versus machine understandable
• a good thing about XML is that it is both human
  and machine readable,
• but there is a long way from machine readability
  to machine understanding
• If we want the DDI not only to hold information
  to be read by humans, but also to drive software
  processes, than we have to design our standard
  accordingly

8
Shortcomings...

• The survey data (rectangular file) bias
• the DDI is derived from the typical survey data
  codebook
• attempts have been made to cover other types of
  data, like hierarchical data, aggregated tables
  (cubes) etc.
• but mostly by conceptual stretching
• what we need is a metadata standard that is able
  to describe all relevant data structures within
  the same coherent framework

9
Shortcomings...

• The single study bias
• the DDI is an excellent tool to describe a single
  independent survey data file
• but what do we do when we have series of related
  or close to identical surveys (Eurobarometers,
  ISSPs etc.)
• the DDI has no efficient way of coping with this
  challenge

10
Shortcomings...

• ...the tyranny of the book metaphor..
• the DDI is built on the metaphor of a (code)book
  - it has chapter and sub-chapters but the DDI
  documents are supposed to be read and processed
  in toto
• would a more modular approach where components or
  modules are allowed to be combined and to
  interact more freely be more appropriate?

11
Shortcomings...

• ...how easily can the DDI inter-operate with
  other metadata standards..
• the DDI has a mapping to another more general
  metadata standard - the Dublin Core
• but the exact relationship between DDI- and
  DC-elements are only expressed as comments
  (messages to the taggers)
• there is no mechanism that allows us to express
  these relationships explicitly (and machine
  understandable)
• And what about other standards (like ISO-11179)

12
Challenges...

• ...Should the DDI stay as an XML-DTD or are there
  more tempting alternatives...?
• What about XML-schema that among other thing is
  more data (as opposed to document) friendly and
  also will allow us to control the content of the
  elements more precisely
• And what about RDF that many see as the
  forthcoming metadata language of the Web
• ...dont forget that the DDI already has made a
  successful move from SGML to XML

13
Challenges...

• ...how should the DDI-work be organised in the
  future...?
• Until now organised as semi-formalised activity
  based on voluntary work by a handful of committee
  members loosely representing different parts of
  the data-provider and data-archive community
• ..should it be more formalised?
• ..should a more strict decision-mechanism be
  introduced
• ..should it be left to a professional
  standardisation body (like ISO)?
• ..who should fund it

14
Challenges...

• ...and how do we organise the co-operation among
  the archives to maximise the profit from the
  DDI-process
• development of best practices, controlled
  vocabularies etc. to facilitate harmonisation
• sharing of tools and software
• sharing of resources (we must avoid 5 archives
  spending resources documenting the same
  international dataset in 5 slightly different
  versions)
• ...do we need a new CESSDA policy for sharing of
  data and division of labour???????????