Metadata Requirements for EOSDIS Data Providers

1
Metadata Requirements for EOSDIS Data Providers

• Siri Jodha Singh Khalsa
• khalsa_at_colorado.edu

HDF-EOS Workshop II
2
Topics

• Why metadata is important
• Types of metadata in HDF-EOS files
• Required metadata
• How metadata is encoded and delivered

HDF-EOS Workshop II
3
What is Metadata?

• Metadata is information that identifies and
  characterizes an information product.
• Sometimes called data about data

HDF-EOS Workshop II
4
Users Need Metadata

• Metadata is needed to answer questions such as
• What time and location does this data apply to?
• Why type of instrument and processing produced
  the data?
• What other inputs were used to generate the data?
• What QA has been performed on this data?
• Who do I contact if I have questions about this
  data?

HDF-EOS Workshop II
5
Metadata is Essential

• Large data archive systems cannot function
  without metadata.
• Metadata is used to keep track of such things as
• where the data is
• what type of operations are possible on the data
• whether there are any access restrictions on the
  data
• how individual data files are logically grouped
  into collections.

HDF-EOS Workshop II
6
Key Concepts

• A granule is the smallest aggregation of data
  that is independently described and inventoried
  by the ECS. A granule consists of 1 or more
  physical files.
• A collection is a logical grouping of granules.
• The ECS Data Model allows for
• Core attributes
• Product-Specific Attributes (PSAs)

7
Types of Metadata

• Metadata in HDF files
• stored as global text attributes
• Types of Metadata used in HDF-EOS files
• Structural Metadata
• Core Metadata (inventory, can include PSAs)
• Archive Metadata (non-searchable,
  product-specific)
• Collection level metadata
• core and product-specific

HDF-EOS Workshop II
8
Required Metadata

• Origins of metadata requirements
• what is required to archive and retrieve files
• what is required to provide search and other
  services on data
• what is federally mandated (FGDC)
• There are 287 attributes in the ECS data model
• only a subset are used for any given product
• 101 are applicable at the granule level

HDF-EOS Workshop II
9
Metadata Coverage

• Science Data that are delivered for archiving in
  ECS must meet what is called the Intermediate
  level of metadata coverage. This involves as few
  as
• 31 collection level attributes
• 4 granule level attributes
• Compliance at this level is not enforced by the
  system.

HDF-EOS Workshop II
10
Collection-Level Metadata for Intermediate
Coverage

• ShortName
• LongName
• CollectionDescription
• VersionID
• ArchiveCenter
• RevisionDate
• VersionDescription
• CollectionState
• MaintenanceandUpdateFrequency
• ECSDisciplineKeyword
• ECSTopicKeyword
• ECSTermKeyword
• ECSVariableKeyword
• ContactOrganizationName
• Role

• SpatialCoverageType
• PointLatitude
• PointLongitude
• TimeType
• DateType
• TemporalRangeType
• PrecisionofSeconds
• EndsatPresentFlag
• CalendarDate
• TimeofDay
• GuideName
• DataCenter
• DocumentVersion
• DocumentUpdated
• Title
• DocumentCreated

HDF-EOS Workshop II
11
Granule-Level Metadata for Intermediate Coverage

• There are only four granule-level metadata
  attributes required
• ShortName
• VersionID
• SizeMBECSDataGranule
• ProductionDateTime
• ShortName and VersionID are identical to the
  collection-level attributes with these names.
• For granules coming into ECS, SizeMBECSDataGranule
  and ProductionDateTime are supplied by the
  system upon insertion.

HDF-EOS Workshop II
12
How is Metadata Supplied?

• Collection-level metadata is carried in an Earth
  Science Data Type (ESDT) Descriptor file.
• Granule-level metadata is defined in the
  descriptor file and populated using a Metadata
  Configuration File (MCF).
• Granule-level metadata is delivered in the
  HDF-EOS granule or in a populated MCF
  accompanying a non-HDF granule.
• The DAAC where a collection will reside is
  responsible for descriptors and ingest routines.

HDF-EOS Workshop II
13
Metadata Work Flow for External Data Providers
HDF-EOS Workshop II
14
Metadata Resources on the Web

• ECS Metadata Homepage
• http//ecsinfo.hitc.com/metadata/metadata.html
• Metadata Works (ESDT Descriptor Tool)
• http//et3ws1.HITC.COM/metadata_works/
• EOSDIS Information Architecture
• http//spsosun.gsfc.nasa.gov/InfoArch.html
• Federal Geographic Data Committee
• http//www.fgdc.gov/

15
QA w/ Experts Panel

• Q If you are a new data provider, how do you
  get your data into an HDF-EOS granule, given the
  bewildering array of utilities and tools
  available? What is the simplest solution for
  this?
• A The recommended solution is to obtain the HCR
  package, which includes the HDF-EOS and HDF
  libraries. For populating the required metadata
  in the granule, obtain the Metadata/Time
  Toolkit_MDT. The steps would be
• 1. Write an HCR and use the tools to turn this
  into a skeletal HDF-EOS granule. (This step is
  optional).
• 2. Use the HDF-EOS library to create a granule.
  (If starting with a skeletal HDF-EOS file
  generated from an HCR then plain HDF calls can be
  used to insert data into the granule ).
• 3. Use Toolkit_MDT calls to insert metadata into
  the granule. This requires generation of an MCF
  in ODL. Metadata_Works is available for doing
  this. As an alternative, a simple HDF call can be
  used to attach minimum metadata (in ODL) to an
  HDF file.
• Note if the data are going to reside in a DAAC,
  or in an archive that must be interoperable with
  ECS, you will need to generate collection-level
  metadata. Metadata_Works is the recommended tool
  for this.