EDG Valids Activities

1
EDG Valids Activities

• Patrick Agbu Frank Corprew
• EOS Data Gateway Science and Operations
• edg_at_killians.gsfc.nasa.gov
• CEOS Subgroup Meeting CINTEX Split
• September 9-13, 2002

Global Science Technology, Inc.
2
Introduction

• To enable and maintain interoperability among EOS
  Data Gateway (EDG) and other catalog systems,
  e.g., ECS Data Model, ECHO, GCMD, and CINTEX
  among others.
• To this end, the science and operations team have
  established a procedure for managing attribute
  keywords (Valids).

Global Science Technology, Inc.
3
Justification

• Valids are the vital components to the operation
  of the EDG.
• Utilized for the building of search queries.
• Must be resubmitted on a regular basis by a data
  provider every time their data holdings change.

Global Science Technology, Inc.
4
Guidelines

• Proper management of Valid keywords ensures
  interoperability with other catalog systems by
  using standard and agreed upon terms.
• Using and sharing keywords from reputable
  sources, e.g., the GCMD.
• Controlling some keywords and checking for
  compatibility with established ones.

Global Science Technology, Inc.
5
Clients

• Local EDG clients are hosted at data centers both
  nationally and internationally.
• The Public and Hidden Operational clients are the
  two clients that require the most frequent update
  of Valids and Valid-related information.
• A regular and frequent schedule of Valid
  processing is needed to maintain and synchronize
  the operational EDG client and the various EDG
  servers.

Global Science Technology, Inc.
6
Operational Requirements

• Components used in the EDG Valids ingest process
  include
• Submission interface
• Various computer systems
• Processing utilities
• Valids files
• Related files and information.

Global Science Technology, Inc.
7
Processing Tool

• The first component of the EDG Valids ingest
  process is the EOS Data Gateway (EOS) Valids
  Processing (EVP) interface.
• The EVP interface is a web-based
  ingest/processing and testing tool for EDG
  Valids, and Valid-related files.
• Valids and Valid-related information can be
  submitted independent of one another.
• EVP handles updates to dependent Valids,
  definitions, package information, and extended
  criteria.

Global Science Technology, Inc.
8
(No Transcript)
9
EVP Access

• There is only one username and password assigned
  to each data center/server. If more than one
  person will be accessing EVP from a particular
  data center/server, please coordinate among
  staff.
• The address of the operational EDG Valid
  Processing (EVP) interface is
• http//pilsner.gsfc.nasa.gov/evp-doc/index.ht
  ml.

Global Science Technology, Inc.
10
Processing Streams

• There are 2 Valids processing streams
• 1) Operation stream (performed twice weekly)
• 2) Test stream (1-2 day turnaround, perhaps
  longer for a NEW Data Center) or custom
  processing
• Flow TEST --gt PREOPS --gt OPS
• Clients Pre-Ops, Public, Pre-Ops Hidden,
  Hidden

Global Science Technology, Inc.
11
Processing Modes

• Handles all modes of Valid processing and
  testing, consisting of
• - public modes (ops and pre-ops),
• - hidden modes (ops and pre-ops), and
• - test modes (ECS-TS1, ECS-TS2, ECS-OPS,
  VATC-TS1, PVC-TS1, PVC-TS2, and PVC-OPS).
• EVP handles complete replacement of Valids,
  deletions, and additions.
• VATC Verification Test Center
• PVC Pretest Verification Center

Global Science Technology, Inc.
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Testing and Checking

• EVP can be used for preliminary testing of Valids
  against your server before a need to promote.
• EVP will check any Valid or Valid related files
  for syntax, semantics, and content.

Global Science Technology, Inc.
17
Schedule

• Processing of Valids is now done on a twice
  weekly basis, rather than a 2-week cycle.
• Valids received between Thursday and COB Monday
  will be processed and released by the following
  Wednesday.
• Valids received between Tuesday and Wednesday
  COB, will be processed and released by the
  following Friday.

Global Science Technology, Inc.
18
Updates

• EVP can only handle one deletion action, or one
  addition action per processing cycle.
• Replacement can be performed repeatedly, so long
  as each successive replacement incorporates
  previous modifications.
• Updates to a Data Centers dependent Valids in
  RCS occur after final ingest processing for a
  particular cycle. At that time, updated files
  are then accessible for further submission
  actions (deletion, addition, and replacement) in
  EVP.

Global Science Technology, Inc.
19
Test client

• The EDG client invoked by EVP is a test instance
  of the client, visible only to the data
  submitter, after Valids and Valid related
  information have passed syntax, semantic, and
  content checks by EVP.
• Files to be uploaded, either from UNIX platforms,
  or Pc's should be text formatted files only, in
  ODL. Microsoft word format, or other non text
  formats will not work.

Global Science Technology, Inc.
20
Promotion

• The use of EVP to upload and test Valids does not
  make them directly visible in the various EDG
  clients (hidden, public, test).
• Conversely, if one desires to test only ones
  Valids, but do not wish to have them processed
  operationally by the EDG Science and Operations
  group at that time, Request for Promotion
  should not be chosen within the EVP interface.

Global Science Technology, Inc.
21
Last Chance for Correction

• When the Request for Promotion is made, a
  template with prepopulated information is
  presented. Additional explanation of any
  additions, deletions, or changes made to any
  Valid and Valid-related files since the last
  submission is appended below the prepopulated
  information.
• EVP can be used to test Valid submission against
  either preoperational or operational server. If
  a request for promotion of Valids for either
  preoperational or operational mode is submitted
  for an operational data center, the submission
  will be treated as an operational submission.

Global Science Technology, Inc.
22
File format

• Complete replacement of dependent Valids, in
  Object Description Language (ODL), are to be
  submitted in their entirety.
• Dependent Valid additions should be submitted in
  complete odl groups for each data product.
• Dependent Valid deletions should be submitted as
  a text file with one dataset on each line, and
  each dataset name enclosed in parenthesis, and
  prefixed with
• DATASET_ID

Global Science Technology, Inc.
23
File Formats (cont)

• Packaging information, in ODL, is to be submitted
  in its entirety as a text file, regardless of the
  type of change made since last submission.
• Definition files, in ODL, are submitted
  separately as a text file. Only new or modified
  definitions are submitted for processing.
• Extended Attributes (Extended Valids)
  information, in ODL, are submitted separately as
  a text file. Only new or modified extended
  search attributes are submitted for processing.

Global Science Technology, Inc.
24
ODL Requirements

• Required fields and character limits
• Dataset_id (85 chars)
• Parameter (80 chars)
• Sensor (30 chars)
• Source (30 chars)
• Processing_level (2 chars)
• MD_ENTRY_ID (31 chars)

Global Science Technology, Inc.
25
ODL Requirements (cont)

• Optional fields and character limits
• Campaign (80 chars)
• Specialized_criteria_avail (None)

Global Science Technology, Inc.
26
Definition Guidelines

• Syntax includes ATTRIBUTE, KEYWORD, and
  DEFINITION fields.
• Definitions should be clear and concise maximum
  length should be from 500 to 1000 characters
  most definitions are much shorter.
• Include units in parameter definition when
  appropriate.
• Acronyms may be used in definitions, but they
  must be defined on first usage.
• Only new or changed definitions should be
  submitted.

Global Science Technology, Inc.
27
Extended Criteria

• In the EDG, search fields that are common across
  all (or most) Data Sets are supported as core
  search attributes. Many data centers may also
  want to provide searching on additional
  attributes, which may be common across several
  datasets, or specific to one.
• The implementation provides for extended search
  criteria as defined by individual data centers,
  and allows data centers to coordinate in
  designing specialized search criteria that apply
  to some subset of the EDG datasets.

Global Science Technology, Inc.
28
Extended Criteria (cont)

• Because the implementation calls for the EDG
  client to construct a custom search screen for
  each extended search criterion, we rely on the
  Valids information provided in the
  SPECIALIZED_CRITERIA group.
• When a user selects standard search criteria,
  only the extended search criteria applicable to
  those datasets making use of such extended
  criteria will be available.
• If there are conflicts within the user's final
  search criteria, the client will notify the user
  of the incompatibilities before the search is
  sent.

Global Science Technology, Inc.
29
Extended Search

• Path/Row is treated as a geographic
  specification, not an extended search. Search by
  Granule ID is implemented as a separate search
  type. Thus, all extended search criteria will be
  compatible with geographic search criteria.
• Unique search criteria (i.e., not variants) will
  be the only members of their categories.
• Extended search definitions are submitted by data
  providers as text file, using ODL objects for
  each SPECIALIZED_CRITERIA group corresponding to
  each extended search criteria.

Global Science Technology, Inc.
30
Advantages

• Alerts by email indicate that Valids have been
  submitted for processing in EVP.
• Gives us a record of submission in the event of
  system or disk failures.
• Indicates the changes from last submission and
  assists us in verifying that processing is done
  correctly.
• Gives us a record of input file name along with
  system generated file name for proper mode of
  processing.

Global Science Technology, Inc.
31
Summary

• Established a procedure for managing attribute
  keywords or Valids.
• Evaluate system Valids for consistency,
  scientific validity ensure compatibility,
  maintain interoperability, and populate keyword
  definitions.
• Valids are submitted via the EDG Valids
  Processing (EVP) tool.
• Data provider is responsible for ensuring proper
  ODL syntax, semantics, and content.
• Final promotion to operational status (visible to
  world) is performed by the Science Operations
  staff.

Global Science Technology, Inc.
32
Finally

• We would like to suggest that INFEO tap into ECHO

Global Science Technology, Inc.