ESDIS Status

1
ESDIS Status

• Richard Ullman
• ESDIS Project
• Richard.E.Ullman_at_nasa.gov

2
Outline

• ESDIS General Status
• HDF-EOS Plans
• Website http//hdfeos.gsfc.nasa.gov

3
ESDIS Science Data Services Today

• EOSDIS provides support for high data volumes
  from Aqua, Terra, and Landsat 7, and continues to
  support QuikSCAT, ACRIMSat, SAGE-III, JASON and
  pre-EOS-era data including TRMM, UARS,
  TOPEX/Poseidon, RADARSat, and others.
• EOSDIS finalizing preparations to support ICESat
  and SORCE
• In all, EOSDIS is providing data processing,
  archival, and/or distribution for over 15 Earth
  science satellite missions.
• EOSDIS has set a new benchmark for data
  management. The total volume of the science data
  in our archives totals over 2 Petabytes. Since
  1998, the science data volume managed by the
  EOSDIS has increased eight-fold, and continues to
  grow at a rate of over 2 Terabytes per day
• In FY02, EOSDIS provided more than 16 million
  data and information products to over 1.8 million
  individuals.

4
Science Operations Timeline



Heritage Missions T/P, UARS, TOMS, ERBE
Science Operations Supported
Landsat 7 QuikSCAT
Terra ACRIMSat
Aqua GRACE
SAGE III JASON
SeaWiFS Radarsat
TRMM
15 Million
Products Distributed
FY94/95 FY96 FY97 FY98
FY99 FY00 FY01 FY02
5
EOSDIS DAAC Data delivery FY02October 1, 2002
through September 30, 2002

• Over 1.8 million distinct users
• 16,013,980 products delivered
• Notes to charts on following slides
• Distinct users includes users accessing DAAC web
  pages, including web-crawlers
• Distinct user type is based on email addresses of
  users or URLs
• Product delivered is defined as the smallest
  deliverable unit of data
• Product delivery breakdown is based on email
  addresses of users receiving ECS and Non-ECS data
• FTP Delivery are to URLs not mapped to specific
  domains
• Foreign Other includes foreign email addresses
  whose country is known but domain-type could not
  be determined

6
16,013,980 Data Products DeliveredOctober 1,
2001 - September 30, 2002
7
Over 1.8 Million Distinct UsersOctober 1, 2001 -
September 30, 2002
8
ESDIS Status

• After years of development, ECS is operational
  and generally recognized as successful.
• Primary ECS development contract is essentially
  completed.
• ECS Maintenance and Development (EMD) will
  emphasize maintenance more than development.
• Already the majority of ESDIS budget has shifted
  from development to operations.
• Still two major areas of new capability (see
  posters at AGU for more information)
• Data Pools
• ECHO

9
HDF-EOS Plans

• HDF-EOS 5 development is nearly complete.
• Continue to maintain, port to newer operating
  systems, bug-fix.
• Need the advise of this community workshop - we
  will discuss this afternoon.
• What tools or capabilities are now needed?
• HDF-EOS 2 and HDF-EOS 5
• When is it the right time to press EOS science
  teams to migrate to HDF-EOS 5?
• What steps should NASA take to facilitate?

10
Data Pools Concept The Right Data, the Right
Way, Right Now!
PIPE Personalized Information Presentation
Engine
11
General Capabilities

• The Right Data Data Location
• Groups, themes, bookmarks and views
• Navigation and machine-based location
• Science views (e.g., science metadata)
• Applications views (e.g., OGIS coverage server)
• Location aids (e.g., geopolitical overlays)
• External location-support services
• The Right Way Data Tailoring
• Data reduction, manipulation and reformatting
  services
• Virtual data products
• Workflow management and execution monitoring
• External tailoring services
• Right Now Rapid Access
• Low latency data transfers
• Secure remote file access
• On-the-fly data compression
• Automated request routing and load balancing
• Near real-time data

12
EOS ClearingHOuse (ECHO) http//eos.nasa.gov/echo

• ECHO is a metadata clearinghouse
• A single Internet portal for Earth science
  metadata search
• Index of data provider inventory-level data
  holdings metadata.
• .ECHO is a data order broker
• Forwards orders for data discovered to the data
  providers to fill.
• Data providers retain customer fulfillment
  service
• ECHO is a data service broker
• Registered service are associated with registered
  datasets
• Four kinds of service association
• Advertised, Context Passing, Brokered, Order
  Option
• ECHO is an open client API for custom user
  clients
• The EOS Data Gateway (EDG) is ESDIS ECHO client

13
ECHO Data Providers

• Current ECHO Data Providers
• EOSDIS Core System DAACs (17 thus far)
• EDC Land Processes DAAC
• Goddard DAAC
• NSIDC DAAC
• ORNL DAAC (100)
• ECHO holds the metadata for over 3 million
  granules, and growing

14
HDF-EOS Tools and Information Web
Sitehttp//hdfeos.gsfc.nasa.gov

• Richard.E.Ullman_at_nasa.gov

15
http//hdfeos.gsfc.nasa.gov
16
Website status

• A resource for discovering about hdf-eos in
  particular.
• This workshop series presentation archive
• Links to hdf-eos tools
• Site has been revamped according to comments
  received at the last workshop.
• Tools download page now has opportunity for user
  feedback.
• Workshop presentations are keyword searchable
• New features planned
• Better introductory material.
• Post and organize documentation of HDF-EOS.
• Better navigation to NCSA site for HDF
• Host hdf-eos web forum
• Incorporate the eostools_at_eos.nasa.gov listserv

17
HDF-EOS Profile

• Richard Ullman
• ESDIS Project
• Richard.E.Ullman_at_nasa.gov

18
History

• September 1993, HDF adopted as baseline standard
  for EOSDIS Core System standard data product
  generation, archival, ingest, and distribution
  capabilities
• Dec. 94 - ECS Engineering Support Directive to
  create HDF-EOS
• June 1996, HDF-EOS v1.0 library released
• Upgrades every 6 mo.,
• Current version 2.8 on HDF 4
• HDF5 support (called HDF-EOS 5) beginning
  November 2000
• Current version 5.1.3 on HDF5

19
HDF-EOS Data Objects

• Point
• Swath
• Profile (Swath subtype)
• Grid
• Zonal (HDF-EOS 5 only)

20
HDF-EOS 5

• Based on HDF5, a complete rewrite of HDF4 with a
  different interface.
• First released in 2000.
• Designed to resemble HDF-EOS 2 to the maximum
  extent possible.
• Support same data structures
• Added prefix HE5_ to HDF-EOS 2 functions.
• Doesnt preclude HDF5 functionality.
• Data Type changes, e.g. INT64 -gt H5T_NATIVE_LONG

21
HDF-EOS 5 Functionality

• Basic File I/O
• Fill Values
• Compression
• Chunking/Tiling
• Swath Interface
• Grid Interface
• Point Interface
• Profile Interface

• Global (File), Group Local Attributes
• External Data Files
• Subsetting
• Unix/Linux Support
• Threadsafe Version
• FORTRAN, C, C
• General Table Interface (proposed)

22
Top Level of HDF-EOS 5
Root -- /
HDFEOS INFORMATION
HDFEOS
STRUC. METADATA
ADDITIONAL
SWATH
GRID
POINT
Global (file) Attributes
The new ADDITIONAL Group has global (file)
attributes The new functionality is added to the
EH(utility) interface.
23
Swath Structure
SwathName
Global Attribute ltSwathNamegtltAttrNamegt
Group Attribute ltDataFieldsgtltAttrNamegt
Data Fields
Geolocation Fields
Profile Fields
Profile Field.n
Latitude
Local Attribute ltFieldNamegtltAttrNamegt
Profile Field.1
Longitude
Data Field.n
Data Field.1
CoLatitude
Time
Each Data Field can have Attributes and/or
Dimension Scales
Shaded Objects are implemented in a fixed way.
User doesnt have direct access via the interface

Dataset
Group
Attribute
24
HDF-EOS Point

• Intended use
• Discrete points in time and/or location.
• Table of data linked to table of geographic
  information.
• 8-level Hierarchical, each level may contain
  indices to the level below

25
HDF-EOS Point

• Hierarchical links
• Every level in a Point data set must be linked
  into the hierarchy.
• Before two levels can be linked, a link field
  must exist

26
HDF-EOS Swath

• Intended use
• Across track scanning instruments.
• Sounding instruments
• Level 1 Geolocated Sensor Units
• Level 2 Geophysical Parameters

27
HDF-EOS Swath

• Data Fields

• Geolocation Fields

Brightness Temperature
Time
Dimension Name Geotrack Size 21
Latitude
Longitude
28
HDF-EOS Grid

• Intended use
• Variables mapped on uniform space-time grid
  scales
• Level 3 - Gridded single measurement parameters
• Level 4 - Modeled or derived from multiple
  measurements

29
HDF-EOS Grid
30
HDF-EOS Grid

• Projections Supported
• Geographic
• Transverse Mercator
• Universal Transverse Mercator
• Hotine Oblique Mercator
• Space Oblique Mercator
• Polar Stereographic
• Lambert Azimuthal Equal Area

• Lambert Conformal Conic
• Polyconic
• Interrupted Goodes Homolosine
• Integerized Sinusoidal
• Compression Methods
• Run-Length Encoding
• Adaptive Huffman
• Gzip

31
Product Levels

• Level 0 - Reconstructed, unprocessed
  instrument/pay-load data at full resolution any
  and all communications artifacts, e.g., synch.
  frames, communications headers, duplicate data
  removed.
• Level 1A- Reconstructed, unprocessed instrument
  data at full resolution, time-referenced, and
  annotated with ancillary information, including
  radiometric and geometric calibration
  coefficients and georeferencing parameters, e.g.,
  platform ephemeris, computed and appended but not
  applied to the Level 0 data.

• Level 1B - Level 1A data that have been processed
  to sensor units (not all instruments will have a
  Level 1B equivalent).
• Level 2 - Derived geophysical variables at the
  same resolution and location as the Level 1
  source data.
• Level 3 - Variables mapped on uniform space-time
  grid scales, usually with some completeness and
  consistency.
• Level 4 - Model output or results from analyses
  of lower level data, e.g., variables derived from
  multiple measurements.

32
Resources

• HDF-EOS on the web
• http//hdfeos.gsfc.nasa.gov/
• http//newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.h
  tml
• HDF and HDF5 on the web
• http//hdf.ncsa.uiuc.edu/
• HDF-EOS and HDF via email
• eostools_at_eos.nasa.gov
• hdfhelp_at_ncsa.uiuc.edu