A Brief History of the 50YearOld Space Physics Data Environment

1
A Brief History of the 50-Year-Old Space Physics
Data Environment

• Joseph H. King
• Perot Systems at NASA/Goddard Space Flight
  Center, Greenbelt, MD

2
A Framework for the Talk

• How do selected, publicly accessible, space
  physics systems and interfaces, created at
  various times, satisfy various functional and
  search requirements?
• Some repository-created systems provide
  value-added access to data held at those
  repositories
• Other systems are specifically oriented
  towards a distributed data environment
• Focus is not on underlying technologies, although
  their evolution has enabled the great progress in
  satisfying functional requirements.
• We suppress programmatic history (e.g., NASAs
  Rice U. and River Bend meetings).
• We underemphasize format and documentation
  issues.
• We do not address data preservation/stewardship
  issues nor creation of value-added data products
  like OMNI.

3
Functional Requirements

• Data Finding
• Data product level
• Data granule level
• Data Retrieval
• File level
• Subset by parameter
• Data Display
• Lists
• Plots
• Data Processing to higher levels
• Data Analysis tools
• Coordinate transformations
• Cross correlations
• Etc.
• Data Usability
• Supporting material, including error bars or
  equivalent, enables
• long-term correct and independent usability

4
Search Requirements

• Free text
• Physical Observables
• Time span
• Time resolution
• Spectral range
• Spectral resolution
• Region sensed
• In situ
• Remote sensing
• Observatory and Instrument
• Person
• Repository
• Rare event occurrence
• Parameter inside or outside specified range
• Concurrence of data in multiple favorable
  locations

5
Some repository-specific systems and interfaces
(many dates are approximate)

• Pre-network
• 1978 CDAW at NSSDC, 10s of data sources
• Pre-Web
• 1981 SDDAS _at_ SWRI for Cluster, DE, Tether,
  Astrid
• 1983 OMNI captive account at NSSDC
• UCLA magnetic field data from Polar, ISEE 1,2 3,
  IMP8
• 1992 SDAC _at_ GSFC for SMM, Yohkoh, SOHO,
• Web era
• 1994 OMNIWeb _at_ NSSDC
• 1994 COHOWeb _at_ NSSDC for Voyager,Ulysses,
  P1011...
• 1994 CDAWeb _at_ SPDF for ISTP s/c and many more
• SPIDR _at_ NGDC for GOES, etc.
• DARTS _at_ JAXA, Japan, for Geotail, Reimei, Akebono
• CDPP (Toulouse), for Cluster, Wind, Ulysses,
  ARCAD3
• ACE Science Center (Caltech) for ACE and SAMPEX
• Etc.

6
Coordinated Data Analysis Workshop

• Developed at NSSDC in 1977-8 to support
  International Magneto- spheric Study analysis
• Major effort to find good events (spacecraft well
  located, interesting geophysical events, good
  data coverage)
• Event selection supported by Satellite Situation
  Center with satellite-satellite and
  satellite-ground orbits and conjunctions
• Participants submitted space and ground data in
  advance, then came to NSSDC for workshop
• Eight CDAWs hosted, 1978-1990. Clusters of JGR
  papers from CDAWs 2, 6, 8. (10s of data sets
  per CDAW, e.g., 65 data sets used for CDAW 6)
• Last 2 CDAWs hosted on VAXs, based on CDF and
  network-accessible
• Paper data catalogs used by participants, plots
  and lists made on the fly at CDAW. New
  parameters creatable on the fly.
• The term CDAW was taken over by the solar
  community in about 1999

7
Southwest (SWRI) Data Display and Analysis System
(SDDAS)

• First created at SWRI in 1981 to support Dynamics
  Explorer data analysis.
• Using calibration and related data and
  information, worked with telemetry-level data and
  produced plots of geophysical parameters for
  analysis.
• Required Instrument Data File Set (IDFS),
  involving a header file, a data file, a virtual
  instrument data file (VIDF), and an optional
  plot information and description file (PIDF).
• Can output data files in several formats (CDF,
  netCDF, ASCII).
• Now operates in client-server mode, with
  IDFS-packaged data at sites in addition to SWRI
  (e.g., MSSL, IRF) significant usage in Cluster
  community.
• Spacecraft supported TSS, UARS, AMPTE, ARIA, DE,
  DMSP, ISEE, Spacerad, Interball, IMP, Astrid,
  Cluster, MARS, Prognoz, Viking, Freja, ISIS,
  NOAA-12, etc.

8
NSSDC Account for OMNI Data Access

• Captive account created by NSSDC in 1983 to
  provide access to OMNI data and then some other
  data.
• Allowed users to select from list of several tens
  of physical parameters per OMNI record and to
  receive output listing or file of just the
  selected parameters for the time span specified.
• Retired when OMNIWeb became available in 1994.
• _________
• OMNI A multi-spacecraft compilation of hourly
  resolution solar wind magnetic field and plasma
  data, energetic particle data and geomagnetic and
  solar activity indices spanning 1963 to near
  current.
• HRO (High Resolution OMNI 1-min and 5-min)
  spanning 1995 to near current recently made
  available data are as shifted to Earths bow
  shock nose.

9
COHOWeb

• System created at NSSDC to provide access to
  coordinate-system-uniformized and
  content-uniformized hourly solar wind magnetic
  field and plasma data from several deep-space
  spacecraft.
• Provides subset, plot and list capabilities.
• Provides additional analysis capabilities
• Scatter plots and linear regression fits
• Cross correlation coefficients
• Distribution functions, means and medians
• Filtering for any of these by any physical
  parameters
• Spacecraft supported
• Voyager 1 and 2
• Pioneer 10, 11 and PVO
• Ulysses
• Helios 1 and 2

10
Main COHOWeb Web Page
11
COHOWEB scatter plot interface
12
Solar Wind flow speed vs. heliolatitude for first
half orbit leg,1992.0-1995.2
13
Solar Wind flow speed vs. heliolatitude for
second half orbit leg,1995.2-1998.0
14
Solar Wind flow speed vs. heliolatitude for third
half orbit leg,1998.0-2001.4
15
Solar Wind flow speed vs. heliolatitude for forth
half orbit leg,2001.4-2004.2
16
Solar Wind flow speed vs. heliolatitude for fifth
half orbit leg,2004.4-2007.3
17
Systems for supporting distributed data

• NSSDCs NASA Master Directory
• SPDSs Subdiscipline-specific web pages
• Virtual Solar Observatory
• Virtual Space Physics Observatory
• Virtual Heliospheric Observatory
• Virtual Solar Terrestrial Observatory
• Others in development

18
NASA Master Directory

• Started by NSSDC in 1985 to enable data product
  finding.
• Discipline horizon spanned space and Earth
  science data.
• Became space science specific after Global Change
  MD separated in early 1990s.
• Query functionality
• Free text
• Parameters measured
• Time span
• Region sensed, location
• Observatory and instrument
• Person
• Network connections via SPAN/DECnet to underlying
  systems holding data or further information
  (about 20 such systems in 1991).
• Abandoned in mid-1990s due to limited usage and
  difficulties maintaining information current.

19
SPDS Information Systems

• A series of information systems about distributed
  data created in 1996-7 now largely abandoned.
• Intended to assist in the data product finding
  effort.
• MOLD Magnetospheric OnLine Data directory, Jeff
  Reeves, LANL. Hosted by GSFC/SPDF in later
  years.
• Cosmic Heliospheric NSSDC
• ITM SWRI(?)
• MOLD and CH reached a reasonable population
  level and had modest-level usage.

20
Virtual Space Physics Observatory

• Started at GSFC (A. Roberts) in about 2003 to aid
  in data finding, retrieval and display.
• Discipline horizon is space and solar physics.
• 400 space- and ground-based product
  descriptions as of today
• Developed in parallel with SPASE, therefore
  highly SPASE compatible.
• Query functionality Free text
• Parameters
  measured
• Time span and
  resolution, Spectral range
• Region or remote
  object sensed.
• Observatory and
  Instrument
• Person, Repository
• Provides parameter-subsetted data retrieval and
  display functionalities for some data by linking
  to underlying interface (e.g., CDAWeb) via Web
  services or by emulating underlying interface
  (e.g., SPDFs FTPBrowser)

21
Virtual Heliospheric Observatory

• In development at GSFC (Szabo et al) with a
  distributed team.
• Provides current access to ACE, Wind, Genesis,
  Helios, IMP 8, SOHO and MGS field and plasma
  data.
• Search by time, location (ranges in X,Y,Z gse),
  physical observable (field or plasma),
  spacecraft, instrument
• Clickable file-level URLs returned.

22
Virtual Solar Terrestrial Observatory

• In development at NCAR (P. Fox)
• Provides current access to data at CEDAR and
  MLSO
• (Coupling, Energetics and Dynamics
  of Atmospheric Regions)
• (Mauna Loa Solar Observatory)
• Significant emphasis on ontology (developed
  independently of SPASE)
• Search by time, instrument or physical parameter,
  with filtering by domain (CEDAR vs. MLSO) or by
  parameter type.
• Deals with registration requirement for downloads
  of CEDAR digital data.
• Get URLs of files satisfying requirements, with
  option to choose format (FITS, ASCII, JPEG, etc.)

23
Thoughts for current VO activities

• Data modeling and ontology work is very valuable
  for data finding, but to ensure data
  understanding and correct and independent
  usability, more data-descriptive documentation is
  needed than that required to pass SPASE (or
  equivalent) validation tests.
• Whole data file delivery is desired even when a
  (desirable) subsetting option is available,
  especially if the files are immediately usable (I
  have a bias for ASCII files!).
• Data environment should facilitate users access
  to either repositories data access paths or to
  VOs access paths. (e.g., VSPO does this for
  VSO-accessible data)
• Data environment should support data access by
  either people or by computer programs.
• In multi-level multi-nodal data environment, new
  resource-acknowledgement practices will be
  important to each nodes desire for continuation.