Ocean PEATE

1
Ocean PEATE

• Fred Patt

2
Ocean PEATE Agenda

• Science Team Introduction
• Ocean PEATE Design
• ODPS Design Overview
• Implementation Plan and Schedule
• Documentation
• Issues

3
NPP/VIIRS Ocean Team

• Chuck McClain Ocean Color PI
• Peter Minnett (U. Miami) SST PI
• Gene Feldman Ocean PEATE Manager
• VIIRS Ocean Science Team
• Wayne Esaias
• Barney Balch (Bigelow Lab)
• Mike Behrenfeld (Oregon State U.)
• Janet Campbell (U. New Hampshire)
• Bob Evans (U. Miami)
• Stephane Maritorena (UCSB)
• Norm Nelson (UCSB)
• Dave Siegel (UCSB)
• Ken Voss (U. Miami)
• Menghua Wang (NOAA/NESDIS)

4
Ocean Team (cont.)

• VIIRS Instrument and Calibration Support
• Kevin Turpie
• Bob Barnes
• Gene Eplee
• Gerhard Meister
• Validation Support
• Sean Bailey
• Jeremy Werdell
• Software Support
• Bryan Franz
• Joel Gales
• Data System
• John Wilding
• Systems Management
• Paul Smith

5
NASA/GSFC Ocean Biology Processing Group (OBPG)
Projects

• Sea-viewing Wide Field-of-view Sensor (SeaWiFS) /
  Orbview-2 active
• Moderate-resolution Imaging Spectroradiometer
  (MODIS) / Terra and Aqua active
• Coastal Zone Color Scanner (CZCS) / Nimbus-7
  heritage
• Ocean Color and Temperature Scanner (OCTS) /
  ADEOS-I heritage
• Glory data system prototype 2009 launch
• Aquarius / SAC-D May 2010 launch
• VIIRS / NPP September 2009 launch

6
Ocean PEATE Agenda

• Science Team Introductions
• Ocean PEATE Design
• ODPS Design Overview
• Implementation Plan and Schedule
• Documentation
• Issues

7
Ocean PEATE Design

• The NPP Ocean PEATE will be implemented within
  the framework and facilities of the current NASA
  Ocean Data Processing System (ODPS)
• This system has been successfully supporting
  operational, satellite-based remote-sensing
  missions since 1996, and its capabilities
  continue to evolve and expand to meet the demands
  and challenges of future missions.

8
Element Overview

• Acquire VIIRS RDRs, SDRs, and Ocean EDRs from the
  SD3E and ADS/CLASS
• Assess the quality of the NPP Ocean EDRs for
  accomplishing NASAs climate research
  requirements
• Provide suggested algorithm improvements to the
  IDPS via the Project Science Office Element
  (PSOE)
• Process selected data subsets in support of
  Evaluation and Validation activities

9
Changes since PDR

• Established interface with RSMAS (U. Miami) for
  SST Validation using MODIS validation for proof
  of concept.
• Expanded and elaborated on xDR evaluation
  methodologies (as presented at July Peer Review).

10
Ocean PEATE Interface Diagram
Analysis Results, Proposed Algorithm Updates
xDRs, IPs, Ancillary Data
Management Direction
xDRs, IPs, Ancillary Data (if unavailable from
SD3E)
OceanPEATE
Pre-flight Algorithms, Data, Info
Software, Data
Alternate Ancillary Data
xDR Eval. Results, Algorithm Updates
Calibration Updates and Evaluations
Interaction
In Situ Data
In Situ Data
Algorithm Updates, Test Requests Results
Matchups
11
Ocean PEATE External Interfaces (1 of 2)

• SDS Science Data Distribution and Depository
  Element (SD3E)
• Provides NRT access to raw data
• Primary source of RDRs
• Provides selected SDRs and EDRs
• SDS Integration and Test System Element (ITSE)
• Build and test updates to operational code in
  mini-IDPS
• Run tests on selected data per request of PEATE
• Archive Distribution Segment (ADS)
• Primary source for archived data
• xDRs, IPs, Ancillary Data, Operational
  Algorithm/Source Code and Calibration Products
• Ancillary Data Providers (ADP)
• Provides alternate ancillary data sets (e.g.,
  ozone, meteorological data sets)
• CasaNOSA
• Serves as the NPP pre-flight repository of
  Government held data for distribution to
  Government user teams
• Place to acquire pre-launch NPP algorithms and
  supported data files

12
Ocean PEATE External Interfaces (2 of 2)

• NASA VIIRS Ocean Science Team (VOST)
• Coordinate activities with PEATEs and PSOE on xDR
  and recommended algorithm improvements. Supports
  Independent Calibration Validation Activities
• NPP Instrument Calibration Support Element
  (NICSE)
• Provides alternative calibration LUTs and
  recommended improvements to calibration
  algorithms
• PEATE provides results of LUT and algorithm tests
• Project Science Office Element (PSOE)
• Provides management direction
• Accepts algorithm update recommendations
• SeaBASS/ODPS
• Provides Ocean Color in situ data
• RSMAS/U. Miami
• Provides SST in situ locations
• PEATE provides SST EDR matchups
• Ocean Science Community
• Relies on Ocean PEATE to provide evaluation
  products and results

13
Assumptions Role of ITSE

• The Ocean PEATE will rely upon the SDS-provided
  ITSE (mini-IDPS) to serve as the testbed for
  algorithm evaluation and potential improvements
  and will not attempt to duplicate efforts by
  running the operational code within the PEATE
  environment.
• The staff of the ITSE will have the ability to
  modify any operational code as per Ocean PEATE
  specification and run the code within the
  operational environment against a set of
  PEATE-specified test data sets.
• The ITSE will receive and install the most
  current version of the code and executable
  programs for all operational IDPS software
  required to produce NASA-evaluated EDRs, starting
  from RDRs.
• The ITSE will maintain all processing code under
  configuration control. The PEATEs will have
  access to the configuration controlled code.

14
Ocean PEATE Support for IDPS Software

• The Ocean PEATE will maintain a working knowledge
  of the VIIRS SDR code and the EDR code for the
  Ocean Color and SST EDRs.
• The Ocean PEATE will maintain an understanding of
  the relevant IP characteristics, but will
  (initially) assume that the IPs are properly
  generated.
• The Ocean PEATE will design changes to the code
  in the ITSE for the purpose of algorithm
  improvement or problem resolution.
• The Ocean PEATE will develop appropriate test
  cases and request runs to verify and evaluate the
  changes.
• The Ocean PEATE will provide recommended changes
  to the PSOE, including a description of the
  proposed change, effect on the EDR performance,
  and the evaluation of the test runs.

15
IDPS VIIRS Ocean EDR Data Flow
Processing Module
VIIRS Product
VIIRS RDR
Dynamic Ancillary Data
Previous VIIRS Gridded Products
Static Ancillary Data
Previous VIIRS Gridded Products
NAAPS TOD
MODIS Land/Water Mask
NCEP Geopotential Height
Ancillary Files
Ancillary Files
DEM
NDT
16
IDPS VIIRS Ocean EDR Data Flow
VIIRS Geolocation
VIIRS SDR
Processing Module
VIIRS_SDR_01 RDR Decompression
VIIRS_GEO_01 Geolocation
VIIRS_SDR.IM 375m SDR
VIIRS_SDR.MOD 750m SDR
VIIRS SDR
VIIRS Product
VIIRS RDR
Dynamic Ancillary Data
Previous VIIRS Gridded Products
Static Ancillary Data
Previous VIIRS Gridded Products
NAAPS TOD
MODIS Land/Water Mask
NCEP Geopotential Height
Ancillary Files
Ancillary Files
DEM
NDT
17
IDPS VIIRS Ocean EDR Data Flow
VIIRS Geolocation
VIIRS SDR
Processing Module
VIIRS_SDR_01 RDR Decompression
VIIRS_GEO_01 Geolocation
VIIRS_SDR.IM 375m SDR
VIIRS_SDR.MOD 750m SDR
VIIRS SDR
VIIRS Product
VIIRS RDR
Dynamic Ancillary Data
Previous VIIRS Gridded Products
Static Ancillary Data
Previous VIIRS Gridded Products
VIIRS_GD_08 750m Granulation
NAAPS TOD
VIIRS_GD_25 NAAPS Granulation
MODIS Land/Water Mask
VIIRS_GD_27 L/W Mask Granulation
NCEP Geopotential Height
Ancillary Files
Ancillary Files
VIIRS_GD_11 Ancillary Profile
VIIRS_GD_09 GFS Granulation
ALL_GD_01 Time Interpolation
VIIRS_GD_28 Surface Pressure Adjustment
VIIRS_GD_12 Bathymetry Granulation
DEM
VIIRS_GD_13 Temperature Granulation
NDT
18
IDPS VIIRS Ocean EDR Data Flow
VIIRS Geolocation
VIIRS SDR
Processing Module
VIIRS_SDR_01 RDR Decompression
VIIRS_GEO_01 Geolocation
VIIRS_SDR.IM 375m SDR
VIIRS_SDR.MOD 750m SDR
VIIRS SDR
VIIRS Product
VIIRS RDR
Dynamic Ancillary Data
Previous VIIRS Gridded Products
Static Ancillary Data
Previous VIIRS Gridded Products
VIIRS_GD_08 750m Granulation
NAAPS TOD
VIIRS_GD_25 NAAPS Granulation
MODIS Land/Water Mask
VIIRS_LN_06 Active Fires
VIIRS_CM_01 Cloud Mask
VIIRS_GD_27 L/W Mask Granulation
NCEP Geopotential Height
Ancillary Files
Ancillary Files
VIIRS_GD_11 Ancillary Profile
VIIRS_GD_09 GFS Granulation
ALL_GD_01 Time Interpolation
VIIRS_GD_28 Surface Pressure Adjustment
VIIRS_GD_12 Bathymetry Granulation
DEM
VIIRS_GD_13 Temperature Granulation
NDT
19
IDPS VIIRS Ocean EDR Data Flow
VIIRS Geolocation
VIIRS SDR
Processing Module
VIIRS_SDR_01 RDR Decompression
VIIRS_GEO_01 Geolocation
VIIRS_SDR.IM 375m SDR
VIIRS_SDR.MOD 750m SDR
VIIRS SDR
VIIRS Product
VIIRS_SN_02 Ice Quality
VIIRS RDR
Dynamic Ancillary Data
Previous VIIRS Gridded Products
Static Ancillary Data
VIIRS_CL_01 Cloud Optical Properties
Previous VIIRS Gridded Products
VIIRS_ST_02 Surface Temp
VIIRS_GD_08 750m Granulation
NAAPS TOD
VIIRS_SN_03 Ice Concentration
VIIRS_GD_25 NAAPS Granulation
MODIS Land/Water Mask
VIIRS_LN_06 Active Fires
VIIRS_CM_01 Cloud Mask
VIIRS_GD_27 L/W Mask Granulation
NCEP Geopotential Height
Ancillary Files
Ancillary Files
VIIRS_ST_01 Sea Surface Temperature
SST EDR
VIIRS_GD_11 Ancillary Profile
VIIRS_GD_09 GFS Granulation
VIIRS_AR_01 Aerosol Type
ALL_GD_01 Time Interpolation
VIIRS_GD_28 Surface Pressure Adjustment
VIIRS_OC_01 Ocean Color / Chlorophyll
VIIRS_GD_12 Bathymetry Granulation
OCC EDR
DEM
VIIRS_GD_13 Temperature Granulation
NDT
20
ODPS MODIS Ocean Product Data Flow
Processing Module
MOD_PR01 Level-0 to 1A
MODIS Product
Dynamic Ancillary Data
MODIS Level-0
MODIS Level-1A
Static Ancillary Data
MODIS 1 km Level-1B
MOD_PR03 Geolocation
MOD_PR02 Level-1A to 1B
MODIS SST
Platform ATTEPH Data
MSl12 Level-1B to 2
MODIS Geolocation
MODIS Ocean Color
MET Ancillary Data
Ozone Ancillary Files
Land/Water Mask
21
Evaluation vs. Product Level

• Level-1 (SDR) Evaluations
• Onboard calibration analyses
• Vicarious calibration
• Level-2 (EDR) Evaluations
• Matchup analyses
• Residual detector (striping) and scan (RVS)
  dependence
• Level-3 Product Evaluations
• Sensor cross-comparisons
• Algorithm comparisons
• Temporal anomaly evaluations

22
SDR Example Vicarious Calibration
23
EDR Example Scan and Detector Dependence
24
Level-3 Example Zonal Cross-Comparisons
25
Ocean PEATE Agenda

• Science Team Introductions
• Ocean PEATE Design
• ODPS Design Overview
• Implementation Plan and Schedule
• Documentation
• Issues

26
ODPS Design Overview

• Fully automated, distributed data system for
  acquiring, processing, archiving, and
  distributing scientific data
• Highly scalable
• Easily adaptable to support multiple concurrent
  missions
• Graphical user interfaces for controlling and
  monitoring system functions and activity
• Non-platform specific

27
ODPS Design Philosophy

• Building-Block approach
• Programs are usually small and do one thing well
• Programs are less complex and subsequently easy
  to maintain
• Promotes reuse
• Programs loosely coupled so testing and
  production can be done in the same environment
• Adopt basic standards
• ANSI, POSIX, C9x
• Use existing technology when possible
• Exit statuses indicate successful or failure
  conditions

28
Components and Subsystems
29
Components and Subsystems (1 of 2)

• RDBMS is the primary element that manages all
  system activity
• Generic core databases support system
  infrastructure and non-mission-specific functions
• Mission databases catalogue products and house
  mission-specific data and procedures
• High level of reuse possible for similar
  missions e.g., MODIS Aqua/Terra, SeaWiFS, CZCS,
  and OCTS are all ocean-color missions and have
  similar product suites and requirements

30
Components and Subsystems (2 of 2)

• Relational Database Management System (RDBMS)
  supports all of the system components
  (subsystems)
• VDC/Scheduler is the primary controlling module
  within the system
• Other subsystems are independent modules, yet
  rely on the VDC/Scheduler for some their
  functions
• Archive Device Manager (ADM)
• Data acquisition and ingest
• Level-3 Scheduler
• File migration and management
• Data distribution

31
ODPS COTS and Freeware

• Linux OS (CentOS 4.x)
• Solaris OS
• Sybase RDBMS
• Subversion (source code management)
• Pro-active DBA
• Interactive Data Language (IDL)
• Generic Mapping Tool (GMT)
• Netpbm (graphic image toolkit)
• HDF5 Library
• Languages C, PERL, SQL

32
ODPS Architecture Hardware

• Processing Servers
• Intel-based dual Xeon / AMD-based dual Opteron
• 8 GB RAM
• Five 72 GB SCSI drives
• Storage Servers
• Intel-based P4 / AMD-based single Opteron
• 1 GB / 2 GB RAM
• 1.5 TB IDE RAID 5 (3ware) / 9.6 TB SATA RAID 6
  (Areca)
• 2 hot spare drives per RAID5
• Database Server
• Sun V880
• 8-16 GB RAM
• 6-12 70 GB SCSI HDD

33
ODPS Current Components
34
Building 28 Room W220 Computing Facility
35
Reliability and Redundancy

• Critical components (database server, network
  systems) have full maintenance contracts to
  ensure rapid response to problems
• Multiple-server components (ingest, processing,
  storage, distribution) have substantial
  redundancy to maintain full capability spares
  maintained for rapid replacement.
• Testing nodes are separated from mainstream
  production components.

36
Technology Refresh

• Hardware technology advances (CPU, storage) are
  continuously monitored to select new components
  for evaluation.
• Typical upgrade threshold is a doubling in
  capacity (18 months).
• Two generations of hardware are generally in use.
• Candidate components are procured, installed in
  testing cluster and rigorously evaluated in a
  production-like environment.
• Following successful evaluation, multiple copies
  are procured, installed, tested and swapped in
  for older components.
• ODPS design allows new components to be rapidly
  added to resource tables without interrupting
  system operations.
• Performance of new components is closely
  evaluated following installation in operations
  environment.
• Critical components are run in parallel with
  existing system to ensure reliability under
  production loading.

37
Ocean PEATE Agenda

• Science Team Introductions
• Ocean PEATE Design
• ODPS Design Overview
• Implementation Plan and Schedule
• Documentation
• Issues

38
Ocean PEATE Gap Analysis (1 of 2)

• Acquire, ingest and catalog NPP VIIRS data
  products RDRs, SDRs and Ocean EDRs (Data
  Acquisition Ingest, Device Manager and File
  Migration and Management).
• Status Development of acquisition and ingest
  scripts underway based on sample products in
  SD3E.
• Process selected Ocean EDRs (SST and OCC) to
  Level-3 to support data product and algorithm
  evaluations (Level-3 Scheduler, VDC and Level-3
  binner).
• Status Prototype Level-3 processing has been
  demonstrated using sample IDPS Build 1.4 OCC and
  SST EDRs.
• Perform VIIRS OCC EDR matchups with SeaBASS Ocean
  Color in situ data (extract code).
• Status Pending completion of acquisition and
  ingest capabilities.

39
Ocean PEATE Gap Analysis (2 of 2)

• Incorporate VIIRS SDR processing for vicarious
  calibration analysis. ()
• Status Pending availability of sample SDRs in
  Build 1.5 format
• Produce VIIRS proxy data using VOST-developed
  software (VDC/Scheduler).
• Status Prototype geolocation and EDR simulation
  developed to test Level-3 binning of EDRs.
• Acquire SST in situ data from RSMAS and perform
  matchups with SST EDRs ()
• Status Pending final MOU
• Support browse and distribution of data products
  for team members (Data Distribution).
• Status Pending completion of acquisition and
  ingest capabilities.
• () New PEATE capability identified since PDR

40
Prototype Level-3 Processing

• EDR to Level-3 processing prototype completed
  using IDPS Build 1.4 EDRs.

41
Existing Software Reuse

• ODPS Components
• Database
• VDC/Scheduler
• Data Acquisition and Ingest
• Level-3 Scheduler
• File migration and management
• Archive Device Manager
• Data distribution
• Level-2 multi-mission software (vicarious
  calibration)
• Level-3 multi-mission software (long-term trends
  and comparisons
• Level-2 to Level-3 comparison software (residual
  sensor errors)
• SeaBASS (in situ data management)
• Matchup/extraction software

42
Basis of Estimate (1 of 2)

• Acquire, ingest and catalog NPP VIIRS data
  products
• 200 LOC (combined UNIX shell, SQL, C) per new
  product, for each of the four VIIRS products
• Process Ocean EDRs (SST and OCC) to Level-3
• 300 C LOC to add VIIRS Ocean EDR input to
  existing binning software (L2bin)
• 300 SQL LOC per temporal range (10 ranges total)
• 200 shell LOC for all ranges
• Perform VIIRS EDR matchups
• 100 C LOC 10 PERL LOC to add VIIRS Ocean EDR
  input to existing extraction software

43
Basis of Estimate (2 of 2)

• Process SDRs for vicarious calibration analysis
• 1000 C LOC to add VIIRS SDR input to existing
  Level-2 processing software (MSl12)
• Produce VIIRS proxy data
• 100 shell LOC for each processing stage
• Support browse and distribution of data products
• 3000 C LOC to implement browse image generation
• 100 PERL LOC to add VIIRS products to existing
  browse and order web site

44
Ocean PEATE Data Storage Estimate
Data Type Daily 1 Year 5 Years
RDR 150 GB 53.5 TB 268 TB
SDR (M-band) 242 GB(2) 8.6 TB(1,2) 43 TB(1,2)
OCC EDR 84 GB 3 TB(1) 15 TB(1)
SST EDR 19 GB 0.7 TB(1) 3.4 TB(1)
Inter. Products 70 GB N/A N/A
Ancillary Data 0.1 GB .04 TB .2 TB
Total 565 GB 66 TB 330 TB

• Assumptions
• Long-term storage is sized for 100 of RDRs and
  10 of SDRs and EDRs
• SDR volume includes geolocation

45
New Hardware for the Ocean PEATE

• 7 Storage Servers _at_ 9.6 TB first-year VIIRS
  data storage
• Additional servers acquired post-launch to handle
  years 2 5
• No new processing or network capacity required
  technology refresh cycle to be continued within
  the ODPS as described.

46
Ocean PEATE Build Schedule

• Build 1 (L-18 months)
• All interfaces fully implemented and tested
• Verify initial versions of operational code
  ported and running in ITSE
• L-3 product code developed and tested
• Prelaunch VIIRS test data storage and SDS
  interface testing support with existing ODPS
  storage capacity
• Initial test products generated for review by
  VIIRS Ocean Science Team

• Build 2 (L-12 months)
• Routine exercise of interfaces to acquire proxy,
  surrogate (Aqua?) and/or simulated data
• Verify pre-launch version of operational code
  running in ITSE
• Browse and distribution capability developed and
  tested
• Test products routinely acquired as available and
  posted for access by VIIRS Ocean Science Team
• Data storage for one year

47
Ocean PEATE Development Schedule
48
External Schedule Dates

• C3S SDS Integration
• October 2008
• IDPS SDS Integration
• October 2008
• NSIPS SDS Integration
• October 2008
• Functional Thread Test Dates
• FTT8 A
• Ingesting xDRs October 2008
• FTT8 B
• Validating xDRs October 2008
• NPP Compatibility Tests
• NCT2C - December 2007
• NCT3 - September 2008

49
Ocean PEATE Testing

• ODPS Testing Philosophy
• Test as you run.
• Test early, test often.
• Phased approach to testing get one thing working
  and move on to the next step until entire
  end-to-end stream is operational.
• Testing VIIRS support to be done within
  operational ODPS environment, with products
  clearly identified for separation from
  operational product stream.
• Testing stages
• Interfaces
• Internal functionality (processing, evaluation)
• End-to-end
• Test data sources
• VIIRS proxy data
• VOST simulation
• MODIS products (e.g., initial SST validation)
• Availability of official test data is an issue.

50
Ocean PEATERequirements Implementation

• 95 of requirements (19) implemented by Build 2
  add additional storage capacity before launch

51
Sustaining Operations and Maintenance

• ODPS runs 24x7 with on-site support 8x5.
• Support is shared across all projects.
• Staffing needs are covered by existing OBPG
  support personnel.

52
Ocean PEATE Staffing Levels

• All staffing needs will be met with existing OBPG
  staff
• Increases will be accommodated as other projects
  (e.g., MODIS Aqua) wind down.

53
Documentation

• Ocean PEATE Level 4 Requirements and Operations
  Concept
• ICDs
• SD3D to PEATEs
• CLASS?
• MOU with RSMAS (TBS)
• ODPS Project Data Management Plan (draft)
• OCDPS Risk Management Plan
• Network IT Security Plan

54
Historical Milestones

• PEATE selected January 2004
• PEATE Peer Review November 30, 2006
• SDS PDR September 19, 2006
• EDR Evaluation Peer Review July 18, 2007

55
Issues/Challenges/Concerns

• Limits of available test data
• Ability to adequately test and flow data across
  interfaces
• Verification of EDR evaluation processes
• Extent of prelaunch end-to-end (observatory
  ground system) data flows
• ADS to PEATE interface
• Complexity of IDPS internal processes and data
  flows
• Version synchronization of operational products
  with IDPS software in the ITSE
• Ability to diagnose software and algorithm
  problems in the ITSE
• Bandwidth of NSIPS/SD3E interface
• Sole source of IPs including OBC products
• Separation of spacecraft diary from science RDRs
• Requires tracking gt4000 small granules/day

56
Conclusion

• Ocean PEATE will meet all requirements
• Next Up Land PEATE Ed Masuoka
• NPP_SDS_Land.ppt

57

Backup Slides
58
Ocean PEATE Level 4Requirements Allocation (1 of
2)
SDS Rqmt Number SDS Rqmt Title Subsystem
3 Ocean PEATE N/A
3.1 Acquire RDRs, SDRs, and Ocean EDRs N/A
3.1.1 3.1.2 3.1.3 3.1.4 3.1.5 3.1.6 3.1.7 3.1.8 Acquire and Ingest VIIRS RDRs from SD3E Submit Requests to SD3E for Subsets of SDRs and EDRs Acquire and Ingest SDRs and EDRs from SD3E Submit Requests to ADS/Class for SDRs and EDRs Acquire and Ingest SDRs and EDRs from ADS/CLASS Provide Storage for RDRs, SDRs and EDRs Support Cataloging, Searching, Ordering and Distribution VDC/Scheduler Data Acquisition Ingest File Migration and Management Device Manager Data Distribution PEATE Personnel
3.2 Assess the Quality of the NPP Products N/A
3.2.1 Assess the Quality of the Ocean EDRs N/A
3.2.1.1 3.2.1.2 3.2.1.3 3.2.1.4 3.2.1.5 3.2.1.6 3.2.1.7 Ground Truth Validation with In Situ Data Cross-comparisons with Concurrent Satellite Data Sets Comparisons with Climatological Data Sets Assessments of Internal Consistency Flagging and Masking Assessment Prelaunch Algorithm Assessment VDC/Scheduler Level 3 Scheduler Level 3 Binner SeaBASS Matchup/Extraction Software
59
Ocean PEATERequirement Allocation (2 of 2)
SDS Rqmt Number SDS Rqmt Title Subsystem
3.2.2 Support VOST in Assessing Quality of the VIIRS SDRs N/A
3.2.2.1 3.2.2.2 3.2.2.3 Assessment of long-term radiometric stability Assessment of instrumental corrections Analysis of prelaunch test results VDC/Scheduler Level 2 Processing Software PEATE Personnel
3.3 Provide Suggested Algorithm Improvements
3.4 Process Selected Data Subsets N/A
3.4.1 Interface with ITSE for Processing Data PEATE Personnel
3.4.2 Acquire Processing Code from ITSE PEATE Personnel ODPS CM
3.4.3 Acquire SDRs and EDRs from ITSE VDC/Scheduler Data Acquisition Ingest File Migration and Management Device Manager Data Distribution
60

ODPS Design Details
61
RDBMS
Components and Subsystems RDBMS
Goal Isolate RDBMS from system software
To use a different RDBMS vendor, swap in a new
Database Services Layer
RDBMS
Vendor Client Library
Vendor Library Module
Database Services Layer
C Interface Functions
Perl DBI Module
Perl Scripts
C Programs
62
VDC/Scheduler
Subsystems VDC/Scheduler

• C program with supporting database procedures
• Runs in a daemon-like state on every processing
  server
• Primary system element responsible for
  coordinating most of the system activity
• Monitors task records in a to-do list database
  table
• Runs tasks according to defined task attributes
• Standard job-shell interface allows new programs
  to be quickly adapted as tasks for Scheduler
  control

63
VDC/Scheduler
Subsystems VDC

• Highly scalable, distributed infrastructure for
  concurrent processing of serial streams (e.g., L0
  ? L1A ? L1B ? L2)
• Suite of C programs with supporting database
  procedures
• Uses recipes to encapsulate data-specific
  processing schemes, parameters, and
  pre-processing rules
• Virtual Processing Units (VPUs) serve as distinct
  distributed processing resources
• VPUs dynamically allocated based on available
  time and current OS load
• Comprehensive, user-defined processing priorities

64
VDC Ancillary Data Stager
VDC Pre-processing Rule Manager

• Runs in a daemon-like state
• Monitors entries in the processing queue and runs
  the rule proceduresthat are associated with the
  rule scheme for each recipe
• Updates queue-entry status when all rule
  procedures return successfully
• Governed by currently configured processing
  priorities

65
VDC MakeVDC
VDC MakeVDC

• Selects processing-queue entries that have passed
  pre-processing-rule requirements
• Generates custom VDC job files from templates
  according to configured priorities
• Runs as a Scheduler task, so it can easily be
  configured to run as often as needed to keep the
  VDC queue full

66
VDC Engine
VDC Engine

• Function performed by the VDC/Scheduler module
• Each instance actively competes for jobs that it
  is allowed to run, based on priority, length of
  time in the queue, and user nudges
• Monitors and manages processing resources
• Initializes processing streams
• Invokes recipe steps and monitors step-execution
  time
• Handles operator-requested stream actions

67
Archive Device Manager
Subsystems Device Manager (DM)

• User defines logical pools of storage devices
• Processes request a device in a specific pool
• DM returns information for a storage device in
  the requested pool
• If auto-cycling is enabled, the DM time-stamps
  the record for the selected device, so a
  different device within the pool will be selected
  for the next request
• When a device in a pool exceeds a user-configured
  usage threshold, the device is no longer eligible
  to be selected
• Disk-monitor process polls all devices
  periodically to record usage statistics and
  invoke threshold handlers

68
Data Acquisition and Ingest (1 of 2)
Subsystems Data Acquisition and Ingest

• Data types and sources are described in the
  database
• Active, passive, and periodic notification
  methods
• Active method scans remote systems for new files
  and populates the ingest queue
• Passive method waits for arrival of e-mail
  messages describing type and location of new file
  and populates the ingest queue
• Periodic method schedules transfers of files at
  user-specified intervals

69
Data Acquisition and Ingest (2 of 2)
Subsystems Data Acquisition and Ingest

• File transfers handled by ingest daemons and
  Scheduler tasks
• FTP, RCP, SCP, WGET transfer protocols supported
• Generic script handles the file transfer and then
  hands off to data-specific post-ingest scripts

70
Level-3 Scheduler
Subsystems Level-3 Scheduler

• Responsible for scheduling processing for level-3
  composite products
• Runs as a Scheduler task
• Configuration is database driven
• Mission-specific stored procedures are invoked to
  identify input files for a composite product

71
File Migration and Management
Subsystems File Migration and Management

• Responsible for compressing files and migrating
  them to their various destinations
• Event- or time-based actions
• Ex1 compress files after they are QCd
• Ex2 remove files that are more than N days old
• Ex3 mirror files upon creation
• Queries associated with each action are run
  periodically by a Scheduler task to select files
  that are eligible for some type of migratory
  action and populate a migration queue
• Command-line queuing for file removal and delayed
  copies
• Migration daemons query the migration queue,
  perform registered actions on the files, and
  update catalog tables

72
Data Distribution
Subsystems Data Distribution

• Interactive, web-based Data Ordering System,
  currently supporting SeaWiFS and MODIS Aqua
• Data Subscription System, currently supporting
  MODIS Aqua, allows users to define region and
  products of interest
• Order and Subscription Manager Daemons monitor
  the order and subscription queues and stage files
  on FTP servers (stage rate 12 GB/hr)
• Near-real-time data extraction and image support
• Web-CGI applications that allow users to view and
  update their orders and subscriptions

73
New Data/Source Acquisition and Ingest

• Insert DB records for new data-source servers and
  data types
• Compose data-specific post-ingest scripts
• Configure ingest daemons if that method is going
  to be used for any of the new data types
• Define archive-device pools for product storage

74
New Data Product Cataloging

• Insert record into the core tables (catalog DB)
  that describe the mission and products
• Create mission specific database and objects,
  reusing objects from existing mission databases
  where applicable
• Compose a program to provide geographical L1
  meta-data information including granule start and
  stop times, day-night flag, and geographic
  coordinates, e.g., MSl1info
• Compose functions for DB-metaload program, so
  meta-data files can be read and product tables
  can be populated
• Configure file migration and management actions
  for new mission data

75
New Data Processing Streams

• Define a recipe for each distinct serial
  processing stream
• Insert records for each recipe and each recipe
  step
• Compose a job template for each recipe
• Compose a ancillary-selection procedure for each
  recipe that requires ancillary data
• Compose scripts for each science program
  associated with the new mission's data
• Insert record in recipe-constraints table for
  each processing host allowed to run a recipe
• Compose AP-load procedure for each base data type
  that can be processed with a recipe
• Update the reproc program to support each base
  data type that has an AP-load procedure
• Configure L3-Scheduler for desired composite
  processing

76
VIIRS Ocean Level-3 Processing

• Develop data input routines to support processing
  of VIIRS Ocean EDRs using existing multi-mission
  Level-3 binning software

77
New Data Product Distribution

• Update Subscription CGI to support new mission
  data
• Compose match-subscription procedure
• If data extraction and mapping is to be
  supported
• Compose extraction and mapping programs
• Compose match-XM-requests procedure
• Modify XM CGI to support new mission products
• Compose browser capabilities for new mission
  products
• Provide FTP access to new mission products