MultiMission Facility Infrastructure MMFI

1
Multi-Mission Facility Infrastructure (MMFI)

• Gian Maria Pinna

DAI meeting Montreal 12 May 2004
2
RATIONALE

• In accordance with the Oxygen concept for
  harmonized ESA E.O. Ground segment
  implementation
• In line with the specific requirement for on line
  Archive data access
• Spin-off project to demonstrate the validity of
  the concepts applicable for future GMES (Global
  Monitoring for Environment and Security)
  challenging requirements
• Basic implementation principles
• Maximum re-use of already proven elements
• Generic ground segment, mission independent
• Standardization of products and formats across
  missions
• Streamlining of common function and
  standardization of interfaces
• Identification of mission specific element and
  common elements
• Adoption of strategy for all future ESA handled
  missions
• Extension of concept to European National
  missions
• Harmonization and rationalization of archives

3
Past Approach

• The implementation of the major missions Ground
  Segment has been based up to now on a specific
  end to end solution for each single mission (no
  reuse), mainly due to the internal ESA project
  management structure
• Only at a later stage, once the mission has
  reached stability, changes may start for reaching
  a minimum rationalization and harmonization level

4
Ground Segment Decomposition
5
MMFI background

• The MMFI architecture is based on the logical
  model developed under the ADAR (Advanced Data
  ARchive) project started under the ESAs General
  Studies programme
• The ADAR project aimed at the following main
  goals
• Define a conceptual model for the ESA EO Payload
  Ground Segment
• Identify the multimission infrastructure elements
  that are potential candidates to be reused in
  future ground segments
• Identify multimission infrastructure elements
  that are potential candidates for future
  development or procurement
• Develop a prototype environment where the main
  concepts and systems are demonstrated

6
ADAR Logical Model (OAIS-based)
7
From the ADAR Model to an actual MMFI
implementation

• The ADAR Logical model streamlines the complexity
  of a ground segment by
• Decreasing the number of interfaces
• Dividing the functions in few common
  infrastructure elements
• The next step is the implementation of an actual
  architecture, by
• Adoption/adaptation of already existing reusable
  elements
• Development of new missing elements
• The final step is the plugging in of the ESA
  ground segments into the MMFI, by
• Revision of present existing requirements (e.g.
  mission specific, like Envisat)
• Adaptation or redevelopment of existing
  interfaces (e.g. SeaSHARK, QC systems)
• Definition of generic requirements for newly
  developed systems (e.g. PGS-ERS)

8
MMFI Elements
9
GFEMMFI Data Ingestion
GFE

• The Ingestion component is the entry point of the
  new products into the facility.
• Its main services are
• Registration of data producers
• Registration of product types
• Ingestion of products (Product validation,
  metadata extract, browse generation)
• Facility data coherency (between the Data
  Management component and the Storage component)
• It relies mainly on two components the GFE
  (Generic Front End) core from EADS and the DRS
  (Data Request Server) from GAEL

10
Data Request Server
GFE
Services supported by DRS in the GFE are

• Product inspection
• Metadata extraction
• Metadata inspection
• Metadata formatting (for Multi Mission Master
  Catalogue, XML-based)
• Browse generation

11
AMS architecture
AMS
AMS Clients
Archive Administration HMI
CORBA Stub
SatSTORE API
CORBA Requests
Product retrieval
Product archiving
SatSTORE server
Product Subsetting
Archive Administration
Requests Management
Foreign Tape Inventory
Resource availability
VFS requests
AMASS API
Foreign Tape
AMASS/FTIF
External Tape Import
DAS
on-line
near-line AML library
off-line
12
AMS services
AMS

• Product Archiving
• Product Retrieval (with Sub-setting)
• Product Status
• List of Products
• Advanced List of Products
• Multiple Product Retrieval
• Product Update
• Product Removal
• Product Move
• Multiple Product Move
• Multiple Product Copy

13
PFD Components
PFD
14
PFD functions
PFD

• PFD Central Server
• Dissemination Order Processing of a set of
  products reformatting with or without subsetting
  (geographical or band subsetting) of input
  products
• Reformatting subsetting with formatters,
  integrated as plugins or based GAELs Data
  Request Broker (DRB)
• Compression performed with compressing algorithms
  integrated in PFD as plugins UNIX-Compress,
  PKZIP, GZIP and LuraWave
• PFD Network Server
• Creation of a random ftp account for each order
• Transfer of the output products and of the
  shipping note into the ftp account by ftp
  throughout the firewall
• Delivery of an e-mail to the End User to inform
  him of the availability of the ordered products
  the e-mail contains the ftp account definition
  allowing the End User to access it

15
Wavelet/JPEG2000 compression
PFD
ERS SAR
16
PFD
PFD interfaces
17
POHProduct Order Handling
POH

• Receives Product Orders from User Order Handling
  system (MMOHS)
• Enables manual input / update of Product Orders
• Validate Product Orders (against data archive,
  processor availability...)
• Schedules the Processors, submits processing
  orders and gets back production statuses
• Reports to User Order Handling system on Product
  Order status

18
POH Architecture
POH
Product Orders Submission Status rep. I/F
POH
Product Orders D.B.
Generic Core System I/F
Generic Core System I/F
Generic Core System I/F
etc
Core System X
Core System Y
Core System Z
19
POH
20
MMFI Deployment

• Tape robotics procured for 6 centres
• STK Powderhorn 9310, with 2000 slots (400TB each,
  max 1.2 PB)
• 8 TB NAS for data storage on line, accessed via
  AMS
• 3.2 TB NAS for MMFI central data cache
• Fast disk arrays for AMASS cache (800 GB) under
  procurement
• PFD installation completed
• GFE replicas procurement on-going
• POH HW procured, under installation at the remote
  centres

21
MMFI Related Activities/ProjectsHARM(Historical
Archive Rationalization and Management)

• Transcription HWSW from old media (D1, DLT) to
  AMS (StorageTek T9940B based)
• Orbits stitching (reducing overlap between
  acquisition stations)
• Rationalization in thematic archives
• SAFE (Standard Archive Format for Europe)
• Based on previous experiences (Wilma, CEOS ICF,
  Cryosat)
• XML metadata
• Self-descriptive
• Pointers to binary data structures

22
MMFI Related Activities/ProjectsFEOMI(Facility
Evolution into an Open Multimission
Infrastructure)

• New project to start mid 2004 (18 months
  duration)
• Porting of Envisat, ERS and TPM ground segments
  to MMFI
• Upgrade of mission specific elements to interface
  with MMFI
• Phase out of mission specific elements where same
  functionality can be ensured by MMFI elements
• Will bring to operations costs reduction starting
  from 2006

23
MMFI Related Activities/Projects Network
rationalization and upgrade (LAN, WAN)

• Production LAN upgrades to Gbit
• Networking our facilities with 34 Mbps links via
  European academic network
• To be used for data circulation inside ground
  segments
• Also to be used to ensure better service to users
  for access to products on-line (automatic and
  transparent re-routing)

24
MMFI Related Activities/ProjectsGRID Activities

• New 64 bi-processors nodes system received
• To be used for mission reprocessing
• Interface with MMFI for data retrieval and
  products archiving