SCOS2000 New Generation Packet Archive PARC

1
SCOS-2000 New Generation Packet Archive (PARC)

• Stefano Curcio, PARC Team Leader
• 8 November 2005, EGOS Workshop

2
PARC Overview

• What is the PARC?
• PARC Heritage
• Architectural Drivers
• PAS study and SpaceOps 2004 experience
• PARC Project
• Benefits

• What is the PARC?
• Architectural Drivers
• PARC Architecture
• Performance
• Benefits

3
What is the PARC?

• What is the PARC?
• PARC Heritage
• Architectural Drivers
• PAS study and SpaceOps 2004 experience
• PARC Project
• Benefits

• Packet based archive
• Permanent storage of SCOS-2000 packets
• Central component within the SCOS-2000 MCS
  infrastructure
• Mirror
• all write operations performed on the master
  archive are reflected on a configured slave
  archive.
• Long Term Archive Support
• separation between operational (short-term)
  archive and the mission (long-term) archive
• provides single LTA or Multi-LTA support
• in case of Multi-LTA the different data
  partitions can be located on different LTA
  machines
• Import of old SCOS-2000 History File Archive
  data

4
Architectural Drivers

• High-Performance data Filing and Retrieval
• Continuous storage of data - up to several
  terabytes
• Multi-user access to the data stored while
  concurrently storing new data
• Retrieval ordered on selected time keys
• Multi-Domain support
• Minimise the dependency from the RDBMS COTS
• Low Coupling
• High-Reliability, Scalability, Robustness

5
PARC Architecture
PARC Architecture
Maintenance I/F CORBA

6
Driver Layer
Driver Layer
Input I/F TCP/IP

Archive Layer
Input/Output I/F - CORBA
Output I/F TCP/IP
DBMdriver (Generic Driver)
RAPIDdriver
Load Use
Load Use
Driver Layer
Oracle Plug-in
MySQL Plug-in
Access
Oracle RDBMS
MySQL RDBMS
Unix FileSystem
7
Driver Layer Benefits

• General Facilities
• Provides multiple search index keys.
• Full usage of transactions taking the advantage
  of parallel writing/reading access to the data.
• Full portability of the data stored on different
  platform (SPARC, Intel).
• Use of Driver Layer with Plug-Ins
• Isolates the high-level functionality (Archive
  Layer) from the COTS specific core functionality
  (Driver Layer).
• Driver Layer based on SQL and designed as multi
  RDBMS plug-in MySQL 4.1.12 (using InnoDB engine)
  and Oracle 9i supported as native drivers.
• Extendable to support new COTS just add a new
  RDBMS plug-in

8
Archive Layer
Archive Layer
Input/Output I/F - TCP/IP
Input/Output I/F - CORBA

Archive Layer
retriever
manager
distributor
filer
replicator
Input/Output I/F - TCP/IP
Driver Layer
Plug-in
9
Archive Layer Benefits

• Flexible design oriented to simplify the
  interface to different systems
• Interfaces based on standard TCP/IP and CORBA.
• Concurrent Filing - stores different data streams
  concurrently.
• Filer and Distributor processes deployable in
  different configurations to optimise the
  performances.
• Concurrent Retrieval - high read access to
  multiple users.
• Mirror and Long Term functionalities allows
  deployment of PARC in different configurations
  (e.g. Mirror or LTA)
• Simplified maintenance activities via MMI (Java
  GUI provided)

10
Flexible Deployment of Filers and Distributors

Host1
Distributor Filer TM
Data
Host2
Distributor Filer TC
RDBMS Server
Host3
Distributor Filer EV
Host1
Distributor TM, TC and EV
Data
Host2
RDBMS Server
Filer TM, TC and EV
11
Multi User Retrieval
Multi User Retrieval

Client1
Client2
Client3
Clientn
Data Request
Boss Retriever
Assigned Job
Return Requested Data
Worker Retriever1
Worker Retriever2
Worker Retriever3
12
Mirroring Long Term Archive Support

• Replicators provide PARC Mirror and Long Term
  Archive functionalities
• Mirror provider replication of data on two
  different systems.
• This architecture is used to
• minimise the risk of data loss
• Independent mechanism from RDBMS
• Reduce load on PARC/RDBMS

13
Long Term Archive and Multi-LTA Support

14
PARC Performance initial results

• Test System IBM eServer 326 (AMD Dual Opteron
  2.4Ghz) Linux OS
• Filing Test
• Total data stored 120 Millions packets for total
  of 230GB
• Average Packet filing rate 1000 packets/sec
• Average bytes filing rate 2MB/seconds 16Mb/sec
• Retrieval Test
• Single client retrieval achieved 11,000
  packets/sec _at_ 23MB/sec 184Mb/sec
• Five Clients retrieving in parallel achieved
• an average of 4,300 packets/sec _at_ 9.2MB/sec PER
  CLIENT
• a total of 21,500 packets/sec _at_ 46MB/sec

15
PARC Benefits

• Flexible architecture and design
• Based on RDBMS technology
• RDBMS plug-in based
• High-performance filing and retrieval
• Easy to interface with external systems
• Adaptable configuration to suit different mission
  requirements

16
Any Questions?
Consulting and Technology Technical Excellence
Pragmatic Solutions Proven Delivery