HMI-AIA Joint Science Operations Center Science Data Processing a.k.a. JSOC-SDP

1
HMI-AIA Joint Science Operations Center Science
Data Processing a.k.a. JSOC-SDP
Overview
2
HMI AIA JSOC Concept
GSFC
White Sands
LMSAL
MOC
SDOGS DDS
HMI AIA Operations
Stanford
HMI JSOC Pipeline Processing System
Redundant Data Capture System
Quicklook Viewing
10-Day Archive
Local Archive
AIA Analysis System
High-Level Data Import
Offline Archive
Data Export Web Service
World
Science Team Forecast Centers EPO Public
3
JSOC Dataflow Rates
4
JSOC-SDP Major Components
5
JSOC DRMS/SUMS Basic Concepts

• Each image is stored as a record in a data
  series.
• There will be many series e.g. hmi_ground.lev0
  is ground test data
• The image metadata is stored in a relational
  database our Data Record Management System
  (DRMS)
• The image data is stored in SUMS (Storage Unit
  Management System) which itself has database
  tables to manage its millions of files.
• SUMS owns the disk and tape resources.
• Users interact with DRMS via a programming
  lauguage, e.g. c, FORTRAN, IDL.
• The name of a dataset is actually a query in a
  simplified DRMS naming language that also allows
  general SQL clauses.
• Efficient use of the system relies on direct use
  of DRMS
• Data may be exported from DRMS as FITS or other
  (TBD) protocols for remote users.
• Several Remote DRMS (RDRMS?) sites will be
  established which will subscribe to series of
  their choice. They will maintain RSUMS
  containing their local series and cached JSOC
  series.
• The JSOC may act as an RDRMS to access products
  made at remote sites.

6
JSOC Pipeline Processing System Components
Pipeline processing plan
Pipeline Operator
SUMS Disks
Processing script, mapfile List of pipeline
modules with needed datasets for input, output
PUI Pipeline User Interface
SUMS Storage Unit Management System
DRMS Data Record Management System
SUMS Tape Farm
Processing History Log
Database Server
7
Simple example find and look at an image
Example of simple utility module called
show_keys First find images in the minute
starting 9 Sept at 2350 then look at one with ds9
P P show_keys "dshmi_ground.lev02007.09.09_
2350/1m" keyFSN,T_OBS FSN T_OBS 0566684
2007.09.09_235001_UTC 0566685
2007.09.09_235006_UTC 0566686
2007.09.09_235011_UTC 0566687
2007.09.09_235016_UTC 0566688
2007.09.09_235021_UTC 0566689
2007.09.09_235026_UTC 0566690
2007.09.09_235031_UTC 0566691
2007.09.09_235036_UTC 0566692
2007.09.09_235041_UTC 0566693
2007.09.09_235046_UTC 0566694
2007.09.09_235051_UTC 0566695
2007.09.09_235056_UTC P P ds9 show_keys
"dshmi_ground.lev0566686" segfile -p -q P
8
JSOC Export

• JSOC will support VSO access
• JSOC will also have a direct web access
• There will be remote DRMS/SUMS systems at key
  Co-I institutions
• ALL HMI and AIA data will be available for export
  at level-0 through standard products (level-1 for
  both and level-2 for HMI)
• It would be unwise to expect to export all of the
  data. It is simply not a reasonable thing to
  expect and would be a waste or resources.
• Our goal is to make all useful data easily
  accessible.
• This means we must develop browse and search
  tools to help generate efficient data export
  requests.

9
DRMS/SUMS Configuration

• DRMS and SUMS use the open-source PostgreSQL
  database engine.
• DRMS will run on a pair of dedicated servers
  likely 4-quad-core processors with up to 10TB
  fast disk.
• SUMS will consist of file servers with attached
  tape systems.
• SUMS will manage 200TB cache disk with 150TB/year
  of permanent archive for level-1 and higher level
  products.
• SUMS archive will use LTO-4 tapes in a robotic
  system with at least 10 drives and 2000 tapes
  near-line.
• The pipeline processing system will have about 50
  processor cores dedicated to level-0 to level-1
  processing and about 450 cores for higher level
  processing in the pipeline. HMI standard
  products will need about half of these.

10

• Extra Info

11
Pipeline client-server architecture
Generic file I/O
Data Segment I/O
JSOC Disks
JSOC Disks
JSOC Disks
JSOC Disks
DRMS socket protocol
Data Record Management Service (DRMS)
Data Record Management Service (DRMS)
Storage unit transfer
Storage Unit Management Service (SUMS)
Data Record Management Service (DRMS)
AllocUnit GetUnit PutUnit
Storage unit transfer
SQL queries
Tape Archive Service
PostgreSQL Database Server
SQL queries
SQL queries
Record Catalogs
Record Catalogs
Series Tables
Record Tables
Storage Unit Tables
12
Pipeline batch processing

• A pipeline batch is encapsulated in a single
  database transaction, DRMS session
• If no module fails all data records are commited
  and become visible to other clients of the JSOC
  catalog at the end of the session
• If failure occurs all data records are deleted
  and the database rolled back
• It is possible to commit data produced up to
  intermediate checkpoints during sessions

Pipeline batch atomic transaction
Module 2.1
Module N
Commit Data Deregister
Module 1
Register session

DRMS API
DRMS API
DRMS API
DRMS API
DRMS API
Module 2.2
DRMS API
Input data records
Output data records
DRMS Service Session Master
Record Series Database
SUMS
13
HMI module status and MDI heritage
Intermediate and high level data products
Primary observables
Internal rotation
Heliographic Doppler velocity maps
Spherical Harmonic Time series
Mode frequencies And splitting
Internal sound speed
Full-disk velocity, sound speed, Maps (0-30Mm)
Local wave frequency shifts
Ring diagrams
Doppler Velocity
Carrington synoptic v and cs maps (0-30Mm)
Time-distance Cross-covariance function
Tracked Tiles Of Dopplergrams
Wave travel times
High-resolution v and cs maps (0-30Mm)
Research codes in use by team
Egression and Ingression maps
Wave phase shift maps
Deep-focus v and cs maps (0-200Mm)
Far-side activity index
Stokes I,V
Line-of-sight Magnetograms
Line-of-Sight Magnetic Field Maps
Stokes I,Q,U,V
Full-disk 10-min Averaged maps
Vector Magnetograms Fast algorithm
Vector Magnetic Field Maps
Vector Magnetograms Inversion algorithm
Coronal magnetic Field Extrapolations
Tracked Tiles
Tracked full-disk 1-hour averaged Continuum maps
Coronal and Solar wind models
Continuum Brightness
Solar limb parameters
Brightness feature maps
Brightness Images
14
AIA Level-2
15
JSOC Data Volumes from Proposal
16
Sample of DRMS/SUMS Resource Assessment
17
JSOC Level-0 processing flow
18
JSOC-SDP Status

• Capture system complete, waiting testing with DDS
  at White Sands
• DRMS and SUMS virtually done
• Level-0 work progressing, to be done by
  instrument deliveries
• Level-1 work to start after delivery
• Getting serious about work on basic pipeline
  modules
• Review of status and computer hardware plans in
  early November

19
Stanford JSOC effort plan