Use Case 1: Sciamachy Data Center

1
Use Case 1 Sciamachy Data Center

• SciaGrid project results

Wim Som de Cerff John van de Vegte Richard van
Hees David Groep Jan Just Keijser Maurice
Bouwhuis Pieter van Beek
2
Content

• What is the NL-SCIA-DC?
• Why Grid?
• Implementation
• Results and outlook

3
What is Sciamachy?

• SCIAMACHY is an passive imaging
  spectrometerSCanning Imaging Absorption
  spectroMeter for Atmospheric CartograpHY
• Satellite instrument on the ESA ENVISAT satellite
• Objective is to perform global measurements of
  trace gases (e.g. ozone, NO2, CH4, aerosols) in
  the troposphere and in the stratosphere.
• The solar radiation transmitted, backscattered
  and reflected from the atmosphere is recorded at
  relatively high resolution (0.2 nm to 0.5 nm)
  over the range 240 nm to 1700 nm, and in selected
  regions between 2000 nm and 2400 nm.
• SCIAMACHY has three different viewing geometries
  nadir, limb, and sun/moon occultations which
  yield total column values as well as distribution
  profiles in the stratosphere and (in some cases)
  the troposphere for trace gases and aerosols.

4
Sciamachy product examples

• Ozone hole Southern Hemisphere (October 2008)

5
(No Transcript)
6
Why is NL-SCIA-DC needed?

• Complementing ESAs distribution facilities
• User need for fast and complete access to GOME
  and SCIAMACHY data
• Supporting the development of Dutch algorithms
• Distribution of Dutch data products
• Domain specific search/query capabilities

7
Goals for NL-SCIA-DC

• Provide to the users
• Access to Sciamachy, GOME, MERIS and AATSR data
• Selection methods, for easy selection of data
• Downloading of selected datasets and products
• Deployment of Dutch data products
• Test environment for new data processors
• (fast) dataset (re)processing capabilities

8
Overview of NL-SCIA-DC
Tape Archive
9
Data

• GOME level 1b, 2 from 1996 up to now
• 1.5 Terabyte of data
• metadata and data products databases
• All pixels can be queried and browsed
• Sciamachy level 0, 1 and 2 from 2002 up to now
• 40 Terabyte of data, and growing
• metadata and data products database
• Accessible through catalogue, including extracted
  metadata
• All pixels can be queried and browsed
• Archive and metadata database are automatically
  updated (satellite dish, ftp, DVD)

All data can be accessed online Via browser or
application
10
Data in the product databases

• PostgreSQL 8.3 with PostGIS extension used
• Database is now 112 Gbyte and growing

11
Users

• The NL-SCIA-DC has 120 registered users from 22
  countries, from 71 different organizations.
• Bulk data users. Data is delivered directly to
  them by sftp. Current bulk data users (standing
  order) are KNMI, SRON, BIRA (Belgium), University
  of Heidelberg (Germany) and ISAO (Italy).
• TEMIS (ESA)Tropospheric Emission Monitoring
  Internet Site (TEMIS) aims to compute and deliver
  global concentrations of tropospheric trace
  gases, and aerosol and UV products derived from
  observations of nadir-viewing satellite
  instruments such as GOME, SCIAMACHY and (A)ATSR.
  TEMIS is part of the Data User Programme (DUP) of
  the European Space Agency (ESA). The service of
  TEMIS centres around four themes Air pollution
  monitoring, UV radiation monitoring, Support to
  Protocol monitoring, Support to Aviation control.
• PROMOTE (GMES)To deliver the Atmosphere GMES
  Service Element a sustainable and reliable
  operational service to support informed decisions
  on the atmospheric policy issues of stratospheric
  ozone depletion, surface UV exposure, air quality
  and climate change

12
User interface
classic client server Java Applet Search,
process, download
13
Why Grid?

• Datasets are large and not easily downloaded to a
  workstation
• Users want to run their algorithm on a larger set
  of Sciamachy data
• Running an algorithm on a large set takes too
  long on a single workstation
• Algorithms are mostly embarrassingly parallel ?
  very much suited to run in a Grid environment!
• Also very interesting for reprocessing of data

Sciamachy chains for metadata extraction, CH4
level 2, level 2 daily average, level 3 daily
and plot processing
14
SciaGrid Project

• Together with NIKHEF and SARA
• NIVR GO financed project
• Aim Griddify the NL-SCIA-DC
• Share archives and databases at KNMI and SRON
• Make data accessible for resources at NIKHEF and
  SARA (Grid)
• Run NL-SCIA-DC jobs on Grid infrastructure,
  using the NL-SCIA-DC GUI
• In the project
• Experiments with Storage Resource Broker (SRB)
• Robot certificate
• Pilot job engine

15
Overview
16
Results SciaGrid

• SRB did not solve our problem Drawbacks
• Adding an existing archive is not easy
• Licensing of SRB
• Future?
• Solved the metadata part in an other way, Grid
  FTP selected for data access
• Certificates NL-SCIA-DC has (first issued) robot
  cert!
• Users can use their own login from NL-SCIA-DC to
  submit jobs
• Pilot Job framework used
• Gain better successful submission ratios
• Minimize Grid component installations at KNMI/SRON

17
SciaGrid setup
18
Status NL-SCIA-DC
Available
Debug
19
Summary and outlook

• Grid experiment was successful ? connection to
  the Grid established
• Data is accessible at Grid resources
• Jobs can be submitted using the NL-SCIA-DC GUI
• Release of User interface asap so users can
  actually use the new functionality
• NL-SCIA-DC operations in SciaVisie project
• Grid component expanded in Big Grid (?)