Ilya Slutsker Data Base Development, Processing and Data Archive

1
Ilya SlutskerData Base Development, Processing
and Data Archive

• Data Collection
• Archives
• Processing
• Data Access Delivery and Distribution

2
Data Collection

• Historically, first data from Automatic Sun
  Photometers were collected manually. Starting
  in 1993 DCP (Data Collection Platforms) have
  become preferable way of getting data.

• Today, about 92 of data is collected using
  automatic DCP transmissions, and only about 8 is
  send using manual dump of the instrument memory
  to PC and submission of this dump to Aeronet
  server.

• Main reasons for the manual downloads are 1)
  Unavailability of the direct satellite uplink
  Far South and North, Landscape profiles 2) Local
  regulations. 3) Equipment malfunctioning. Most
  of those problem eventually can be remedied by
  using cellular or digital communication
  technology.

3
Until May 1993 We did not use DCP Satellites
Uplinks
In 1993 we started transmitting using GOES East
from US, Canada and Brazil
Soon we extended our transmission coverage area
to GOES West Alaska, Hawaii
Meteosat gave us access to European, African and
Near East sites
Finally by the end of the last century We covered
Far East and Pacific using GMS
4
DCP Data Collection
DCPs transmit data and satellites retransmit them
to Data collection Stations. These Stations
receive data, demodulate them, sort them and then
send to users.
Goes East and West data are processed by NOAA
DAPS at Wallops Island, Virginia.
Meteosat are processed by EUMETSAT in Darmstadt,
Germany
GMS data are processed by Vitel built
receiver, Located at the NOAA PTWS, Ewa Beach,
Hawaii
5
Even manual data download changed very much
during recent years. Now Site Managers can submit
data on-line 24 hours a day, and the data Will
be automatically archived and processed within 1
hour
6
After being submitted to Aeronet server Data are
processed and placed to several archives
EUMETSAT
DAPS
PTWS
K7_upload.html
Hourly Auto Upload
Manual upload
DCP Messages Archive
K7 Archive
DCP decoding
K7 decoding
Daily Reports Archive
Processing Software
Processed Data Archive
Raw Data Archive
7
We maintain several archives that allow us
properly process the data.

• Filters Archive. It has spectral response
  function for filters installed in our
  instruments. Now the information about more that
  3000 filters is stored there.
• Sites Archive. All information of Aeronet sites
  where instruments have been operated. Currently
  496 Sites.
• Instrument Configurations Archive. Many
  instruments have their configurations changed
  over years. The archive keeps that information
  including types of instruments and filters used.
• Calibration Archive. Sun , Sky and Flux
  calibrations for all instruments.

8
Filters Archive
Filter Functions are sorted first by nominal
channel (i.e. 1020, 440 ) Then by the year of
shipment (1997, 1998 ) Then by the name of batch
(1020_1, Lot2401 ) And finally by the number
of the filter.
We use the program administrator to manage this
archive.
9
Sites Archive
Sites are sorted first by types Permanent,
Seasonal, Temporary, by Region and Country/State,
then by PIs name. We can search sites using
all the above parameters and also by the
coordinates and name of the site. The archive is
managed by administrator
10
Instruments Configurations Archive
There have been more than 20 different instrument
types over years. Most common are Standard,
Polarized and Advanced. Each type characterized
by its own set of filters. The Archive is
managed by administrator.
11
Calibration Archive
This archive is managed by demonstrat program.



Example Calibration history.
12
Data Archive

• This is where all the measurements data and
  Aerosol Property products are kept.
• There are 2 archives Raw Data Archive and
  Processed Data Archive.

Processing Software
Processed Data Archive
Raw Data Archive
Direct Solar Measurements
Aerosol Optical Depth
Sky Measurements (Almucantars, Principal Planes,
BRDF)
Quality Assured AOD
Fluxes from Flux sensors
Water Vapor
Sky Radiance retrieval products ( Size
distribution, Phase Function, Single Scattering
Albedo, Refractive Index )
Temperature, Internal and External battery
voltage, Dark current, resets, hardware statuses
and errors
13
Processing Sequence and Software

• Once the data are collected and archived in the
  Aeronet server, several processing tasks are
  performed.
• Each of these tasks is managed by its own
  software.
• Individual raw solar measurements are used and
  level 1 AOD and Water Vapor are computed. Runs
  every hour.
• Daily blocks of raw solar measurements are used
  and level 1.5 cloud screened AOD and Water Vapor
  are computed. Runs every day.
• Individual Almucantars and Principal Planes and
  Daily blocks of Level 1.5 AOD are used and Sky
  retrieval products are computed. Runs every day.
• All these tasks run automatically
• After they are performed, the data become
  available to our group. Normally all the data
  from the previous day are available by the noon
  of the current day.
• The data can be reprocessed, due to 1)
  recalibration, 2) coordinates adjustments 3)
  quality assurance. Reprocessing programs are
  started by the authorized user and reprocess long
  records of data , sometimes months and years per
  run.

14
Data Access

• All the data are accessible through several
  interface
• On the System Level we have access to the binary
  files which can be backed up, cloned or, if
  necessary deleted. There are encoding and
  decoding programs that can retrieve the data into
  ascii text, that can be looked at, modified, and
  saved.
• On the User Level users have access to
  demonstrat , which is data browsing tool. They
  can look at the data, download the data, and
  perform several research task. Authorized users
  can calibrate devices, reprocess data and raise
  the quality level of data.
• On the Public Level everybody have access to the
  Web Site where they can look at the data and
  download them.
• Our goal is to bring as much data as we can to
  the Public Level limiting Public Access by only
  actually changing the data. Only Authorized users
  are going to be able to modify the data. As for
  the raw data, they are locked and not modified.
• Next slide will show some example of the
  demonstrat use.

15
Demonstrat Main Window
Langley Calibration Window
We can click on almost anything, And also drag
the mouse across space and time. Right click on
Site or Instrument number will limit selection
to That site or instrument.
AOT Time Dependence Window
Almucantar Window
16
Conclusion

• As the network of instruments and sites grows and
  now scattered across the world with no continents
  to spare, the reliability and performance of data
  storage and processing software have become of
  the essence.
• Over the years we developed the system which
  collects, archives, processes and distributes
  Aeronet data from hundreds of sites and
  instruments and does it fully automatically in
  real time.
• Although all the routine processing and storage
  is done automatically, we keep complete control
  over the system performance and can monitor and
  troubleshoot it at any moment.
• The system development continues, as new data
  type and products become available.