The Astrowise pipeline

1
The Astro-wise pipeline

• Python
• Data processing - the low-level interfaces
• Data administration - persistent objects

2
Python

• Object Oriented scripting language
• clean, easy-to-understand syntax
• extensive library
• powerful built-in data types (str, dict, list,
  file)
• byte-code interpreted, dynamically typed
• Rapid development and easy maintenance

3
Python, a 1-slide course

• wordcount
• for line in file('my_thesis.txt')
• for word in line.split()
• if word not in wordcount
• wordcountword 1
• else
• wordcountword 1
• print wordcount

4
Image processing

• eclipse libraray in C (ESO, N. Devillard)
• wrapped with SWIG
• fits images and headers as built-in data types
• gtgtgt from eclipse import image, header
  gtgtgt img image('science.fits')
  gtgtgt hdr header('science.fits')
  gtgtgt flat image('flat.fits')
  gtgtgt img img / flat
  gtgtgt hdr'FLATFLD' 'flat.fits'
  gtgtgt img.save('flatfielded.fits',
  hdr)
• Co-addition with SWARP (IAP, E. Bertin)

5
Catalog Processing

• Sextractor (IAP, E. Bertin)
• LDAC (OmegaCAM, E. Deul)
• Astrometry
• Photometry
• Common interfaces, but no Python types
• Persistent Sourcelists (see KGB)

6
Astro-wise Pipelines

• Pipelines are about data-administration, not
  about data-processing
• Persistent objects
• Object's state persists across program boundaries
• Python classes representing SQL tables
• Should also work without back-end
• Distinguish meta-data and bulk (FITS) data
• meta data in persistent objects
• bulk (FITS) data through file-server

7
The make metaphor

• 'Making' objects
• think unix Makefile
• targets and dependencies (recursive)
• 
  gtgtgt bias BiasFrame()
  gtgtgt bias.raw_frames RawBiasFrame('bias1.
  fits'),
  RawBiasFrame('bias2.fits'),
  ...
  RawBiasFrane('biasN.fits') gtgtgt
  bias.make()

8
An example with Queries

• Queries are Python expressions (see Danny)
• Dependencies can be filled through queries
• 
  gtgtgt bias BiasFrame()
  gtgtgt query ((RawBiasFrames.chip.name
  'ccd50') (RawBiasFrames.DATE_OBS
  gt a_date-1) (RawBiasFrames.DATE_
  OBS lt a_date1)) gtgtgt bias.raw_frames
  list(query) gtgtgt bias.make()
  gtgtgt bias.store()
  the bulk FITS data gtgtgt
  bias.commit() the meta data

9
Image Pipeline