A Virtual Survey SYSTEM

1
A Virtual Survey SYSTEM

• Astro-Wise
• NOVA/Kapteyn OA Capodimonte
• ESO Terapix US Munich/MPE
• National WFI datacenters NL-I-D-Fr/ESO
• EU FP5 RTD programme parallel to AVO
• 5 year programme -gt dec 2006

Edwin A. Valentijn
2
Basic objectivesWide Field imaging EU

• Facilitate handling, calibration, quality
  control, pipelining, user tuned research,
  archiving, disseminating results
• 100s Tbyte of image data and
• 10s Tbyte of catalogue data
• With production spread over EU
• What-ever gt object model / scalability
• Where-ever -gt federations, GRIDS
• Who-ever -gt Python as glue (GUIs)

(O)MegaCAM
3
status

• VO conference 2002 - design
• Build information system - working
• Implemented, qualified
• WFI_at_2.2m, WFC_at_INT, MDM, OmegaCAM_at_ILT
• gtgtQualify with OmegaCAM_at_VST-2006
• gtgttune to run Public Surveys
• gtgtquality control
• gtgtOptimize federation/replication

4
new paradigmtarget processing
to -hunting /full linking -data pulling -target
processing -raw data -archive -all in archive
-request driven

• from
• -waterfall/ multi-tier
• -data pushing
• -raw data processing
• -raw data delete
• -result -gtarchive
• -releases

• raw pixel data ? pipelines/cal files ? catalogues
• all integrated in one information system
• distributed services ?Virtual Survey Telescope
• processing GRID
• Storage GRID
• Methods/services GRID

5
Astro-Wise VO Properties Benefits integrated
dynamic db

• on-the fly re-processing
• 5LS 5 Lines Script
• All bits are traced
• Administration for parallel processing
• compute GRID SETI_at_home
• Global solutions astrometry/photometry
• Buildin workflow
• Fully user tunable own provided script
• Context projects/surveys, instruments, mydb
• Publish directly in EURO-VO

6
components

• Procedures Cal plan at telescope
• Data model -gt object model -gtdb
• Central db server/clients
• All I/O except images
• Meta data
• Source lists catalogues associate lists
• Links references joints
• Fileserver distributed- via db
• Python clients
• CVS distributed code base - opipe

7
Astro-Wise Pipelines
8
Target processing the make metaphor

• awegt targethotHotPixelMap.get(date'2003-02-14',
  chip'A5382')
• The processing chain is
• ReadNoise lt-- Bias lt-- HotPixels
• gt class HotPixelMap(ProcesTarget)
• gt gt def self.make()
• gt class ProcessTarget()
• gt gt def get(date, chip) if not
  exist/up-to-date then make()
• gt gt def exist() does the target
  exist?
• gt gt def uptodate() is each dependency up
  to date?
• Fully
  recursive

9
Intra-operability peer to peer

• code base docs CVS
• Db Advanced Replication evolving to streaming

WRITE
WRITE

• READ-ONLY

• READ-ONLY

• REPLICATION

• REPLICATION

• REPLICATION

• REPLICATION

• READ-ONLY

• READ-ONLY

10
Contents of federation

• Raw data
• Observed images
• Ancillary information
• Calibration results
• Calibration files time stamped
• Reduced images
• Single observation
• Co added images
• Software
• Methods (pipelines) for processing calibration
• Configuration files
• Source lists catalogues
• Extracted source information
• Associated among different data objects

11
Example 5LS

• Find ScienceFrames for a ccd named ccd53 and
  filter
• Awegt q (ReducedScienceFrame.chip.name 'ccd)
  and (ReducedScienceFrame.filter 841)
• From the query result, get the rms of the sky
  in image
• Awegt x k.imstat.stdev for k in q
• get the rms of the used Masterflat
• Awegt y k.flat.imstat.stdev for k in q
• Make a plot
• Awegt pylab.scatter(x,y)

12
Astro-Wise PORTAL
13
Web services- object viewer
14
QC - calibration scientist monitoring
15
QC - calibration scientist monitoring
16
Web services- object maker
17
VST - Virtual Survey Telescope
18
Lofar
IBM- Blue Gene/L
19
Thanks!

• Welcome at next Astro-Wise tutorial!
• October 2005
• Netherlands
• www.astro-wise.org