Data Management Challenges of the Large Synoptic Survey Telescope

1
Data Management Challenges of the Large Synoptic
Survey Telescope

• Christopher Stubbs
• Department of Physics
• Department of Astronomy
• Harvard University

2
LSST

• Basic idea
• Survey entire sky every few nights
• Subtract images to find variability
• Add images to go deeper

3
The Large Synoptic Survey Telescope Massively
Parallel Astronomy

• Survey the entire sky every 2-3 nights, to
  simultaneously detect
• Potentially hazardous near earth asteroids
• Tracers of the formation of the solar system
• Fireworks in the heavens GRBs, quasars
• Periodic and transient phenomena
• Dark Matter via Weak gravitational lensing
• Thousands of supernovae per year, in multiple
  passbands
• ......

4
 
5
(No Transcript)
6
The Flavor of Astronomical Data
Sources at r infinity
Wavefront distortion due to atmospheric structure
seeing
2-d Gaussian distribution of flux from point
source, Point Spread Function
Optical system maps angles to focal plane position
2-d array of 16 bit integers
7
Point Sources Appear as 2-d Gaussians
8
Galaxies Saturated star Charge blooming down
CCD columns Diffraction spikes from optics
support
9
Typical Image Processing Stages
Pre-processing (K-op/pixel) Coordinate
registration Crosstalk removal
Cleanup and source detection (5
Kops/pixel) Subtract bias structure Divide by QE
frame Artifact removal Optimal spatial
filtering Threshold detection Object
Characterization
Catalog insertion Classification tags Index
update
10
Image Subtraction

• Geometrical warp compensation
• Determine and apply additive and multiplicative
  scalings
• Determine and apply convolution kernel to match
  PSFs
• Subtract frames

11
Frame Subtraction Works (well, most of the time,
anyway...)
(High-z Supernova Team)
12
Data management challenges
Analysis Pipelines Change detection Classificati
on of variability Optimal Shear
detection Calibrations Efficiencies
Infrastructure Flexible framework for algorithm
implementation Facilitate reprocessing and assoc
book-keeping Ease of data distribution
Access Useful and effective data
access Facilitate fusion with other data
sets Scope NVO interface
Database Aggregation of detections into
objects Schema Optimal indexing Data
pedigree
13
Aggregation of Detections into Objects

• LSST will provide detections of all objects
  across the entire sky
• Under variable sky and PSF, these need to be
  aggregated (clustered) into well-defined
  astronomical objects, with robust extracted
  parameters.

14
Classifying Variability has been Hard
Done by hand at present AGN/Supernova
confusion Microlensing/Supernova
confusion Eigenvector approach? What
minimal set of parameters would allow offline
queries to carry out classification?
15
LSST Database Challenges

• Fundamental Representation of information
• Images produce detections
• Image quality and other factors determine
  resolution S/N ratio
• Extracted parameters as f(2 angular coordinates
  time)
• Clustering of detections into astronomical
  objects
• Blended flux from multiple objects
• Done on the fly, or at ingestion?
• Statistical classifications?
• 20 asteroid, 70 supernova, 10 stellar....
• Priors?
• Optimal indexing
• How do we structure information to optimally
  extract shear down a line of sight, as a function
  of redshift?
• Scaling
• Terabytes per night forthcoming, petabyte data
  sets
• Scope, Schedule and Cost.....

16
Our Approach

• Take full advantage of precursor projects that
  are now under way
• Build end-to-end prototypes, even if crude
• Management challenge is to capture best of
  top-down and bottom-up approaches....