Data reduction for the HTN

1
Data reduction for the HTN

• Iain Steele
• Liverpool JMU

2
What do we need?

• Depends on science
• Many projects just get the data and worry about
  it later (offline) in the way we have
  traditionally done.
• Other time critical projects require on-line
  reduction to maximize science return

3
A typical telescope is supporting many science
programmes

• http//150.204.240.8/data/archive/webfiles/quicklo
  ok/lt/

4
GRBs

• Optimized followup needs complex data reduction
• WCS fitting
• Catalogue comparisons
• Variability analysis
• Differential imaging

5
Exoplanet microlensing
6
Even more complex pipeline

• High precision, difference imaging
• PSF matching
• Locating the object (very crowded fields)
• Computationally intensive
• http//star-www.st-and.ac.uk/robonet/2005/blg-390
  .stamps.html

7
Problems

• Where should the pipelines run?
• How can we make them telescope independent?
• Differences in instrument behaviour
• Differences in computational environment
• Operating systems, memory, disk space
• Software availability

8
At the telescope

• Easy to do telescope specific stuff
• Hard to do science specific stuff
• Risks telescope operation
• http//telescope.livjm.ac.uk/Fault/Bugzilla/show_b
  ug.cgi?id1015
• Another cost on telescope operators

9
At the UserAgent

• Easy to do science specific stuff
• Hard to to telescope specific stuff
• Performance overhead in data transfers can be
  prohibative

10
Somewhere else?

• Hard to do science specific stuff
• Hard to do telescope specific stuff
• Maybe the above 2 points are a good thing!
• Access to high peformance resources
• Shifts the load from the telescope operators
• Remember the grid - ship the job not the data

11
Remember the grid?

• Ship the job to the data, not the data to the job

12
Conclusion

• If we want to do complex science programmes on
  the HTN we need to start to pay as much attention
  to data reduction as data gathering
• Do we need a common execution environment for
  data reduction?
• What higher level data products can we
  standardize without losing vital information?
• Do we need a common scripting envirionment?
• What can we exploit to help us (e.g. ORAC-DR,
  AstroGRID Workflow builder)?