Overview and Status http:www'astrogrid'ac'uk Nicholas Walton IoA, Cambridge

1
Overview and Status(http//www.astrogrid.ac.uk)
Nicholas WaltonIoA, Cambridge
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p1 Printed 30/10/01
2
The Sociology of Astronomy

• Continuing Collectivization
• Facility class (common user) instruments
• Central development of supporting s/w (e.g. Iraf)
• Calibrated archives and access tools (e.g. IPAC)
• Information services (e.g. ADS, NED, astro-ph)
• Consortium projects (e.g. MACHO, SLOAN, VISTA)
• Evolving Developments
• Inter-operable archives, joint queries (e.g.
  MAST)
• Data mining (exploration and analysis tools)
• Information discovery tools

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p2 Printed 30/10/01
3
Major Background Drivers

• The Growth of Data
• Significant increase in number and size of
  telescopes
• In the optical ESO's 4x8-m VLT, Gemini's 2x8-m
• In the x-ray XMM-Newton, Chandra
• In the mm ALMA
• Significant increase in size and multiplex
  capabilities of associated instrumentation and
  detectors, e.g.
• In the optical VISTA will have a Gpixel array
• In the radio e-Merlin with data rates of 320
  Gbps will generate

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p3 Printed 30/10/01
4
Major Background Drivers

• The Growth of Data Archives
• Many observatories hold multi-TB archives
• New initiatives set-up to support new observing
  capabilities (e.g. TeraPix)
• In the optical, need to ingress all-sky survey's
• Whole sky at 0.1 arcsec/pix is 100TB
• Increasing Importance of Archival Data
• Time on expensive facilities (e.g. HST) only
  awarded if the archive has been searched before
  hand
• This trend is continuing, driving observatory and
  user demand for more accessible archival data

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p4 Printed 30/10/01
5
Empowering Science Driven Observational Astronomy

• Break down the 'wavelength' barriers
• e.g. Greater focus on science driven proposals
  encouraging the use of data sets from across a
  wide range of wavelengths
• Increase Access
• e.g. East European countries may not be able to
  fund a major new telescope but can contribute to,
  and access 'Virtual Observatories'

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p5 Printed 30/10/01
6
Astronomical Drivers Enabling New Science

• Linking the near and distant Universe
• Comparison of rest frame samples to study
  evolution of galaxies
• Combination of UV, optical and IR datasets
• Creating the 'Digital Sky'
• Temporal data measures motions in the Galactic
  centre, probes the creation of our Galaxy
• The search for extra-solar planets
• The planet-transit technique using federated
  survey data for millions of stars

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p6 Printed 30/10/01
7
Astronomical Drivers New Era of Surveys

• SuperCOSMOS (UK now till 2002)
• Based on Schmidt plates - Science database 2TB
• Sloan Digital Sky Survey (US now till 2005)
• Dedicated CCD survey telescope Science database
  10TB
• UKIDSS (UK from 2003)
• IR camera on 4-m UKIRT Science database 30TB
• VISTA (UK from 2005)
• IR camera on 4-m VISTA Science database 300TB
• LSST (US from 2007-8?)
• Dedicated 8-m telescope All sky/few
  nights5000TB/yr

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p7 Printed 30/10/01
8
Astronomical Drivers Pre-Discovery Mining

• Investigating the progenitors of sources that
  show variability
• Dark matter revealed by microlensing events
• Planets revealed by stellar variability
• Formation of neutron stars revealed by GRB's
• Death of massive stars revealed by Type II SN

The progenitor of SN1999gi is lt9 M? found from
mining pre-discovery HST images. (Smartt et al,
2001, ApJ, 556, L29)
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p8 Printed 30/10/01
9
Astronomical Drivers Rare Objects

• Huge data sets open up the possibility to find
  rare objects
• Those that are hard to find
• Brown dwarfs, have unique red colours
• Those that are intrinsically rare
• High-z quasars, stand out due to suppression of
  their blue colour by the Ly-? forest

Hi-z QSO's found from SDSS multi-colour data the
shaded area is domain of QSO's, solid line is
track for increasing z (Fan et al, 2001, ApJ,
121, 31)
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p9 Printed 30/10/01
10
Astronomical Drivers New Objects

• Large datasets open the possibility to discover
  new objects
• Those that have been missed before because they
  are extremely rare or short-lived
• Those that have previously been misclassified,
  revealed as outliers in new parameter space
  correlations

DPOSS group, during searches for high-z quasars,
have discovered peculiar objects, this one
perhaps a BAL QSO (Djorgovski et al, 2001, PASP,
225, 52)
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p10 Printed 30/10/01
11
AstroGrid Meeting the Challenge

• Recognising the specific need to support UK
  survey science and associated key datasets was
  the backdrop to the genesis of the AstroGrid
  program
• Initiative driven by key dataset users and
  archive centres
• Cambridge, Edinburgh, Leicester, RAL, MSSL,
  Jodrell Bank
• Major input from specialist s/w CS groups
• QUB and RAL (Starlink)
• Programme funded through the PPARC e-science
  line, beginning 9/01 for 3 years with 5M

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p11 Printed 30/10/01
12
The AstroGrid Project Structure
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p12 Printed 30/10/01
13
AstroGrid Project Finance Staffing

• Finance
• Nominal 3 year funding envelope 5M
• Phase-A commitment 0.8M
• AstroGrid funding includes element to produce the
  VISTA e-pipeline
• Additional funding provided through the AVO
• Staffing
• 10 FTE/yr (over 15 people) from Jan 2002
• Some new appointments, some displacement
• Additional 3 FTE/yr employed via AVO

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p13 Printed 30/10/01
14
The AstroGrid Project Timelines
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p14 Printed 30/10/01
15
AstroGrid The Phase-A Study

• Intensive one-year RD phase
• Science requirements analysis
• Community consultation
• Development of use cases
• h/w s/w benchmark tests
• e.g. Beowulf vs SMP, Object vs Relational
• Design benchmark problems and run tests
• Pilot federations, e.g.
• SuperCOSMOS/ SDSS/ INT-WFS
• Merlin/ VLA

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p15 Printed 30/10/01
16
Phase-A DataBase Technology

• Support data structures and indexing required by
  the problem domain
• Application areas include
• Access to existing and interface to new archives
  (e.g. VISTA)
• Support of data mining activities
• Statistical queries access to remote databases
• Support of multi-dimensional indexing
• Support for heterogeous range of data structure
• Tasks include
• Development of functional requirements based on
  use-cases
• Survey of existing commercial and open source
  products
• Evaluation of sky indexing methods
• Integration of DB's with h/w and grid technologies

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p16 Printed 30/10/01
17
Phase-A Grid Technology

• Evaluation of 'Grid' Technologies
• Globus 'middleware'
• XML FITS options for meta-data handling
• Storage Resource Broker
• Jini
• CORBA and object orientated approaches
• Development of trial Data-Grids suitable for
  astronomy
• Initially link two distributed data centres
  (Edinburgh, IoA)

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p17 Printed 30/10/01
18
Phase-A Grid Technology
Three Key Problems for Astronomical Data Grids

• Access to grid and authentication of users
• Selection and location of relevant data
• Tell me what data you have relating to x, but
  dont send me the data yet.
• Transport of data and metadata to the processing
  site
• Processing includes visualisation.

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p18 Printed 30/10/01
19
Phase-A Grid Technology
Learn by experiment the current test
architecture
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p19 Printed 30/10/01
20
Phase-A Grid Technology
An experimental portal
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p20 Printed 30/10/01
21
Phase-A Grid Technology
Enable future development and deployment of
AstroGrid and External (e.g NVO) tools
Automated detection of outliers in SDSS two
colour data (Connolly et al, 2001, AJ in press)
A NVO prototype of an automated discovery tool
for arcs developed by A Szalay
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p21 Printed 30/10/01
22
Phase-A Compute/Storage Options

• Development of benchmark tests
• Assessment of options
• Availability
• Performance
• Scalability
• Cost
• Recommendations of Options for Phase-B
• Enable integrated access to terascale computing
  and data facilities

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p22 Printed 30/10/01
23
Phase-A The Pilot Programme

• Federations of key datasets to assess performance
• Large scale object catalogues SDSS, SuperCOSMOS,
  INT-WFS, INT-IR data (Opt/IR)
• Possible provision of limited access to pixel
  data
• Event lists XMM-Newton, Chandra (X-ray)
• Generate event lists/spectra from image
• Fourier Data (Radio)
• radio/optical, postage stamp generation,
  deconvolution
• Image and Movie Libraries (Solar)
• Image transport, heliocentric coordinates,
  on-demand proc
• Time Series Data (STP)
• Tabular data, metadata, search by time period

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p23 Printed 30/10/01
24
Phase-A The Pilot Programme

• Problems in matching multi-? survey data
  Differences in angular resolution, s/n ratios,
  backgrounds, etc (Djorgovski et al, 2001,
  astro-ph/0108346)

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p24 Printed 30/10/01
25
The VISTA e-pipeline

• VISTA major new UK 4-m IR-survey
  telescope www.vista.ac.uk
• Located on Paranal and integrated into ESO VLT
  ops
• IR camera consists of 16 off 2kx2k IR detectors
• e-pipeline use the 'grid' for distributed image
  processing
• On the fly image processing, stacking/tiling,
  catalogue generation, object identification
• Use for QC feedback for ESO operations

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p25 Printed 30/10/01
26
Towards Phase-B The Science Case

• Rapid development of science requirements
• Emphasis on user input via consultation
• Development of 'use cases'
• Example Formation of large scale structure
• Construct unbiased cluster of galaxies sample
  over z to test various cosmological models of
  galaxy formation
• Need to operate on large data sets, constructing
  simultaed surveys to test for selection effects
• Generate predictions of observed sample
  properties and compare with observed samples
• Development of 'White Paper' developing
  AstroGrid's role for Phase-B and in the context
  of larger VO picture

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p26 Printed 30/10/01
27
AstroGrid and the AVO

• Astrophysical Virtual Observatory (AVO)
  
  http//www.eso.org/projects/avo
• European Commission supported programme, 2M
• 3 year Phase-A study of a European astronomicalVO
  
• Major partners ESO, ST-ECF, AstroGrid, CDS,
  Terapix, Jodrell
• Has links with similar US and Australian
  initiatives
• 3 initial areas of study
• Science requirements (ST-ECF)
• Interoperability (CDS)
• Key relevant technologies (AstroGrid)
• AstroGrid taking lead in Key Technologies'

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p27 Printed 30/10/01
28
AstroGrid's AVO Workpackages

• AstroGrid AVO work aligned with it's Phase-A
• Grid Technologies
• Compute/ Storage Technologies
• Data Base Technologies
• Other AstroGrid AVO inputs
• The science case, towards a 'Virtual Observatory'
• Interoperability, integrate with Pilot
  Federations
• AstroGrid receives funding for 3 EU PDRA's
• Includes one specfically to Jodrell Bank

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p28 Printed 30/10/01
29
VO initiatives in the USA

• NSF have recently funded a 5 year CalTech/JHU led
  VO initiative 10M http//www.us-v
  o.org
• GriPhyn The Grid Physics Network
• Aiming to deploy Petascale Virtual Data Grids
• Supporting a number of major experiments
• CMS and ATLAS at the LHC (CERN)
• LIGO http//www.griphyn.org
• SDSS
• SDSS SkyServer public access
  
  http//skyserver.fnal.gov

http//www.griphyn.org
N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p29 Printed 30/10/01
30
Concluding Remarks

• AstroGrid is a major new UK funded e-science
  initiative
• In partnership with EU centres it will play a
  lead role in the realisation of a European
  Virtual Observatory
• AstroGrid is poised to significantly enhance the
  opportunities of the UK astronomical research
  community

N A Walton AstroGrid Overview PPGrid
Collaboration Meeting, Nov 5-6, 2001
p30 Printed 30/10/01