February 13th, 2004

1
HEASARC Status and Plans

• Nick White

2
The HEASARC Charter
Established December 1990

• Maintain and disseminate data from previous and
  concurrent high-energy astrophysics missions
• Provide software and data analysis support for
  these data sets
• Maintain and provide the necessary scientific and
  technical expertise for the processing and
  interpretation of the data holding
• Develop and maintain multi-mission analysis and
  support tools
• Provide catalogs of observations and ancillary
  information for the data holdings
• Coordinate data, software and media standards
  with other astrophysics sites
• Support education and outreach activities in high
  energy astrophysics (not part of original
  charter)

3
The HEASARC and High Energy Astronomy Missions
Swift
2
GLAST
HETE2
Integral
Chandra data_at_CXC
HEASARC provides the multi-mission infrastructure
that is used by parallel GOFs and science support
centers in the LHEA and elsewhere
4
HEASARC Science Personnel

• Nicholas White HEASARC Head
• Steven Murray HEASARC Deputy Head (at CXC)
• Lorella Angelini Data Restoration, Image and
  timing analysis, BeppoSAX, Swift (inc. AKBAR),
  Astro-E2
• Keith Arnaud Data selection software, spectral
  analysis
• Michael Corcoran Calibration, GLAST, HETE-2,
  ROSAT
• Stephen Drake Catalogs, website archive
  quality assurance, EUVE, RXTE, CGRO
• James Lochner Education and public outreach
• Thomas McGlynn Archives, Virtual Observatory,
  SkyView
• Bill Pence FITS, Software, HEASOFT, Hera

5
Recent HEASARC Highlights

• Hardware
• Acquisition of 8.4 TB (12 TB raw) TB network
  appliance (March 2003)
• Upgrade to Gigabit network (Oct 2003)
• Science Archive
• XMM downloads now 2nd in volume to RXTE
• HETE-2 archive reaches 150 GB and 51 bursts
• Specialized Swift Browse interface
• Software
• HEAsoft 5.3 release (November 2003) Improved fv
  integration with DS9
• Browse 7.0 release (January 2004) new upload
  download features
• XSPEC 12 testing by Integral SPI team (ongoing)
• Catalogs
• GRBCAT master catalog of gamma-ray bursts (Feb
  2004)
• Education and Outreach
• SkyView SciAm Top 50 award (June 2003 2nd year
  in a row)
• Teachers Hera beta implementation (July 2003)
• Services
• First VO Tool Data Inventory Service (July 2003)

6
The Physical Archive
Active Missions RXTE (1995- ) Chandra (1999- )
data at CXC HETE-2 (2000- ) Integral
(2001-) XMM-Newton (1999- )
Past Missions Ariel 5 EXOSAT ASCA
Ginga BBXRT BeppoSAX CGRO HEAO
1 Copernicus HEAO 3 COS B OSO 8 DXS ROSAT
Einstein SAS 2 EUVE SAS 3 Vela 5B
Upcoming Missions Swift (2004 Launch) Astro-E2
(2005 Launch) GLAST (2007 Launch)
Data from 24 missions currently in the
archive 355 astronomical catalogs mission
tables The archive volume was 3250 Gigabytes
as of the end of 2003

7
Usage Data Statistics
Gigabytes transferred per
year 1998 1999 2000 2001 2002 2003 Datasets
by ftp http 844 1406 1880 2270 2917 3943 Scienc
e Web Pages Images 198 234 371 399 580
872 Main EPO Web Pages Images 203 358
678 1010 3108 7581
8
Data Transfers by Mission over Time
9
Current Web Service Statistics

• Archive volume 3.3 TB (mostly compressed)
• Current ingest rate 400 GB/yr
• End of 2004 anticipated rate 1.4 TB/year, with
  increased or new flows from Swift, Integral, and
  XMM
• Missions in archive
• Ariel-5, ASCA, BBXRT, BeppoSAX, Chandra, Compton,
  Copernicus, COS-B, DXS, Einstein, EUVE, EXOSAT,
  Ginga, HEAO-1, HEAO-3, HETE-2, OSO-8, ROSAT,
  RXTE, SAS2, SAS3, Vela 5B, XMM-Newton
• Downloads (annual) 3.9 TB of data (by FTP and
  HTTP), 0.9 TB of science web pages (by HTTP) and
  7.6 TB of EPO web pages
• Browse 355 local tables ( 3000 VizieR)
  2,150,000 queries
• SkyView gt60 surveys gt1,000,000 images generated
  
• 26,000 bibliography entries linking ADS to
  HEASARC datasets (ROSAT, ASCA, XMM-Newton, and
  some RXTE datasets)

10
HEASARC Hardware Configuration
Web servers
HEASARC1 2 GHz Dual Processor
GSFC Portal
HEASARC (load balancer)
HEASARC2 2 GHz Dual Processor
More slots available
100 Mb
Database (Sybase) 2 GHZ dual Processors
Hera (x5) 1 GHz
SkyView (x2) 1 GHz
Gigabit
Online archive backup
6 TB Assorted Storage
Safehouse
400 GB User Cache
150 GB Surveys (x2)
240 GB External Survey Cache
Linux, Solaris, Mac, Windows,
11
Web site maintenance

• Main HEASARC web site has 15,000 static pages and
  hundreds of dynamic pages (excluding the FTP area
  and data archive) comprising 7 GB in total
• Ensuring valid and up-to-date links pages
  requires constant attention
• In the past, invalid links left unattended grew
  to 500.
• Concentrated effort has now eliminated all but a
  few broken or out of date links (Newly broken
  links are fixed weekly)
• HEASARC is now 99 Section 508 compliant
• Assuring appropriate access to HEASARC Web pages
  involved changes to most Web pages
• Compliance changes were often beneficial for all
  users.
• Maintenance cost of 508 compliance is lt0.05 FTE
  effort

12
Browse Catalog and Archive Interface

• Capable of querying all local and VizieR catalogs
• Local and remote archives (e.g., CXC)
• HTML, text, VOTable, FITS and Excel ouputs
• New capabilities
• Expressions allowed in search criteria (
  fluxX/fluxY gt 2)
• Better integration with SkyView
• Upload files of targets (in Browse 7.0)
• Download queries for later re-use (7.0)
• Dataset ID info added when available (7.0)
• Links between tables

13
Current Active Missions

• Chandra
• Browse service fully linked to CXC archive
• RXTE
• Dominates current archive downloads in terms of
  GB
• Simple XTEIndex table developed and deployed
  (more simplification needs to be done)
• HETE2
• Some HEASARC-written FITS products for GRBs
• Difficult to use most data delivered to archive
• Similarities to data recovery efforts of early
  HEASARC
• XMM-Newton
• In 2003, second biggest HEASARC mission in
  archive volume downloads
• Archive contains both US non-US Guest Observer
  data
• Integral
• Use modified version of HEASARC archive software
  at ISDC
• HEASARC Browse can link to ISDC Browse tables
• Archive at HEASARC will open in July 2004

14
Preparations for Swift

• New HEASOFT package
• MOU delineating HEASARC responsibilities
• End to end testing including archive ingest.
• Customized mission interfaces
• Individualized interface to Browse
• HTML wrappers for standard Browse scripts
• Links between Browse tables
• AKBAR new service to import external information
  from observers
• HEASARC responsible for some high-level data
  products e.g., overall burst evolution
• Mission requires rapid dissemination of data

15
Swift Browse Interface
16
Preparations for GLAST

• PDMP review
• Consultations on archive and catalog design
• Review proposed FITS formats

17
Standards

• FITS
• Subsumed GSFC FITS Office Web site
• Develop and coordinate domain conventions for
  high energy data (OFWG))
• Organize periodic HEADCC meetings
• FITS Libraries Support
• CFITSIO is essentially reference implementation
  of FITS
• HEASARC also supports widely used FITS libraries
  in Java and IDL.
• New C object implementation based on CFITSIO
• Standard Dataset IDs
• VO Standards
• Browse support for VO Cone search
• SkyView support for VO Simple Image Access
  Protocol
• VOTable (and FITS) outputs available through
  Browse
• VO Service Metadata descriptions for HEASARC
  resources
• Publication of metadata using Open Archives
  Initiative (OAI) protocol

18
Software Tools and Services

• HEASARC software is frequently downloaded
• fv (FITS browser and editor) 500 downloads/month
• CFITSIO library 500/month
• HEASOFT 200-300/month
• HEASOFT
• New packages for Swift and Astro E2 missions
  under active development
• Use HEADAS in order to provide a simple
  platform-independent procedure for new software
  builds and updates
• XSPEC New major release (12.0) in alpha test
• Complete re-write of Xspec internals in object
  oriented fashion
• Can support coded aperture mask data sets

19
Hera

• Provides a full, on-line analysis environment to
  users.
• Directly connected to archive, no downloads
  needed
• Up to date versions of HEASOFT tools
  automatically available
• Easily upload user data and download results.
• Uploaded data screened to ensure service is not
  used inappropriately.
• Uses standard HEASARC FV FITS browser tool as
  user interface.
• Why?
• Large volume of data (e.g., Integral, RXTE)
  downloads and/or software installation may be
  infeasible for some users
• With Hera, users can start using HEAsoft and
  other software without the need for local
  installation
• Status
• Initial system operational and being used
  actively.
• Proposal for continued funding has been submitted
• Both teacher and student interfaces in beta
  testing

20
New and Updated Catalogs

• 24 new catalogs or major catalog updates in last
  year.
• Major correlative catalogs VERONCAT, GB6, etc
• High energy source catalogs XMMSSC, BMWHRICAT,
  INTREFCAT
• Master catalogs updated regularly
• GRBCAT Compilation of all gamma-ray bursts
• VizieR connection for transparent access to
  massive catalogs (2MASS, USNO B, GSC 2)

21
Education and Outreach

• Education Web Sites
• Imagine the Universe (Grades 6-12) 500K
  hits/month.
• StarChild (Grades 1-8) in collaboration with L.
  Whitlock 1.25 M hits/month.
• New poster on origin of the elements
• X-ray analysis school
• Highly successful and oversubscribed
• Host for APOD Web site
• StarChild/Imagine/APOD CDs

22
Other Projects

• SkyView
• Java redesign
• New surveys (DSS2, Halpha, SFD) SDSS soon.
• Request for continued funding submitted
• ClassX
• Automated classification of entire RASS and
  WGACAT samples of X-ray sources using supervised
  learning algorithm

Density of source requests in SkyView
23
Virtual Observatory Involvement

• Data Inventory Service
• HEASARC metadata inventory
• Standardized metadata describing all HEASARC
  tables,data file types, surveys and services.
• Registered with VO to enable effective discovery
  of HEASARC resources.
• Table metadata
• HEASARC support for VO standards

24
Data Inventory Service

• Access to distributed services that use VO
  protocols.
• Many sites contributing data
• Dynamic service lookup using VO registries
• Catalog, observations, Images
• http//heasarc.gsfc.nasa.gov/vo/

25
Near Term Missions and Deadlines

• Launches
• Swift Fall 2004
• Astro-E2 Early 2005
• GLAST Early 2007
• HEASARC Senior Review Proposal Due March 17,
  2004

26
Current and Upcoming Issues

• Science
• May need greater emphasis on supporting
  gamma-ray astronomers as Integral and GLAST
  Archives come online and become an increasing
  fraction of the HEASARCs archive
• Must deal with ever increasing complexity of data
  and calibration
• Coded aperture masks (Swift, HETE2, Integral)
• Complex source dependent response ( GLAST)
• Multi-satellite observations (Con-X, LISA)

27
Current and Upcoming Issues

• Hardware
• CPU and disk space needs achievable, but
  bandwidth is a potential limitation
• Datasets may be growing faster than bandwidth
• Demand for data will spike after GRB detections
• Higher speed linkage to Internet possible but
  costly
• and doesnt help users who have slower
  connections
• Move towards more user processing at HEASARC
• Hera, grid computing
• On-line backup for archive is presently in same
  building as primary HEASARC archive.
• Planned move of backup archive to another
  building when space and network is ready (Summer
  2004)

28
Current and Upcoming Issues

• Local Software and Services
• Integration of tools and services
• Strengthen SkyView/Browse, Browse/NVODIS,
  Hera/Browse, cross-links.
• Enable coupling archive/web services and HEASOFT
  tools.
• Systematic attention to internal documentation of
  HEASARC systems