Atlas and the Grid

1
Atlas and the Grid

• Jennifer Schopf
• Argonne National Lab
• Northwestern University

GriPhyN Coordination Meeting University of
Chicago August 2-3, 2001
2
Atlas Structure

• Very large, international project (2000 people,
  40 countries)
• Many different funding sources, lines between
  them often unclear
• Currently - Various independent forays into grid
  technologies
• Next step Unification of approaches

3
Two software trees

• Suite of applications that make up the Atlas SW
  2 pieces currently in focus
• ATLSim
• Older, Fortran based
• Used by physicists for results today
• Generally CPU bound-1K to 10M events, several min
  per event
• Athena and ATLFast
• OO framework based on Gaudi (LHCb)

4
Virtual Data in the Atlas Arena

• In Atlas, an input file is often a previous
  output file
• Need for a data signature-
• Fully specify the environment, conditions,
  algorithm components, inputs etc. required to
  produce a given file
• Given a fully specified data signature, a data
  set could be reproduced
• NOT CURRENTLY POSSIBLE

5
Summary
Saul Youssef
Condor (G)
GRAPPA/ Gardner
GRAM
GridView/ De
GSI
MDS/ GIIS/GRIS
DBYA/ Wenaus
GridFTP
Grid Data Access/Malon
Replica Cat
Replica Mgr
6
Outline

• 1. Background
• 2. GRAPPA portal
• 3. DBYA - MySQL data replica
• 4. Grid Data Access - Athena with Data Replica
  Cat or GDMP
• 5. Grid View - monitoring
• 6. Next steps

7
GRAPPA Grid Access Portal for Physics
Applications

• Provide a point of access to ATLAS grid resources
• Collaborative effort
• Indiana (Physics and CS)
• Northwestern (ECE)
• Argonne (CS)
• Boston (Physics)

8
GRAPPA Plan

• Provide a simple interface for physicists to
  submit and monitor jobs on the Grid
• Web-based as well as script-based
• Ability to replay
• Compatible with
• ATLSim
• Athena architecture
• Adaptable and/or extensible to new developments
  in Grid software, Athena, etc

9
GRAPPA Components

• User Interface
• Job submission
• Monitoring
• Bookkeeping
• Resource selection

10
Use of IU Active Notebook project

• Tool to help design and manage grid experiments
• Five pieces
• The Java Tomcat web server
• A "database" of notebooks
• web pages, appl. configuration, and launch
  scripts.
• A Script Execution Engine (GridJPython)
• interfaces to GSISSH, GSIFTP, COGKit, etc
• A performance monitoring utility
• A SOAP-based Grid Event publish and subscription
  service

11
(No Transcript)
12
Current GRAPPA

• Uses Globus GRAM, GSI, etc
• Executes ATLSim jobs
• Provides bookkeeping
• Prototype running at IU
• Available for general beta-testing early
  September.

13
Next Steps

• Run with ATLFast/Athena jobs
• Incorporate existing tools for resource discovery
• Store user annotations with job metadata
• Display redirected stdout and stderr
• Stage input files with GSIftp
• Integrate Condor
• Interface with Athena Python scripting
• Interface with PPDG Job description language

14
Web Links

• Main GRAPPA page
• http//lexus.physics.indiana.edu/griphyn/grappa/
• Active Notebook page
• http//www.extreme.indiana.edu/an/index.html
• List of goals and objectives
• http//lexus.physics.indiana.edu/griphyn/grappa/GR
  APPA_Project_PhaseI_Updated.html
• Draft GRAPPA architecture
• www.cs.indiana.edu/hyplan/bramley/arch.html
• Robs GRAPPA talk from Atlas telecon 7/01
• lexus.physics.indiana.edu/griphyn/grappa/grappa-s
  tatus-7-01.ppt

15
Outline

• 1. Background
• 2. GRAPPA portal
• 3. DBYA - MySQL data replica
• 4. Grid Data Access - Athena with Data Replica
  Cat or GDMP
• 5. Grid View - monitoring
• 6. Next steps

16
DBYA- Database for yall

• Database rapid prototyping tool for design and
  development of distributed data management
  services
• Part of former NOVA project
• Some pieces will use Globus work
• Replica catalog, replica manager, grid ftp
• Some pieces are separate
• tool to crawl over ATLAS data stores and send
  their contents to the catalog
• metadata catalog

17
Status

• File catalog for CERN and BNL
• Resident ATLAS data
• Stable and running since May 2001
• 40k files, representing 2TB data
• Automated file replication tool between CERN and
  BNL
• Since July 2001

18
Next Steps

• Definition of logical file name/physical file
  name schemas
• Integration with Globus tools
• Separate project Algorithmic Virtual Data

19
Web Links

• DBYA info
• http//atlassw1.phy.bnl.gov/dbya/info
• Running version of the system
• /afs/rhic/usatlas/users/wenaus/dbya
• /afs/cern.ch/user/w/wenaus/public/dbya

20
Outline

• 1. Background
• 2. GRAPPA portal
• 3. DBYA - MySQL data replica
• 4. Grid Data Access - Athena with Data Replica
  Cat or GDMP
• 5. Grid View - monitoring
• 6. Next steps

21
Grid-Enabled Data Access in Athena

• David Malon, ANL
• CHEP paper
• Integrate Grid Data Access techniques (Globus
  replica catalog and/or GDMP) into the Athena
  event selection module
• When a file is needed, a check is done to see if
  the file is local
• if not, use Grid Data Access tools to make it
  local

22
Athena and Globus

• Search Globus replica catalog and select
• Transfer file using the protocol associated with
  the location object in the catalog (gsiftp,
  https/globus-url-copy from a remote gass_server,
  ...)

23
Athena and GDMP

• Use the GDMP import/export catalogs
• Supports certain subscription-based approaches
• Use GDMP to copy files

24
Status

• Paper submitted
• Prototype up and running on ANL systems
• Next Steps
• GDMP will use Globus data replica work
• Further integration with Grid tools (e.g., EDG
  WP1 co-allocation work)
• Metadata work extending the Athena Event
  Selector properties to allow for data
  signature/virtual data

25
Outline

• 1. Background
• 2. GRAPPA portal
• 3. DBYA - MySQL data replica
• 4. Grid Data Access - Athena with Data Replica
  Cat or GDMP
• 5. Grid View - monitoring
• 6. Next steps

26
GridView (Kaushik De)

• Tool to show the status of the 8 test bed
  machines on the web
• Uses Globus GRAM to query sites every 30 mins
• Hostname, Uptime, Idletime, users, and Load
  average.  

27
(No Transcript)
28
Next Steps

• Add pointer to additional information stored in
  the Globus MDS
• Java and/or Python implementation

29
Outline

• 1. Background
• 2. GRAPPA portal
• 3. DBYA - MySQL data replica
• 4. Grid Data Access - Athena with Data Replica
  Cat or GDMP
• 5. Grid View - monitoring
• 6. Next steps

30
Summary
Saul Youssef
Condor (G)
GRAPPA/ Gardner
GRAM
GridView/ De
GSI
MDS/ GIIS/GRIS
DBYA/ Wenaus
GridFTP
Grid Data Access/Malon
Replica Cat
Replica Mgr
31
Plan for Unification

• Developed a critical mass of people
• Various independent forays into grid technologies
  (necessary step in the direction of a unified
  program of work)
• We aren't completely there yet but soon

32
Next steps

• Working on both a software and a facilities plan
• Craig Tull (LBL, CERN) mapping of Athena
  functions to Globus
• Preliminary integration of Athena with
• Virtual data
• Resource management
• Gsi
• Design and requirement documents
• Testbed involving Athena/ATLFast
• 8 sites already running basic Globus

33
Additional Information

• Contact Info
• jms_at_mcs.anl.gov
• USAtlas Computing
• http//www.usAtlas.bnl.gov/computing/

34

• Extra slides follow this

35
In this example, a parameter file is generated
and saved on the host where execution will occur.
36
Then the job is executed with user supplied
command line parameters. For remote execution
GRAM is used
37
Parameters are automatically stored under the job
name for future use
38
(No Transcript)
39
(No Transcript)