GANGA, LHCb and ATLAS Planning

1
GANGA, LHCb and ATLAS Planning

• Glenn Patrick
• GRIDPP11 14 September 2004

2
ATLAS/LHCb GANGA Planning
LHCb
ATLAS
BaBar
User Grid Interface. Both experiments already use
common GAUDI/ATHENA framework. Different
analysis projects (DaVinci, DIAL).
3
GANGA User Interface
Submit job
Grid/Batch System Gatekeeper
Worker nodes
Send job output
File Transfer
Storage Element
Get Monitoring Info
Get job output
Send
Local Client
Ganga Job object
Ganga Job object
Ganga Job object
Ganga Job object
Ganga Job object
Job Factory (Job Registry Class)
Job Options Editor
Strategy Selection
Data Selection (Input/Output Files)
Job Requirements (LSF Resources, etc)
Database of Standard Job Options
Strategy Database (Splitting scripts)
4
User actions to submit an analysis job
Without Ganga
Using Ganga
5
GANGA ATLAS/LHCb

• GRIDPP1 (Karl Harrison, Alexander Soroko A.Tan,
  J.Martyniak) delivered a prototype in April 2003.
• Versions being deployed for 2004 Data Challenges.
• For GRIDPP2, develop production version capable
  of submitting full range of ATLAS/LHCb
  applications to various Grids. Support complex
  analysis environments.
• 7 Level-1 deliverables.
• 25 Level-2 deliverables.
• Level-1 deliverables relate to releases of
  increasing functionality leading up to real
  data.
• Large agreement between the two experiments.
  Still some fine tuning.

6
GANGA ATLAS LHCb

• Name Grid front-end for ATLAS analysis tasks,
  end-to-end
• analysis solution for LHCb.
• Why Shield the user from Grid middleware and
  provide an
• easy way to change between different Grid
• implementations.
• Client ATLAS, LHCb
• Achievement Effective and significant use by
  ATLAS and LHCb
• user communities.
• High level Risks
• Contention between ATLAS and LHCb objectives.
• Low take-up because of competition from ad hoc
  non- generic solutions.
• Sudden changes in experiment frameworks.
• Delays in middleware development.
• Delays in metadata project.

7
GANGA Level-1 Deliverables Deliverable G1 (end
Feb 2005) Optimise GANGA design for compatibility
with other relevant systems. (Metric software
release) Deliverable G2 (end April 2005) Release
with full support for plug-in modules. (Metric
software release and associated
documentation) Deliverable G3 (end October
2005) Release with full job-building
capabilities. (Metric software release and
associated documentation) Deliverable G4 (end
March 2006) Release with support for full range
of ATLAS and LHCb applications. (Metric software
release and associated documentation) Deliverable
G5 (end November 2006) Release that supports all
available functionality on the full range of
Grid/batch back-ends. (Metric software release
and associated documentation) Deliverable G6 (end
April 2007) Release that maximises Ganga
portability and allows for user
mobility. Deliverable G7 (end September
2007) Physics optimised release of GANGA. Metric
software release and associated documentation)
8
LHCb-specific GridPP Development

• LHCb development is on three
• fronts
• Production Control
• and Monitoring
• Gennady Kuznetsov (RAL)
• Data Management
• Carmine Cioffi (Oxford)
• GANGA (with ATLAS)
• Alexander Soroko (Oxford)
• Karl Harrison (Cambridge)

All developed in tandem with LHCb Data Challenges
UK Contribution DC2003 34 events DC2004 26
events
9
DIRAC Services and Resources
User interfaces
Production manager
GANGA UI
User CLI
Job monitor
BK query webpage
FileCatalog browser
FileCatalogSvc
BookkeepingSvc
DIRAC Job Management Service
JobMonitorSvc
DIRAC services
InfomarionSvc
MonitoringSvc
JobAccountingSvc
AccountingDB
Agent
Agent
Agent
DIRAC resources
DIRAC Storage
LCG
Resource Broker
DIRAC Sites
CE 3
DIRAC CE
gridftp
bbftp
DIRAC CE
DIRAC CE
DiskFile
CE 2
CE 1
rfio
10
LHCb Production Control

• GRIDPP1 (Gennady Kuznetsov)
• provided the Production Desktop
• for building complex workflows from
• components DC04.
• For GRIDPP2, a Production Console
• is being written to support/configure
• the production environment for the
• Production Manager.
• 6 Level-1 deliverables.
• 17 Level-2 deliverables.
• Dependent on Data Challenge
• schedule, LCG releases, etc .
• ? 2 GRIDPP1 deliverables included.
• ? All deliverables well established.
• ? Agreed with post-holder.

11
LHCb Production Control

• Objective Production Control and Monitoring
• Name Production Control and Monitoring
• Why Provide environment for Production Manager
  to control and monitor LHCb production.
• Client LHCb Collaboration
• Achievement Provision of effective central
  control of LHCb
• production so that system can be run by single
  person.
• High level Risks
• Undermining by changing underlying technology
• (Grid deployments, etc).
• Loss of expertise or delays due to possibility
  of illness/relocation of single developer.
• Radical changes in requirements due to
  fundamental change in philosophy of distributed
  production system and data model.

12
LHCb Production Control Level-1
Deliverables Deliverable LP1 (2004, end
Q4) Integration of Production Desktop with DIRAC.
(Metric software release. demonstration) Deliver
able LP2 (2005 end Q1) Requirements and
Architecture of the Production Console. (Metric
documents) Deliverable LP3 (2005 end Q2) Release
0 (first prototype) of the Production Console for
2005 Data Challenge. (Metric software
release) Deliverable LP4 (2006 end Q1) Release 1
for 2006 Data Challenge with capability for
continuous production rather than periodic data
challenges. (Metric software release,
demonstration) Deliverable LP5 (2006 end
Q4) Release 2 of the Production Console with data
management functions and to support processing of
real data. (Metric software release,
demonstration) Deliverable LP6 (2007 end
Q3) Final release based on production experience
of real collisions (if available). (Metric
software release, demonstration)
13
LHCb Data Management

• GRIDPP1 (Carmine Cioffi) contributed to the
• POOL File Catalogue Browser and integration
• of the POOL persistency framework into
• Gaudi (new EventSelector interface).
• Focus for GridPP2 is on metadata and the
  deployment of a new Metadata Catalogue Service.
  Optimised for LHCb, but work with common ARDA and
  UK metadata projects (web-service technology,
  ARDA test-suite).
• 9 Level-1 draft deliverables.
• 23 Level-2 draft deliverables.
• This post had no formal GRIDPP1 milestones.
• Need to also align with deliverables of UK
  metadata posts.
• Still iterating - under discussion this week
• (LHCb s/w week).

14
LHCb Data Management

• Objective Data Management
• Name Data Management
• Why A dynamically optimised and high
  availability database service is required to
  provide a metadata and job provenance catalogue
  for the experiment. This will provide efficient
  storage and query mechanisms to the experiment
  for metadata job provenance info.
• Client LHCb Collaboration
• Achievement Deployment of a Metadata Catalogue
  Service meeting the full demands of the LHCb
  experiment for metadata storage with
  significant UK input.
• High level Risks
• Incompatibilities with future common
  technologies.
• Radical changes in metadata requirements due to
  fundamental
• change in philosophy of distributed production
  system and data
• model.
• Continued existence of ARDA project for those
  deliverables that
• envisage common solutions/middleware for all
  LHC experiments.

15

• LHCb Data Management Level-1 Deliverables
• Deliverable LD1 (2004, end Q2?)
• Evaluation of the most efficient distinction
  between metadata and job
• provenance within the LHCb context as a
  contribution to ARDA.
• (Metric public document)
• Deliverable LD2 (2004, end Q3?)
• Implementation of the prototype ARDA interface
  based on the
• existing LHCb book-keeping database as a backend.
• (Metric Software release compatible with DIRAC
  and GANGA)
• (Dependencies GANGA and DIRAC interfacing)
• Deliverable LD3 (2005, end Q1?)
• Tests and evaluation of prototype interface for
  2005 Data Challenge.
• (Metric interface deployed for DC05, user
  survey, document)
• (Dependencies schedule for 2005 Data Challenge)
• Deliverable LD4 (2005, end Q3)
• Evaluate performance of alternative prototype
  implementations of
• interface to catalogues from other experiments.
• (Metric document)
• Deliverable LD5 (2006, end Q1)

16
LHCb Data Management Level-1 Deliverables
continued Deliverable LD6 (2006, end
Q3) Release prototype of catalogue. Grid enabled
through an ARDA approved web-service
interface. (Metric software release,
document) (Dependencies ARDA agreed web-service
interface) Deliverable LD7 (2007, end
Q1) Prototype of general suite of tools to ensure
a production quality service. These tools will
monitor performance, availability and integrity
of the catalogue. Identify solution for private
metadata. (Metric software release,
document) Deliverable LD8 (2007, end
Q2) Production release of catalogue and tool
suite to support processing of real data.
(Metric Software release) Deliverable LD9 (2007,
end Q3) Revised production (physics ready)
release incorporating user feedback, updated
documentation and private metadata
functionality. (Metric production software
release) (Dependencies availability of real data
collisions at LHC)
17
ATLAS-specific plans

• RWL Jones
• for GridPP11, 14th Sept 2004

18
ATLAS-specific plans

• All ATLAS plans are integrated with the GANGA
  developments and with the ATLAS eScience project
  and responsibilities
• Focus is on the development of a working
  fully-functional analysis environment
• Metadata is a key issue, identified in the
  proposal. This effort is now a major strand of
  the metadata project, and so will report in that
  strand but also to ATLAS
• Non-metadata GridPP manpower is Alvin Tan 50
  of Frederic Brochu

19
Release validation and integration

• GridPP (F Brochu) has provided an essential
  component of the ATLAS validation and integration
• It is essential that this continue with LCG/EGEE
  and other releases (Objective 1.1)
• Timing becomes less certain as these projects do
  not have a well-developed timetable of major
  releases to 2008 and have a bad track-record of
  functional releases!

20
New ATLAS Production System
Much of the problem is data management This must
cope with gt 3 Grid catalogues The demands will
be greater for analysis
21
Production Tools and Framework

• Alvin and Frederic have made important
  contribution to the ATLAS Grid production system.
  (Objective 1.2)
• The overall system has the right structure, but
  the performance in DC2 is inadaquate
• Major revision in the next 9 months
• Periodic review thereafter
• The system must continually adapt to new
  middleware releases (executors for each system)

22
ATLANTIS visualisation on the Grid

• Several applications need to be Grid-integrated
• The most important is ATLANTIS event and detector
  graphical display (Objective 2)
• UK responsibility
• Evolving workflows
• Will help other application integration
• Work co-ordinated with effort in the ATLAS
  eScience programme

23
Schedule of Deliverables

• Deliverable A1 (2005 end Q3, Objective 1)
• Refactored ATLAS production system integrated
  with EGEE middleware.
• (Metric software release)
• Deliverable A2 (2005 end Q4, Objective 2)
• ATLANTIS Grid usage using plug-in operating the
  harvesting model for DC3 data and the metadata
  tools. )
• (Metric demonstration, software release)
• Deliverable A3 (2006 end Q4, Objective 1)
• Distributed analysis and production in
  DC4/Commissioning run.
• (Metric Analysis job submission, 90 or greater
  success rate.)
• Deliverable A4 (2006 end Q4, Objectives 1 and 2)
• Releases for commissioning run.
• (Metric software release, documentation)
• Deliverable A5 (2007 end Q3, Objective 2)
• Physics-ready release of ATLANTIS plug-in.
• (Metric software release)
• Deliverable A6 (2007 end Q3, Objectives 1 and 2)
• Final release of Grid analysis and production
  tools for data-taking.
• (Metric software release)

24
(No Transcript)