GANGA Project

1
GANGA Project

• Aims and Goals
• Project Organization
• Design
• Current implementation
• Future plans

ARDA Meeting, 22 July 2003 P. Mato / CERN
2
The Vision

• Help the Physicist to do its analysis work
• Help configuring and submitting analysis jobs
  (Job Wizard)
• Help users to keep track of what they have done
• Hide completely all technicalities
• Provide a palette of possible choices and
  specialized plug-ins (pre-defined application
  configurations, bath/grid systems, etc.)
• GANGA knows what the user is running, so it can
  really help him/her.
• Single desktop for a variety of tasks
• Friendly user interface is essential

3
GANGA Gaudi ANd Grid Alliance
GANGA
GUI
Collective Resource Grid Services
Histograms Monitoring Results
JobOptions Algorithms
GAUDI Program
4
Required Functionality

• Job preparation and configuration
• Resource booking, input selection, application
  configuration, etc.
• Job submission
• Selection between interactive/batch, selection
  between Grid and local system, etc.
• Job monitoring and control
• Monitor job execution, control, display results,
  etc.
• GUI for the resource browsing
• VO active services, CEs, SEs
• Query existing files in the Grid
• GUI for data management tools
• Dataset registration, copy file, replication,
  etc.

5
Goals

• Short-term aims
• Help with configuring and running Gaudi-based
  applications
• Help with submitting and monitoring jobs to/on
  distributed (Grid) and local batch systems
• Longer-term aim
• Develop Gaudi services for use in Grid
  environment (enable querying of replica
  catalogues, enable publication of information for
  Grid monitoring, etc.)

6
Gatekeeper
Submit job
Send script
Request software
Receive software
Send job output
Send job output
User running Ganga client
Worker nodes
Request software
Receive software
Software server
7
Project Organization

• GANGA is being developed as an ATLAS/LHCb common
  project, with support in UK from GridPP
• Current main contributors to GANGA are
• Developers K.Harrison, W.Lavrijsen, A.Soroko,
  C.L.Tan
• Technical direction P.Mato, C.E.Tull
• GridPP coordination N.Brook (handing over soon
  to G.Patrick), R.W.L.Jones
• Ganga-related information regularly updated on
  web site http//ganga.web.cern.ch/ganga
• A mailing list has been active since November
  2002 project-ganga_at_cern.ch
• Usually have telephone meeting at least once
  every two weeks
• details of times are placed on web site, and are
  circulated to mailing list
• Presentations of Ganga status and plans given at
  various other meetings of ATLAS, LHCb and GridPP

8
Design

• User has access to functionality of GANGA
  components both through GUI, and through CLI,
  layered one over the other above the software bus
• Components used by GANGA can be divided into
  three categories
• Ganga components of general applicability (to
  right in diagram)
• Ganga components providing specialized
  functionality (to left in diagram)
• External components (at bottom in diagram)

9
Components of General Applicability

• Components potentially have uses outside ATLAS
  and LHCb
• Could be of interest for LCG PI project and other
  eScience applications
• Core component provides classes for job
  definition, where a job is characterized in terms
  of name, workflow, required resources, status
• Workflow is represented as a sequence of elements
  (executables, parameters, input/output les, etc.)
  for which associated actions are implicitly
  defined
• Required resources are specified using a generic
  syntax

10

• Other components perform operations on, for, or
  using job objects
• Job-registry component allows for storage and
  recovery of job information, and allows for job
  objects to be serialized
• Script-generation component translates a job's
  work flow into the set of instructions to be
  executed when the job is run
• Job-submission component submits work flow script
  to target batch system, creating JDL
  (job-description language) file if necessary and
  translating resource requests as required
• File-transfer component handles transfer between
  sites of input and output files, adding
  appropriate commands to work flow script at
  submission time
• Job-monitoring component performs queries of job
  status

11
Components Providing Functionality for LHCb and
ATLAS

• Components incorporate knowledge of the Gaudi
  framework
• Component for Gaudi job definition adds classes
  for work flow elements not dealt with by
  general-purpose job-definition component, for
  example applications packaged using CMT
  component also provides work flow templates
  covering common tasks
• Other components provide for job-option editing,
  job splitting, and output collection

12
External Components

• Additional functionality obtained using
  components developed outside of Ganga
• Modules of python standard library
• Non-python components for which appropriate
  interface has been written, for example Gaudi
  framework itself (GaudiPython) and ROOT (PyROOT)

13
GUI

• The UI/GUI is an essential part of GANGA
• Design inspired by e-mail reader metaphor
• Configuring and submitting a job should be as
  easy as composing and sending an e-mail
• Jobs are organized in folders. Some of them
  associated to states
• Commands to clone, split, (re)submit, etc.
• Embedded interpreter to execute commands
• User scripts (repetitive or bulk operations)
• External scripts (interoperability)

14
Status on the GUI

• GUI is created using wxPython extension module.
• All job configuration data are represented as a
  hierarchical structure accessible via tree
  control.
• Most important job parameters are brought to the
  top of the tree.
• A simplified User view makes access to the top
  level parameters more easy.
• All job parameters defined by the user can be
  edited via GUI dialogs.
• All implemented tools are available via GUI, but
  some of them require more elaborated interface,
  e.g., Job Options browser/editor.
• Python interpreter is embedded into the GUI and
  allows user to configure interface from the
  command line.

15
Ganga GUI
Job parameters
Jobs tree
Embedded Python interpreter
16
Ganga GUI (2)

• User view
• Access to main job parameters is more
  straightforward
• User can add, e.g., a new input file with a few
  mouse clicks

17
Python Software Bus

• Prototype software bus, PyBus, has been developed
  as user-level python module, with no privileges
  over other modules
• Standard python modules correctly handled by
  PyBus
• PyBus components treated as modules by python
  interpreter
• Allows component to be loaded by logical,
  functional or actual name
• First two not necessarily unique choose on basis
  of PyBus conguration, priority scheme or user
  input
• Performs complete unloading and reloading of
  components, and allows component replacement
• Allows components to register their parameters,
  permitting component conguration

18
Cloning and Splitting of Jobs

• Sub-jobs from cloning or splitting a Gaudi/Athena
  job are near copies of one another, but are
  distinguished by name and may have a different
  values for one or more of the job-option
  parameters
• Experimenting with generic approach, where a
  splitting function returns, for all sub-jobs,
  the job-option parameters that differ from those
  of the initial job
• Splitting functions for common cases should be
  supplied with Ganga more specialized splitting
  functions can be added by the user
• In the typical DaVinci case, the splitting
  function should examine the list of input les
  associated with EventSelector.Input in the job
  options, and assign some group of les to each
  sub-job
• The Ganga job handler dispatches the sub-jobs,
  and stores information for each in separate
  directories

19
Release Roadmap

• Jul 28, 2003 - GANGA Release 1.1
• Job Options Editor
• Pure-Client submit
• LXPLUS_LSF, OXFORD_PBS
• PDSF_LSF, BNL_LSF
• PDSF_Gatekeeper, BNL_Gatekeeper
• Magda
• Sep 29, 2003 - GANGA Release 2.0
• Pre-execute Release Installation
• NorduGrid tools
• CET Scripts
• Pacman
• PyBus
• GUI CLI Integration

• Nov 10, 2003 - GANGA Release 3.0
• Enhanced Monitoring
• NetLogger Events
• Enhance the Current GANGA Model
• Data Management
• POOL File Catalog
• Web Portal Interface
• Grappa Integration

20
Conclusions

• The aim of Ganga is to facilitate the Physicist
  to submit and manage the analysis jobs in a Grid
  environment
• Design based on a python bus with customizable
  plug-ins and interfaces to a verity of systems
• A lot of work has been done on GANGA since its
  launching (May 2002)
• Code for GANGA v1r0 is in Gaudi CVS repository
• Ganga v1r0 is not production quality, but is
  useful for giving a feel of how things should
  work
• Ganga team has a well-defined plan of additions
  and improvements, but would welcome user feedback
  on what is already implemented, and on priorities
  for the things that are missing