The ETICS Job management architecture

1
The ETICS Job management architecture
EGEE 08 Istanbul, September 25th 2008
Valerio Venturi INFN - ETICS valerio.venturi_at_cnaf.
infn.it
2
Outline

• Deploying to the Grid infrastructures
• Identification of deployment issues
• Re-engineering the ETICS services
• Submitting ETICS jobs to the EGEE infrastructure
• Technical issues
• Organizational issues
• Submitting ETICS jobs using UNICORE
• Submitting ETICS jobs to EC2

3
Integrating ETICS in Grid InfrastructuresProblem
statement

• Software projects maintain a variable-sized pool
  of machines for the various tasks of development
  and maintenance
• Building software releases
• Running continuous builds of the development
  release
• Testing the software
• For a number of reasons (needs of
  high-availability for release-rush, needs of
  possibly a lot of different platforms) this
  machines are likely to have high idle times.
• When a software projects happens to have other
  computing needs, it would great benefit to be
  able the idle time of their resources for their
  other computing needs.

4
Integrating ETICS in Grid InfrastructuresProblem
statement

• Providing resources for software development
  tasks is what ETICS does for its customers.
• So far the machines are owned and managed by the
  ETICS project, thus eliminating the problem of
  allocating resources.
• Not sustainable. Where is gLite going to be build
  in a couple of years from now?
• We propose integrating the ETICS services in
  infrastructures (such as EGEE) to solve the
  sustainability problem and the more general
  problem of wisely using the resources projects
  allocate to software development tasks.

5
Integrating the ETICS servicesAnalysis of
deployment issues
Repository
Web Service
Web
Build/Test
Portal
Artefacts
Report/Metrics
DB
Configuration
Web Service
Configuration
DB
Execution Engine
Command Line
User Interface
ETICS Infrastructure
1
6
Build/Test Jobs Management in ETICSArchitecture
ETICS Submission Engine
7
Build/Test Jobs Management in ETICSRe-engineering
the ETICS services

• This wouldn't be necessary if the various
  infrastructures had interoperable interfaces and
  security mechanisms
• Could have used BES and VOMS to submit to both
  gLite and UNICORE
• We will separate the part of the ETICS service
  that handle the jobs submission so that it can be
  deployed separately
• A submitting interface implemented using
  Metronome over a Condor pool, gLite over the EGEE
  infrastructure, and UNICORE over DEISA
• Operations to submit, cancel, getStatus
• Why not a standard interface like BES?
• ETICS jobs are pretty simple, doesn't really fit
  the bill

8
Submitting ETICS Jobs to EGEEOverview
1
9
Submitting ETICS Jobs to EGEETechnical issues

• We don't envision big problems from a technical
  point of view, as gLite is thought to be easily
  integrated
• The WMS has a WS interface that makes it pretty
  easy to use
• Jobs are described using an established language
  as JDL
• The GLUE Schema allows for a (quite) standard
  description of resource requirements
• Use information published by sites through the
  GLUE Schema elements to land on nodes with the
  required architectures, operating system and
  compiler
• Future works on virtualizations will eliminate
  this problem
• Unfortunately VOMS doesn't a WS interface
• Hopefully gLite can get to it in the future

10
Submitting ETICS Jobs to EGEEOrganizational
issues

• In order to have access to the EGEE
  infrastructure, you have to be in one of the EGEE
  VOs.
• Two solutions
• Apply for an ETICS VO, moving some resources to
  the infrastructures and gaining share quotas
• Use the current EGEE Vos
• With either solutions, we have to decide whether
• Use delegation to submit the ETICS jobs with the
  user credentials
• Have the ETICS service registered to the VO and
  knows which credentials to use for submitting jobs

11
Submitting ETICS Jobs to EGEEAn ETICS Virtual
Organization?

• Apply for an ETICS VO
• Moving part of the nodes that are currently under
  exclusive ETICS management to the EGEE
  infrastructure
• Getting a quota with which to run build and test
  jobs
• In this case ETICS will manage its internals
  resource sharing, assigning priority and quotas
  to its projects.

12
Submitting ETICS Jobs to EGEEAn ETICS Virtual
Organization?

• PROS
• Simple solution
• ETICS may provide resources for building projects
  that are not in a VO and so don't have resources
  on the infrastructure
• CONS
• Not complete integration, ETICS would buy
  resources on the infrastructure and act as a
  broker to projects using it
• Does ETICS fit the profile for a VO
• Typically scientific collaborations

13
Submitting ETICS Jobs to EGEEIntegrating with
the current EGEE VOs

• ETICS uses an EGEE VO credentials for submitting
  build jobs, thus consuming resources assigned to
  that VO
• If a VO decide to use ETICS for building its
  software, it can use its resources shares for
  doing that
• For example, the gLite middleware builds could be
  accounted to DTEAM
• Two ways of doing that
• Using delegation
• The VO is able to grant single users the
  permission to build and test, and have fine
  grained control
• Having the ETICS service credentials registered
  in the VO, and the ETICS services maintaining a
  mapping to know which VO credentials to use for
  each builds

14
Submitting ETICS Jobs to EGEEIntegrating with
the current EGEE VOs

• Real-life example, building the gLite middleware
• The DTEAM VO in EGEE is used by gLite developers
  for testing the developments
• The DTEAM VO may have developer, integrator,
  tester, roles (or groups)?
• The DTEAM VO assigns shares to those roles, so
  that importance of the work is reflected
• This can be done dynamically, so to handle
  possible bug-fix-release-rush situations
• Developer, integrator and tester submits their
  jobs and these are executed on the infrastructure
  according to the policy decided by the VO
• Can be handled both by using delegation or the
  ETICS user in the VO

15
Submitting ETICS Jobs to EGEEIntegrating with
the current EGEE VOs

• PROS
• Complete integration, resource shares are
  arranged within EGEE and not between EGEE and
  ETICS
• CONS
• Projects that don't have EGEE shares won't build
• Registering the ETICS service credentials to the
  VO may scare someone
• Delegation would be the way round this
• The VO would assign to the ETICS service the same
  level of trust it assigns to its user
• The system provides controls to stop misuses

16
Submitting ETICS Jobs to EGEEMixed approach

• We will go both ways, and will use the ETICS VOs
• We'll be able to provide resources for those
  projects that don't have shares on the
  infrastructures
• Will compete for resources, and likely have low
  priority
• We'll let VOs use their shares to build and test
  their software
• According to the priority that the
  infrastructures assign them, and the VO choices

17
Submitting ETICS jobs to UNICORE

• The ETICS project is going to implement a
  submitter able to send jobs to UNICORE, so to be
  able to integrate into infrastructures using it
  (such as DEISA)
• Submits to UNICORE OGSA-BES
• Describe the job using JSDL

18
Conclusions

• The ETICS System provides a grid-based remote
  build and test functionality
• The current implementation is based on Condor,
  but generalization and extension to additional
  infrastructures is required
• The ETICS submission mechanism has been
  generalized to be able to plugin different
  middleware implementations
• We are adding support for EGEE and gLite and
  investigating the possible authorization and
  usage models based on the VO concept
• Further extensions to UNICORE and other
  implementations are foreseen