Grid korisnicki pristup i razvoj aplikacija

1
Grid - korisnicki pristup i razvoj aplikacija
Meteo VO Training, Belgrade 24. June 2008
Branko Marovic RCUB - UoB
2
SEE-GRID-2 Applications Adaptation Support

• Application lifecycle defined in Gridification
  Guidelines document.
• SG Gridification Guide (wiki pages) structured
  and contributed by developers.
• Policy Applications initially supported at the
  country level.
• Application Support Group (ASG) created for
  gridification support.
• Application Review Committee(ARC) created to
  assess existing and new candidate applications.
• Application selection criteria defined by ARC.
• MoU Between Applications and SEEGRID-2.

3
SEE-GRID-2 Strategy for New Applications

• There is Gridification Guideline document to
  provide initial help to developers. (application
  life-cycle, policy and methodology to run and
  support an application on Grid)
• There is SG Gridification Guide on SEE-GRID-2
  Wiki to provide comprehensive help to developers
  (porting an existing application to the Grid)
• ARC (Application Review Committee)
  recommendations push applications towards
  collaborations with other applications and
  projects.
• Applications are required to provide update on
  their established user community and
  collaborations within their periodic Application
  Progress Report.

4
(No Transcript)
5
SEE-GRID2 Applications
6
Support the deployed applications

• Application support group (ASG) created to
  provide fast and continuous gridification support
  to developers.
• Experienced developers admins
• ASG team members assigned for each application.
• National level application support
• SEE-GRID - global level application support
• SG Gridification Wiki online.
• SEE-GRID helpdesk available and used for
  problems.
• Useful Helpdesk Tickets for Developers.
• SEE-GRID Helpdesk Applications Support
  knowledge base.
• GOODs responsible to dump useful tickets of the
  week.
• Work in close collaboration with WP5 (training)
  and WP3 (software requirements, maintenance of
  performance)

7
Application Lifecycle
8
SEE-GRID-2Gridification Guides

• An initial guide for the developer including the
  application life-cycle, VO and site policies,
  gridification methodology
• Gridification Guideline
• http//www.see-grid.eu/content/modules/downloads/S
  EEGRID2-WP4-TR-008-Gridification_Guideline-n-2006-
  11-22.doc
• Detailed wiki pages with illustrations for
  gridification issues
• SG Gridification Guide
• http//wiki.egee-see.org/index.php/SG_Gridificatio
  n_Guide

9
SEE-GRIDGridification Guide

• Relevant topics for application developers
  identified trough online questionnaire system
• Some investigation areas identified as well
  candidates for future GG topics
• Gridification guide will provide information on
  these topics
• Wiki as collaboration medium

10
SEE-GRIDGridification Guide

• 1 Job Management
• 1.1 Running Jobs (T1)
• 1.2 Software Installation Management (T3)
• 2 File Management and Access
• 2.1 Data Management (T2)
• 2.2 Data Access (T4)
• 2.3 Logical Files (T6)
• 2.4 Advanced Replica Management (T2, T6 related
  investigation area)
• 3 Inter-Job Communication
• 3.1 Workflows (T7)
• 3.2 MPI (T9)
• 3.3 Interactive Pilot Jobs (T1 related
  investigation area)
• 3.4 Java networking
• 4 Information Services
• 4.1 Application-Level Event Logging Performance
  (T5)
• 4.1.1 Applications and Systems monitoring tools
• 4.1.2 Benchmarking
• 4.2 Metadata Catalogues
• 5 Development Environments and Portals

11
(No Transcript)
12
VO Resource Allocation Policies

• Seegrid VO is used for all applications
• Utilization of groups and roles/seegrid/COUNTRY_
  CODE/SHORT_APP_NAME/Roleproduction/seegrid/COUNT
  RY_CODE/SHORT_APP_NAME/Rolesoftware
• For gridification testing phase,
• no role or group is used, all sites support all
  applications.
• For deployed applications,
• sites are encouraged to support as many
  applications as possible.
• MoU template between SEE-GRID-2 Project and the
  Application created.

13
SEE-GRID-SCI
14
Meteorology VO

• Members
• National Observatory of Athens (coordinator)
• South Environment and Weather Agency of Serbia
• Hydrometeorological Institute of Montenegro
• RBI of Zagreb
• University of Zagreb
• Federal Hydro-Meteorological Institute of Bosnia
  and Herzegovina
• Republic Hydrometeorogical Institute, Banjaluka
• Two sets of applications for initial deployment
• Regional scale Multi-model, Multi-analysis
  ensemble forecasting system
• Study of the interaction of airflow with complex
  terrain

15
Developer Resources

• Grid environment is constantly evolving, but
• Useful features persist
• New are constantly being added
• Bugs are being fixed
• Gained knowledge remains relevant, must be
  updated
• Applications can be easily migrated to
  new/updated APIs
• gLite User Guide
• https//edms.cern.ch/file/722398/1.2/gLite-3-UserG
  uide.pdf
• WMS user guides
• https//edms.cern.ch/document/572489/1
• SEE-GRID Gridification Guide
• http//wiki.egee-see.org/index.php/SG_Gridificatio
  n_Guide
• SEEGRID Wiki
• http//wiki.see-grid.eu/
• gLite documentation
• http//glite.web.cern.ch/glite/documentation/

16
Grids a foundation for e-Research
Enabling a whole-system approach Collaborative
research / engineering / public service
17
Nacini upotrebe grida

• Komandna linija, JDL, input/output sandbox i/ili
  rucna manipulacija podacima
• Primena skriptova u generisanu poslova, obradi i
  manipulaciji podacima
• Koricenje postojecih Grid portala
• Primena grid API-ja i servisa iz koda koji se
  izvrava na gridu
• Pravljenje aplikativno specificnih pristupnih
  tacaka-servisa

18
Complexities of grid applications

• Simple jobs submitted to WMS to run in batch
  mode
• Job invokes grid services
• To read write files on SE
• Monitoring
• For outbound connectivity (interactive jobs)
• To manage metadata
• Complex jobs
• An environment controls multiple jobs on users
  behalf
• High-level services
• Portals with workflow
• Software written for the VO (or by the user)

19
Invocation of applications

• From the UI
• Command Line Interfaces / Scripts
• APIs
• Higher level tools
• From desktop Windows applications
• Use Grids without awareness of them!
• But gLite not (yet) fully supporting Windows
• (more info http//jessica.trigrid.it/grid2win/
  )
• From portals
• For recurring tasks core grid services as well
  as application layer
• Accessible from any browser
• Tailored to applications
• Different portal solutions, and wide range of
  capabilities. Second part of this course P-GRADE
  Portal

20
Characteristics of VOs

• What is being shared?
• resources of storage and/or compute cycles
• software and/or data
• Distinct groups of developers and of users?
• Some VOs have distinct groups of developers and
  users
• Biomedical applications used by clinicians,.
• . Some dont
• Physics application developers who share data but
  write own analyses
• Effect need to
• hide complexity from the 1st type of VOs
• expose functionality to 2nd type of Vos
• many security issues

21
Challenges

• Research software is often
• Created for one user the developer
• Familiarity makes it useable
• Short-term goals Used until papers are written
  and then discarded

• Grid applications are often used
• by a VO
• Without support from developer
• In new contexts and workflows

• Grid application developers are
• In a research environment
• Yet their s/w must have
• Stability
• Documentation
• Usability
• Extendability
• i.e. Production quality

• Need expertise in
• software engineering
• application domain
• grid computing

22
Consequences

• Team work!
• Engaged in world-wide initiatives reuse, dont
  make your own! Cross disciplines for solutions.
• From research to production software 5 times
  the effort.
• 80 of the time for last 10 of the
  functionality reliability
• Standardisation is key
• For re-use, for dynamic configuration of
  services,..
• Both for middleware and domain specific
• Need to follow a deliberate development process
• Waterfall? Rapid prototyping?
• Requirements engineering, design, implementation,
  validation, deployment
• Engaged with the user community

23
Basic tasks while Porting applications to the Grid

• Developing a non-grid application (or inheriting
  and updating an ancient one)
• Go/no-Go decision about gridification
• Is it suitable for the Grid environment?
• Cost/profit analysis / feasibility study
• Typical Questions Groups
• Current structure of the application
• Dependencies of the application
• Available resources (manpower, knowledge, etc.)
• Requirements for the gridified application
• Expected impact of gridification
• Requirements for the grid infrastructure
• More info Application Description Template
• http//www.lpds.sztaki.hu/gasuc/?m4

24
Basic tasks while Porting applications to the
Grid (contd.)

• Grid environment access
• Requesting Certificates / VO membership
• Accessing Grid environment
• Appropriate VO UI machine account for command
  line
• Portal GUI account
• Executing, Testing and Debugging the application
• Testing the non-grid application (debugging in
  Grid environment is a hard task), creating use
  cases for single (non-grid) runs
• Constructing the job suite JDL (Job Description
  Language) files, executables, auxiliary scripts
  and input/output data files
• Submitting the job to the Grid as small-scale
  pilot application

25
Basic tasks while Porting applications to the
Grid (contd.)

• Executing, Testing and Debugging the pilot
  application
• IF something goes wrong
• THEN GOTO 4
• IF everything seems to work
• THEN increase the scale of the application
  (increase problem size, amount of used
  resources)
• Optimizing the grid application