BalticGrid

1
BalticGrid

• Focus on Cracow contributions

Mariusz Witek IFJ PAN
2
Outline

• PART I About Project
• A bit of history
• Basic information partners, objectives,
  activities
• Cracow contibutions
• HEP applications
• BaltiGrid impact on IFJ PAN
• Future
• PART II Tools development
• BAT
• OCM-G
• OCM-G Java API
• CANDLE

3
History

• 2003 white places on the EGEE map
• F. Galliardi, P. Öster, M. Turala idea of EGEE
  extension to Baltic countries
• 2004 meeting in Vilnius
• A. Kusznir, P. Nowakowski presentations about
  CrossGrid
• 2005 meeting in Tallin,
• proposal preparation
• 2005 - negotiations in Brussels

4
BalticGrid in One Slide

• Partners
• 10 Leading institutions in six countries in the
  Baltic Region and Switzerland (CERN)
• Budget
• 3.0 M
• Coordinator
• KTH PDC, Stockholm
• Compute Resources
• 17 resource centres
• Duration
• 30 Months
• Started 1 November 2005

SA - Specific Service Activities NA - Networking
Activities JRA - Joint Research Activities
5
BalticGrid Partners
Details on www.balticgrid.org

• Estonia
• Tallinn, Tartu
• Lithuania
• Vilnius
• Latvia
• Riga
• Poland
• Kraków, Poznan
• Switzerland
• Geneva
• Sweden
• Stockholm

6
(No Transcript)
7
Timeline
Second Review
First review
Kick-off
EAC
EAC
EAC
1st AHM
2nd AHM
3rd AHM
Summer School
Summer School
8
Seven ActivitiesIntegrated Infrastructure
Initiative (I3)

• Networking Activities
• NA1 Management of the I3
• NA2 Education, Training, Dissemination and
  Outreach
• NA3 Application Identification and Support
• NA4 Policy and Standards Development
• Specific Service Activities
• SA1 Baltic Grid Operation
• SA2 Network Resource Provisioning
• Joint Research Activity
• JRA1 Account Service Level Agreements, Markets
  and Dynamic Account Management

• NA2, NA3, SA1 significant Cracow contributions

9
Education Training Dissemination and Outreach
Activity leader Zofia Mosurska
10
Grid Operations
(Status end of 2006)
Activity Leader Lauri Anton Cracow Coordinator
Marcin Radecki
11
Application Identification and Support
Activity Leader Algimantas Juozapavicius Applicat
ion Support Coordinator Tomasz Szepieniec IFJ
PAN 5 persons to support 20 users (LHCb, Atlas)
12
HEP at IFJ PAN applications for BalticGrid

• Experimental applications ready to use on GRID
  (ATLAS, LHCb in Krakow )
• Monte Carlo production of Atlas and LHCb
• Relatively easy installation of experimental
  software
• Comprehensive test of LCG
• Statistical methods of data analysis (LHC
  experiments)
• CPU intensive studies of expected sensitivity of
  a given measurement

13
HEP at IFJ PAN LHCb experiment
14
HEP at IFJ PAN ATLAS experiment
15
BaltiGrid impact on IFJ PAN

• Start of LHC early next year
• Grid infrastructure has to be ready on time
• Development of local GRID installations
• Access GRID from local UI
• Support for HEP users
• Installation of experimental applications
• Development and tests of user algorithms
• Submit jobs to GRID distributed analysis

• Mini cluster (blade technology)
• Purchase from associated national project
• 32 cores, 2 GB RAM/core, 2 TB disks
• To be extended in future (local Tier 3)

16
Future

• Continuation BG4Science, proposal submitted in
  May
• Extension of the BalticGrid infrastructure to
  Belarus and eventually Ukraine
• New participants (13 in total 10 old 3 new)
• 2 institutions from Belarus, 1 from Lithuania
• Identifying and addressing specific needs of new
  scientific communities such as nano-science and
  engineering sciences
• Establishing grid services for linguistic
  research, Baltic Sea environmental research, data
  mining tools for communication modeling and
  bioinformatics, the project will deploy an
  extended and reinforced e-Infrastructure to the
  scientific communities within the Baltic region

17
Outline

• PART I About Project
• A bit of history
• Basic information partners, objectives,
  activities
• Cracow contibutions
• HEP applications
• BaltiGrid impact on IFJ PAN
• Future
• PART II Tools development
• BAT
• OCM-G
• CANDLE

18
BAT Batch Analysis Tool

• Developed in BG SA1
• Site admin needs in-depth knowledge on site
  usage
• are VOs getting their contracted resources?
• are priorities between VO respected?
• are VOs utilizing resources or just blocking
  (wasting) them?
• are specific VO requirements respected? e.g. to
  run job within 30 minutes.
• Verification of a site resource-sharing policy is
  troublesome currently...
• BAT is a grid tool that collects data from sites
  to central database and allows querying
• User specifies time period ? coarse/fine grain
  view
• Various data views
• Site overview (all VOs summarized)
• Per VO view
• Job efficiency view (in VO scope)
• Summarized view (a kind of accounting)
• Shows (examples)
• jobs queued, jobs running,
• jobs arrived and sent to queue, jobs arrived and
  started immediately
• percentage VO/job efficiency
• CPU/walltime consumed

19
BAT Some views
20
BAT Plans

• Accounting
• CPU normalization
• User-level data user privacy issues
• Other batch systems
• Sun Grid Engine
• Scalability
• Data caching
• Smart algorithms for aggregation and
  summarization
• Interested in using or installing BAT?
  radecki_at_agh.edu.pl

21
in
Tomasz Szepieniec Tomasz Duszka Jakub Janczak
http//grid.cyfronet.pl/ocmg
22
Application Support Objectives

• By Application Support we mean
• Organize Application Expert Group to
• Organize and initialize communication between
  application experts and GRID experts and speed-up
  application adaptation
• Provide analysis of applications to show
  application developers the way to the GRID
• Provide useful tools and support developers with
  using
• Graphical user interface for grid application
  Migrating Desktop
• Performance analysis tools for grid application
  OCM-G and G-PM
• DISCLAIMER Application Support is NOT for
• organizing support for user like help desk, call
  center, etc. User support is parts SA1.
• developing grid-enabled extensions to
  applications. All alterations in applications
  should be done by application developers.
• ONLY THIS SUMMER!BG Summer School on Grid
  enabling applications for gLite
  environmentDetails soon at www.balticgrid.org

23
OCM-G G-PM On-line Monitoring

• OCM-G and G-PM
• OCM-G is a grid-enabled application monitoring
  system enables possibility of on-line monitoring
  and steering of distributed application
• Special support for performance analysis of MPI
  applications
• G-PM tool for performance analysis
• They enable possibility to study performance
  bottle-necks in grid applications
• newly added
• support for IA64
• Support for Globus 4
• Developed within CrossGrid and maintained now by
  IFJ PAN with cooperation with CYFRONET

24
Architecture and Request Distribution
Tool
SM
SM
LM
LM
LM
Stop
AP1
AP2
AP3
AP4
node1
node2
node3
site1
site2
25
Extensions in OCM-G

• Integration with gLite
• New options of installing OCM-G (not only
  RPM-based installation)
• Scripts for installing in Shared Workspace
• Quick installation with job
• gLite job ID available internally
• Process list could be obtain using it
• New services
• Listing remote directory
• Downloading files (supports parts of files)
• Uploading files
• Running shell command on remote nodes
• Monitoring of CPU usage, free memory, free disk
  space, open files, etc.
• Forking and managing other processes (including
  attaching to standard I/O)

26
Support of Other Platforms and Features

• Full support of Globus Toolkit 2.4 and 3 and 4
• MCI reactivated!
• Option to compile without Globus (pure sockets)
• Partial support of IA64
• MPI instrumentation based on PMPI
• Improved management of components life-cycle
• Local Monitor now can be safely disconnected and
  re-connected

27
Facing BG Application OCM-G Frameworks

• Two non-standard use cases of OCM-G
• A toolkit that enable launching application under
  control of OCM-G
• recompilation of application not required
• application watch-dog

28
GAMESS Framework

• GAMESS widely use computation chemistry
  application
• typically long running time lost of data
  possible due to failure on worker node or break
  queue limits
• Feature to restart computation basing output
  files
• Using the framework
• normal JDL as input
• automatic transformation of JDL and OCM-G
  environment start-up
• Automatic synchronization (downloading to UI) all
  output sandbox

29
DNLP Framework

• DNLP MPI-Prolog based application to natural
  language (Latvian) syntax analysis
• Interactive usage
• Single task is relatively quickly computed
• Framework enable
• Multi-site, dynamic, interactive farm of jobs
• OCM-G is used to distribute work between worker
  and collect results

30
OCM-G Java API

• Java package enabling access to OCM-G
• Using COG (Java Globus API) or pure sockets
• Multi-layer interface
• Layer 1 handling connections GSI/MCI and
  sending/receiving text-based OMIS messages
• Layer 2 stateless objects handling tokens and
  operates on them
• Layer 3 stateful objects representing OCM-G
  tokens

31
CANDLE Successor of G-PM

• Managing and visualization of data collected by
  OCM-G
• Using OCM-G Java API
• Extensions of G-PM functionality
• Separation of data and visualization
• Easy integration with web portals.
• Support of dynamic applications
• Better GUI
• Development in progress
• advanced prototype planned for August 2007.

32
Summary

• BalticGrid is progressing well
• Even such kind of project gives opportunity to
  develop useful tools
• BAT
• OCM-G
• CANDLE
• We hope to continue in BG4Science
• and develop more e.g. RAP