The PGRADE Portal

1
The P-GRADE Portal

• Peter Kacsuk
• MTA SZTAKI, Hungary
• Kacsuk_at_sztaki.hu

2
Acknowledgement

• This tutorial is based on the work of many
  people
• Gergely Sipos, Gabor Hermann, Gabor Dozsa, Csaba
  Nemeth, Tamas Boczko, Zoltan Farkas, Robert Lovas
  (SZTAKI developer team)
• Tamas Kiss, Thierry Delaitre, Ariel Goyeneche
  (Univ. of Westminster developer team)

3
Technology concerns of Grid systems

• Fast evolution of Grid systems and middleware
• GT1, GT2, OGSA, OGSI, GT3, WSRF, GT4,
• Many Grid systems are built based on these
  different technologies
• LCG-2, HCG, NorduGrid, JiniGrid, etc.
• These Grid systems should be interoperable

4
The inner structure of the Grid
The Grid consists of a large number of Grid
resources and services
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
The different colors represent different
technologies! (Globus, Jini, Web services, )
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
5
Dynamicity of the Grid
NorduGrid

GS
GS
GS

GS
GS
GS
LCG-2
GS
GS
GS
GS

GS
GS
GS
GS
HCG
These services are registered at the same
information system forming a Grid.

GT3
GS
GS
GS
GS
GS

GT3

GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
LCG-2
6
User concerns of Grid systems

• How to cope with the variety of these Grid
  systems?
• How to develop/create new Grid applications?
• How to execute Grid applications?
• How to observe the application execution in the
  Grid?
• How to tackle performance issues?
• How to port legacy applications
• to Grid systems
• between Grid systems?
• How to execute Grid applications over several
  Grids in a transparent way?

Portal technology could be an answer for the users
7
The vision of a dynamicGrid system with portal
The Grid
Grid service
Grid service
P-GRADE Grid Portal with workflow
All the services can be WS, Jini, Globus, etc.
services
Legacy code
GEMLCA
8
General properties of the P-GRADE Portal

• Portal for computational grids
• Supported services functionalities
• MyProxy proxy credential management
• GT2/GT3 GRAM job execution
• Mercury job/resource monitoring
• PROVE workflow/job execution visualization

9
Workflow support in the P-GRADE Portal

• Contains
• built-in workflow editor and
• workflow execution manager
• Components of the workflow can be
• Job to be executed as a Condor job in GT2/GT3
  Grids (like LCG-2, GridLab, etc.)
• GEMLCA service to be in a GT-3 Grid
• Job to be executed on a job-executor service in
  the JGrid (a Jini based Grid)

10
What is a workflow?

• The workflow is a graph where
• Nodes are jobs (or services)
• Arcs represent file transfer between the jobs
  (services)
• Semantics of the workflow
• Job can be executed if all the necessary file
  transfers represented by the arcs are completed

Jobs can be sequential
Jobs can be parallel
11
What is a workflow?

• Semantics of the workflow enables parallel
  execution of workflow nodes
• The workflow concept enables the efficient
  solution of complex problems in a distributed
  environment like Grid

12
Example Ultra-short range weather forecast
(Hungarian Meteorology Service)
Forecasting dangerous weather situations (storms,
fog, etc.), crucial task in the protection of
life and property
25 x
Processed information surface level
measurements, high-altitude measurements, radar,
satellite, lightning, results of previous
computed models
10 x
5 x
25 x

• Requirements
• Execution time lt 10 min
• High resolution (1km)

13
Why is it called P-GRADE Portal?

• P-GRADE is a Parallel Grid Run-time and
  Application Development Environment
• Containing workflow editor and execution support
• The P-GRADE workflow editor is compatible with
  the workflow editor of the portal

14
The layers of the P-GRADE Portal today
Client browser
P-GRADE Portal
GT2 GRAM
GT2 GRAM

Mercury PROVE
Mercury PROVE

MyProxy
MyProxy

The supported technologies
Internet
host
host
host
host
host
host
host
host
host
host

15
The layers of the P-GRADE Portal today
Client browser
Tomcat
GridSphere
Makes the low-level protocols invisible
Java 2 SDK API
Mercury Java API
CoG Java API
shell scripts
DAGMan
Condor-G
GT2/3 GRAM
GT2/3 GRAM

Mercury PROVE
Mercury PROVE

MyProxy
MyProxy

Internet
host
host
host
host
host
host
host
host
host
host

16
P-GRADE Portal ? Login
17
Principle of the P-GRADE portal
Certificate server
Remote Clusters to be controlled
CERTIFICATE (download)
WORKFLOW MANAGER (submit)
CERTIFICATE (upload)
EDITOR (saveupload)
Portal server
Workflow (result)
EDITOR (open)
WORKFLOW MANAGER (output)
18
Certificate management(old)
19
Certificate management(new)
20
Setting the Grid (old)
Add new site
21
Setting the Grid New(1)
22
Setting the Grid (New 2)
Add new site
23
User concerns of Grid systems

• How to cope with the variety of these Grid
  systems?
• How to develop/create new Grid applications?
• How to execute Grid applications?
• How to observe the application execution in the
  Grid?
• How to tackle performance issues?
• How to port legacy applications
• to Grid systems
• between Grid systems?
• How to execute Grid applications over several
  Grids in a transparent way?

24
Main interactions 2.
Certificate server
Remote Clusters to be controlled
WORKFLOW MANAGER (submit)
CERTIFICATE (download)
CERTIFICATE (upload)
EDITOR (save / upload)
Portal server
Workflow (result)
EDITOR (open)
WORKFLOW MANAGER (output)
25
Workflow editor (old)
26
Workflow editor (new)
27
(No Transcript)
28
Workflow editor Save
29
User concerns of Grid systems

• How to cope with the variety of these Grid
  systems?
• How to develop new Grid applications?
• How to execute Grid applications?
• How to observe the application execution in the
  Grid?
• How to tackle performance issues?
• How to port legacy applications
• to Grid systems
• between Grid systems?
• How to execute Grid applications over several
  Grids in a transparent way?

30
Main interactions 3.
Certificate server
Remote Clusters to be controlled
WORKFLOW MANAGER (submit)
CERTIFICATE (download)
CERTIFICATE (upload)
EDITOR (saveupload)
Portal server
Workflow (result)
EDITOR (open)
WORKFLOW MANAGER (output)
31
Submitting a Workflow
32
Workflow Manager detailed new view
33
Main interactions 4.
Certificate server
Remote Clusters to be controlled
WORKFLOW MANAGER (submit)
CERTIFICATE (download)
CERTIFICATE (upload)
EDITOR (saveupload)
Portal server
Workflow (result)
EDITOR (open)
WORKFLOW MANAGER (output)
34
Downloading the results
35
Downloading the results
36
User concerns of Grid systems

• How to cope with the variety of these Grid
  systems?
• How to develop new Grid applications?
• How to execute Grid applications?
• How to observe the application execution in the
  Grid?
• How to tackle performance issues?
• How to port legacy applications
• to Grid systems
• between Grid systems?
• How to execute Grid applications over several
  Grids in a transparent way?

37
Workflow execution visualization
38
Job execution visualization
39
User concerns of Grid systems

• How to cope with the variety of these Grid
  systems?
• How to develop new Grid applications?
• How to execute Grid applications?
• How to observe the application execution in the
  Grid?
• How to tackle performance issues?
• How to port legacy applications
• to Grid systems
• between Grid systems?
• How to execute Grid applications over several
  Grids in a transparent way?

40
GEMLCA Grid Execution Management for Legacy Code
Architecture
41
GEMLCA Objectives

• To deploy legacy code applications as Grid
  services without reengineering the original code
  and minimal user effort
• To create complex Grid workflows where components
  are legacy code applications
• To make these functions available from a Grid
  Portal

GEMLCA
GEMLCA P-GRADE Portal Integration
42
GEMLCA client sideP-GRADE portal
43
GEMLCA Demo Traffic Simulation Workflow
44
GEMLCA Demo Defining GEMLCA jobs
45
GEMLCA Demo Workflow execution
Traffic
46
Integration with GEMLCA
Client browser
P-GRADE Portal (Service-oriented portal)
Connect the portal tonew services that apply
different technologies
GT2 Job executor
Mercury PROVE
MyProxy certificate storage
GT3 Job executor
GEMLCA
Internet
host
host
host
host
host
host
host
host

47
Expandability of the portal

• The P-GRADE portal is based on the Gridsphere
  portal framework developed in the EU GridLab
  project

48
Visualisation applet
49
Results

• Any legacy code
• written in any language
• Either sequential or parallel code
• without any modification
• can be used as a GT-3 service
• You simply
• deploy GT-3 and GEMLCA on your server
• Install GEMLCA client on your client machine
• You can avoid of installing GEMLCA client if you
  use P-GRADE portal that is connected to GT-3 Grid

50
User concerns of Grid systems

• How to cope with the variety of these Grid
  systems?
• How to develop new Grid applications?
• How to execute Grid applications?
• How to observe the application execution in the
  Grid?
• How to tackle performance issues?
• How to port legacy applications
• to Grid systems
• between Grid systems?
• How to execute Grid applications over several
  Grids in a transparent way?

51
Todays realityTechnology specific Grid portals
NorduGrid

GS
GS
GS

GS
Grid portal GT-3
GS
GS
LCG-2
GS
GS
GS
GS

GS
GS
GS
Web browser
Grid portal HCG
GS
HCG

GT3
GS
GS
GS
GS
GS
Grid portal LCG-2

GT3

GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
LCG-2
52
Future Grid portals Functionality specific
portals

GS
GS
GS

GS
GS
GS
Medical Grid portal
GS
GS
GS
GS

GS
GS
GS
Web browser
GS
Computa-tional Grid portal

GS
GS
GS
GS
GS

Physics Grid portal

GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
53
Future Grid portals Functionality specific
portals

GS
GS
This is the P-GRADE Portal!
GS

GS
GS
GS
Medical Grid portal
GS
GS
GS
GS

GS
GS
GS
Web browser
GS
Computa-tional Grid portal

GS
GS
GS
GS
GS

Physics Grid portal

GS
GS
GS
GS
GS
GS
GS
GS
GS
GS
54
The problem of current portals

• They tightly connected and tailored to only one
  particular Grid
• If the user wants to move to another Grid she has
  to learn the new environment
• They do not support the simultaneous access of
  several Grids
• The new release of the P-GRADE portal aims at
  solving these problems

55
Current viewCo-existing Grids and Grid portals
Provides interfaces to grid services
Client browser
Client browser
Client browser
Client browser
Client browser
Client browser



Grid Portal
Grid Portal
Grid Portal
Make local service discovery possible
grid information system

GS
GS
grid information system
grid information system




GS
GS
GS
GS
GS
GS
grid-level functionalities
Internet
Communication layer
host
host
host
host
host
host
host
host
host
host

The provider devices
56
Future view
Make local service discovery possible
grid information system

GS
GS
grid information system
grid information system




GS
GS
GS
GS
GS
GS
grid-level functionalities
Internet
Communication layer
host
host
host
host
host
host
host
host
host
host

The provider devices
57
Dynamic and portable Grid access

GS
GS
GS

GS
GS
GS
WS
GS
GS
GS
GS

GS
GS
GS
GS

GT3
Jini
GS
GS
GS
GS
GS

GT3

GS
GS
GS
GS
GS
GS
GS
GS
GS
WS
GS
58
The P-GRADE Portal summary
59
P-GRADE portal for constructing complex Grid
applications

• Goal To provide
• a graphical environment for solving complex
  problems in the Grid by connecting existing
  sequential or parallel components into an
  abstract workflow

Sequential component
GRP component
MPI component
GEMLCA component
P-GRADE Workflow
60
P-GRADE portal for various Grids

• Goal To map and execute the abstract workflow
  into various Grid systems

P-GRADE Workflow
Condor Grids Hun ClusterGrid
GT-2 Grids NorduGrid GridLab testbed
OGSA Grids UK OGSA Test-bed
61
Applied Grid and cluster technologies
Condor
EU GridLab
GT-2 GT-3
Condor-G
DagMAN
GridSphere
GridFTP
GSI, GRAM
Mercury
SGE
P-GRADE Workflow
MPI
PBS
62
P-GRADE portal configuration

• Web browser

Portal server (P-GRADE portal GUI, workflow
engine, Mecury client)
Mercury server Local job manager, GT-2 or GT-3
server
Desktop 1
Web browser
Globus-2 Grid
Desktop N
63
Applied Grid infrastructure in details
Site 1
Portal server machine
Site 2
Portal server Mercury client
Condor DAGman
Condor-G
GT-2
GT-3
Condor
Sun Grid Engine
MPI
MPI
Mercury
Mercury
64
Status of the P-GRADE portal

• The P-GRADE Portal is already connected to the
  following Grids
• LCG-2
• Hungarian Supercomputing Grid
• EU GridLab testbed
• UK OGSA testbed
• New connections are under development
• Hungarian ClusterGrid
• JGrid
• NorduGrid

65
Status of the P-GRADE portal

• P-GRADE Portal version 1.0 is now AVAILABLE to
  download from
• www.lpds.sztaki.hu/pgportal/
• On-line demo version (at the URL above) can be
  used by anyone from anywhere

66
Plans for P-GRADE portal version 2.0

• Connection to Grid info systems
• Understanding the schema of various Grid info
  systems
• Presenting the data retrieved from the Grid info
  systems
• Connection to Grid brokers
• LCG-2 broker
• Other brokers
• Extension with data management support
• LCG-2 specific
• Generic
• Support for submitting different jobs of a
  workflow to different Grids
• Parameter Study

67
Information System
68
MDS like info sysytem
69
MDS Like info system detailed
70
LCG Like Info System
71
VO selection in LCG
72
Single VO in LCG selected
73
Single Site with all VO selected
74
(No Transcript)
75
New Feature Parameter Study
76
Input File
Principle of external parameter study
Output File
77
FIRST PART - EDITING
TRADITIONAL JOB OF WORKLOW
PARAMETER STUDY
PAR
ltTYPEgt
78
Port Definition Window
general case of Workflows
special parameter input
79
Substitution for Parameter key
FirstKey
NEW KEY DEFINITION WINDOW
Inherited name of key to define
Type of parameter to be substituted
Hitting Generate the values will be created
according the selected algorithm, and the result
set will be displayed in the output list showing
the order and the total number of elements as
well
Value definition
If Format is not Free special formatting can
be defined
Hitting OK returns to Port definition
Built in random generator will be used to define
the elements. The result is normalized to 1 and
can be mapped in any range in a linear way.
The values to be substituted by the Parameter
key can be defined by the algorithms selected
here
Simple enumeration of elements separated by the
user defined string
Simple enumeration of elements stored in the
defined file, separated by the user defined string
Values defined as a DO cycle, i.e with constant
increment
80
Job Properties Window
general, common with Workflow
extended with Parameter study
Resource is suggested to remain empty to indicate
dynamical allocation
Path of executable
Elements display the number of elements to be
generated
Number of processors
Resource
Monitoring is not a sound option incase of
parameter study
On OK the user defined data will be
serialized together with the generated
container Java objects and will be forwarded to
the server upon subsequent Save
By up and down the user can modify the order
of generation in the cartesian product
The User can influence the submission policy by
restricting the number of parallel jobs
81
Product family

• If you want to run legacy code (written in any
  language, either sequential or parallel code) as
  Grid services gt use GEMLCA
• www.cpc.wmin.ac.uk/ogsitestbed/GEMLCA/
• If you want to develop new parallel code for the
  Grid gt P-GRADE.
• www.lpds.sztaki.hu/pgrade/
• If you want to execute your GEMLCA or P-GRADE
  developed code in production Grid systems
  gt use P-GRADE Portal.
• www.lpds.sztaki.hu/pgportal/
• If you want to observe your workflow running in
  a Grid gt Mercury.
• www.lpds.sztaki.hu/mercury/

82
Final conclusions

• Portal technology like the P-GRADE portal helps
  the end-users in many ways
• Easy-to-use workflow concept for solving complex
  problems
• Execution visualization support to observe Grid
  execution
• Switching between Grid technologies will be
  transparent to the end-user
• Interoperability between different Grid systems
  can be solved
• Simultaneous use of different Grid systems can be
  solved
• Learn once, use everywhere
• And now enjoy the demo of the P-GRADE portal
  presented by Robert Lovas