P. Kacsuk, G. Sipos, A. Toth, Z. Farkas, G. Kecskemeti and G. Hermann MTA SZTAKI www.lpds.sztaki.hu/pgportal [e-mail]

1
P. Kacsuk, G. Sipos, A. Toth, Z. Farkas, G.
Kecskemeti and G. Hermann MTA
SZTAKIwww.lpds.sztaki.hu/pgportalpgportal_at_lpds.s
ztaki.hu
Defining and Running Parametric Study Workflow
Applications by the P-GRADE Portal
2
Outline of the talk

1. Introduction to P-GRADE portal
2. PS-extension motivations
3. Principles of the PS-portal
4. Usage of the PS-portal
5. Distributing workflows among different Grid
   resources
6. Conclusions

3
P-GRADE portal in a nutshell
4
References

• P-GRADE Portal service is available for
• Central European VO of EGEE
• GILDA Training VO of EGEE
• HUNGRID VO of EGEE
• SEE-GRID infrastructure
• GIN VO
• National Grids (UK National Grid Service,
  HunGrid, etc.)
• Under preparation for
• US Open Science Grid, TeraGrid
• Economy-Grid, Swiss BioGrid, Bio and Biomed EGEE
  VOs, BioInfoGrid, BalticGrid
• Many national Grids

5
Users feedback

• Users want parameter study (PS) support at
  workflow level
• It means
• If the user has an existing workflow in a
  repository, he would like to run
• the same workflow (without any change)
• with many different parameters

6
Related research

1. PS at job level main subject is scheduling
2. PS at workflow level The workflow description
   supports the creation of PS runs
3. What our users want executing workflows stored
   in workflow repositories with many inputs as PS

7
Principles of the PS-portal
Storage elements
4 input files

1. Take the existing workflow as a black box
2. Modify only the external input ports to accept
   many input files
3. Create the cross-product of the input files and
   run the workflow with all combinations
4. Because of the large number of files both PS
   inputs and outputs are stored in the Grid!

3 input files
Storage elements
12 outputfiles
Storage elements
8
Generating cross products and executable
workflows (e-WFs)
4 x 3 normal workflow execution
1 PS workflow execution
PS port 4 instances of the input file
PS port 3 instances of the input file

9
Steps of creating a PS-workflow1. Start from a
tested Workflow
10
2. Define the PS Port(s)
Any external input port can be defined as PS-port
11
3.Define Subdirectory of Inputs
PS port is represented by a directory and not by
a single file
12
4. Define target Grid and subdirectory for the
results
13
5. PS Workflow ready to Submit
14
Progress of submissions in PS detailed view
Total Init Submitted Rescue Error
Finished
15
How to distribute workflows among different Grid
resources?
Goal Assign the 24 jobs to 24 different Grid
resources
1 PS workflow execution
PS port 4 instances of the input file
PS port 3 instances of the input file

16
Single-Grid, Static Assignment
Assigns the 24 jobs to 2 Grid Resources within 1
Grid
1 PS workflow execution
PS port 4 instances of the input file
A
A
A
A
B
B
B
B
PS port 3 instances of the input file

A
A
A
A
B
B
B
B
Assign resource A in Grid X
Assign resource B in Grid X
A
A
A
A
B
B
B
B
17
Single-Grid, Dynamic Assignment via Broker
Assigns the 24 jobs to 24 Grid Resources within
1 Grid
1 PS workflow execution
PS port 4 instances of the input file
C
D
A
B
E
G
H
F
PS port 3 instances of the input file

I
J
K
L
O
P
M
N
Assign to broker of Grid X
Assign to broker of Grid X
V
S
T
R
Y
Z
U
X
18
P-GRADE portal is a multi-Grid portal
TeraGrid (GT2/GT4)
P-GRADE-Portal
SEE-Grid (glite)
London
Rome
Athens
19
Static Multi-Grid Assignment
Assigns the 24 jobs to 24 Grid Resources within
2 Grids
1 PS workflow execution
PS port 4 instances of the input file
XC
XD
XA
XB
YE
YG
YH
YF
PS port 3 instances of the input file

XI
XJ
XK
XL
YO
YP
YM
YN
Assign to broker of Grid X
Assign to broker of Grid Y
XV
XS
XT
XR
YY
YZ
YU
YX
20
Dynamic Multi-Grid Assignment
Assigns the 24 jobs to 24 Grid Resources within
2 Grids
1 PS workflow execution
PS port 4 instances of the input file
YC
XD
XA
XB
YE
YG
XH
XF
PS port 3 instances of the input file

YI
XJ
XK
XL
YO
YP
YM
XN
Assign to meta-broker
Assign to meta-broker
YV
XS
XT
XR
YY
XZ
YU
YX
21
Conclusions

• Our solution
• Keeps both the workflow GUI and the parameter
  study support concept as simple as possible.
• Enables run any existing workflow with different
  parameter sets without modifying the structure of
  the workflow.
• Manages the execution of the workflows on as many
  Grid resources as possible.
• Enables the collection of Grid resources from
  several Grids even if they are based on different
  Grid technology.

22

Thanks for your attention
?
Further information www.lpds.sztaki.hu
23
Distributing workflows among different Grid
resources

1. Single-Grid, Static Assignment Statically
   assigning each node of the workflow to different
   resources in the same Grid (done)
2. Single-Grid, Dynamic Assignment Dynamically
   assigning each node of the workflow to different
   resources in the same Grid using the broker of
   the Grid (done)
3. Static multi-Grid Assignment Statically
   assigning different Grids to different nodes
   (done)
4. Dynamic multi-Grid Assignment Dynamically
   assigning the nodes to different Grids using a
   meta-broker (under investigation)

24
Multi-Grid portals with meta-broker
EGEE Grid
Provides the largest achievable parallelism for
PS workflows
Broker
UK NGS
Broker
Selects Grid for each node of the PS workflow