GridBench: A Tool for Benchmarking Grid Resources

1
GridBench A Tool for Benchmarking Grid Resources

• Marios Dikaiakos
• Dept. Of Computer ScienceUniversity of Cyprus

2
Acknowledgments

• In collaboration with
• George Tsouloupas, Ph.D. student, Univ. of
  Cyprus.
• Thanks to
• Wei Xing, Ph.D. student, Univ. of Cyprus.
• Artemakis Artemiou, undergraduate student, Univ.
  of Cyprus.
• Dr. Pedro Trancoso, Univ. of Cyprus

3
Motivation

• Investigating the performance of Grid resources
  and computations is a key challenge for
• Design and configuration of emerging Grid
  infrastructures.
• Realization of open marketplaces based on
  performance negotiation.
• Developing models for performance prediction.
• GridBench
• A software tool for evaluating the performance of
  Grids and Grid resources at different levels.
• GridBench Suite
• A hierarchical suite of benchmarks deployed on a
  Grid testbed.
• Geared towards high-performance and
  high-throughput computing needs.

4
Context CrossGrid Test-bed
http//mapcenter.lip.pt
5
Context CrossGrid architecture
1.4 Meteo Pollution
1.3 Data Mining on Grid (NN)
1.3 Interactive Distributed Data Access
1.2 Flooding
1.1 BioMed
Applications
3.1 Portal Migrating Desktop
2.4 Performance Analysis
2.2 MPI Verification
2.3 GridBench
Supporting Tools
Applications Development Support
MPICH-G
1.1, 1.2 HLA and others
App. Spec Services
1.1 Grid Visualisation Kernel
1.3 Interactive Session Services
1.1 User Interaction Services
3.1 Roaming Access
3.2 Scheduling Agents
3.3 Grid Monitoring
3.4 Optimization of Grid Data Access
DataGrid Replica Manager
Globus Replica Manager
Generic Services
GRAM
GSI
Replica Catalog
GIS / MDS
GridFTP
Globus-IO
DataGrid Job Submission Service
Replica Catalog
Fabric
Resource Manager (CE)
Resource Manager
Resource Manager (SE)
Resource Manager
3.4 Optimization of Local Data Access
CPU
Secondary Storage
Instruments ( Satelites, Radars)
Tertiary Storage
6
Motivation
7
Why Grid benchmarking?

• End-users
• Shop around for resources and VOs (define
  expected QoS).
• Developers
• Investigate fitness of Grid for classes of
  applications.
• Compare job submission services, resource
  allocation policies, scheduling algorithms.
• Architects/Administrators
• Improve system design, detecting faults, compare
  implementations, systems, configurations.
• Researchers
• Derive insights into how Grids work and perform,
  understand the nature of grids in general

8
Difficulties with Grid benchmarking

• Grid architecture
• Heterogeneous, dynamic, non-exclusive use, extra
  layers
• Relevance
• Prevalent application and programming models
• Metrics
• Quantitative, qualitative, statistical, relative
  to machine configuration or to grid resource
  units spent?
• Job Submission and execution
• Definition, resource allocation, result
  collection, multiple domains
• Understanding results
• Proper context, interpretation of results

9
Grid benchmarking the challenge
10
Elements of our approach

• An abstract hierarchical model for the Grid
  architecture.
• A layered suite of benchmarks characterizing the
  performance of abstract-model elements HPC, HTC,
  MPI.
• A platform independent language (GBDL) for
  specifying the configuration and for representing
  the conditions and results of benchmarking
  experiments.
• GridBench a tool for administering Grid
  benchmarks, archiving, publishing and browsing
  results.

11
GridBench a hierarchical approach
Wide Area Network
VirtualOrganization
Central Services (VO, Resource Broker, etc.)
Site
Storage Element
Computing Element
Storage Element
Computing Element
Computing Element
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Site
Site
Worker Node

• Performance measurements at the different levels
  of the Grid architecture.

12
A hierarchical approach
Wide Area Network
VirtualOrganization
Central Services (VO, Resource Broker, etc.)
Individual Resources(cluster nodes, mass
storage)
Site
Storage Element
Computing Element
Storage Element
Computing Element
Computing Element
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Site
Site
Worker Node

• Performance measurements at the different levels
  of the Grid architecture.

13
A hierarchical approach
Wide Area Network
VirtualOrganization
Central Services (VO, Resource Broker, etc.)
Individual Resources(cluster nodes, mass
storage) Sites (clusters, SMPs)
Site
Storage Element
Computing Element
Storage Element
Computing Element
Computing Element
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Site
Site
Worker Node

• Performance measurements at the different levels
  of the Grid architecture.

14
A hierarchical approach
Wide Area Network
Central Services (VO, Resource Broker, etc.)
VirtualOrganization
Individual Resources(cluster nodes, mass
storage) Sites (clusters, SMPs) Middleware (midd
leware layer providing access to shared
resources)
Site
Storage Element
Computing Element
Storage Element
Computing Element
Computing Element
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Site
Site
Worker Node

• Performance measurements at the different levels
  of the Grid architecture.

15
A hierarchical approach
Wide Area Network
Central Services (VO, Resource Broker, etc.)
VirtualOrganization
Individual Resources(cluster nodes, mass
storage) Sites (clusters, SMPs) Middleware (midd
leware layer providing access to shared
resources) Grid Constellation(multiple sites,
VOs)
Site
Storage Element
Computing Element
Storage Element
Computing Element
Computing Element
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Worker Node
Site
Site
Worker Node

• Performance measurements at the different levels
  of the Grid architecture.

16
The GridBench suite current status
17
GridBench requirements functionality

• Supports
• Specification and execution benchmarks on a Grid.
• Collecting and archiving of results.
• Archives together
• Benchmark specifications and measurements for
  publication and further analysis.
• Monitoring information to help with result
  interpretation.
• Supports
• Retrieval and graphical representation of metrics.

18
GridBench components front-end

• GBDL Translator
• XML benchmark description (GBDL) to job
  description language
• Supports JDL (EDG, Condor) and RSL (Globus).
• Benchmark Definition UI
• GUI for defining and executing benchmarks.
• Benchmark Browser
• GUI for browsing and analyzing results.
• Information Provider
• Publishes results to Metacomputing Directory
  Service (MDS).

19
GridBench components back-end

• Orchestrator
• Manager of execution and result collection.
• Web Service (it must submit the job and wait for
  the output).
• Co-located with work-load management client of
  CrossGrid.
• Archiver Database
• Stores benchmark definition, results, and
  monitoring.
• Web Service.
• Requires network connectivity to the host running
  the apache database.
• Benchmark Components
• Benchmark executable code.
• Monitoring Component
• Collects information using e.g. R-GMA or OCM-G.

20
Software architecture at work
21
GridBench front-end
1-Pick a benchmark
2- Configure it
22
The Generated GBDL
23
The Browsing Interface
List of Benchmark Executions
Query
Metrics from Selected Executions
Metrics
24
Result management and presentation
Metrics from Selected Executions
(Can be used to compare Similar metrics)
Drag Drop
25
MPPTest experiments
26
MPPTest measurements
27
EPWhetsone experiments
28
EPWStream experiments
29
CacheBench experiments
30
MPPTest
31
Monitoring information
32
Summary and Remarks

• Benchmarking the Grid is different from
  traditional benchmarking.
• The GridBench approach
• Focuses on the characterization of Grid
  performance.
• Is based on a simple hierarchical model for the
  Grid.
• Incorporates a hierarchical suite of benchmarks
  of different complexity.
• Produces a large database of specs, metrics and
  monitoring info.
• GridBench attaches performance meta-data to a
  simple model of the Grid architecture.

33
Summary and Remarks

• GridBench software is a complete framework that
• Defines Benchmarks platform-independent GDBL
• Executes benchmarks layered suite of benchmarks
• Archives and Publishes results
• Graphical User Interface
• Porting Benchmarks to the Grid not as
  straight-forward as anticipated (heterogeneity of
  resources, configuration, libraries).
• Benchmarks are a great tool for detecting flaws
  in hardware, software and configuration.

34
For more information

• http//grid.ucy.ac.cy
• http//www.cs.ucy.ac.cy/Projects/crossgrid/
• http//www.crossgrid.org
• mdd_at_cs.ucy.ac.cy

35
EPWhetstone submitted to two Computing Elements
EPWhetstone measurements

• Different Colors represent different Worker Nodes
• Measures Whetstone MIPS

Three EPWhetstone submissions to
apelatis.grid.ucy.ac.cy
36
EPStream submitted to three Computing Elements
EPStream measurements

• Different colours represent different Worker
  Nodes
• Measures Memory Bandwidth in MB/s

Two EPStream submissions to cluster.ui.sav.sk