GridRM: Grid Resource Monitoring

1
GridRM Grid Resource Monitoring

• Mark Baker Garry Smith
• mark.baker, garry.smith_at_computer.org
• Distributed Systems Group,
• University of Portsmouth.

2
Contents

• Introduction
• Grid Monitoring in general
• What is monitored?
• Related work
• Criteria for Resource Monitoring
• Initial prototype (Globus MDS)
• Grid Monitoring Architecture (GMA)
• Grid Resource Monitoring with GMA and SNMP
• GridRM Architecture
• LocalRM layer
• GlobalRM layer
• Summary
• Ongoing work

3
Grid Resource Monitoring (1)

• In order to construct and execute e-Science
  applications, an understanding of the type and
  availability of Grid resources is necessary.
• Grid Information Services can provide information
  about the health of Grid resources.
• These services are used to discover and describe
  Grid resources.
• Lack of knowledge about resources will hamper
  resource scheduling, allocation and usage.
• Important that Grid information service failures
  are reported and dealt with in a timely fashion.

4
Grid Resource Monitoring (2)

• Many efforts
• Heartbeat Monitor (Globus), NetLogger (LBNL),
  Network Weather Service (Univ. Calif), AutoPilot
  (Pablo Toolkit, Univ. Illinois), Remos (CMU),
  JAMM (LBNL)
• Our Focus
• Grid resources NOT applications, ie
• Compute resources (servers, clusters)
• Database resources (relational, OO)
• Specialist equipment (radio telescope, sensors)
• Network resources (inter-site comms links,
  network devices)

5
Criteria for Grid Resource Monitoring

• Seamless and transparent user access to
  information,
• No single point of failure,
• Easy to use Web-based interface,
• Collation of information from multiple sources,
• Combination of resource data and
  administration/project information, via a single
  user interface,
• Efficient (scalable) monitoring of current and
  recent status across Grid sites,
• Shield user from implementation details multiple
  reporting mechanisms.

6
Initial Prototype (1)

• Client-pull, communications through firewall.
• Polled data recorded in database for historical
  analysis and replay.

7
Initial Prototype (2)
Site Support Information
MDS Failures
Compute Resources
Site Status
Available MDS
Globus MDS Version
8
Initial Prototype

• Limited Gateway architecture (servlet
  interactions)
• No network link monitoring between sites,
• Queries are made to Globus information servers
  and not resources directly,
• Globus information servers require customisation
  for non-standard resource monitoring (i.e.
  Beowulfs)
• No Security

9
Grid Monitoring Architecture (GMA) (1)

• Defined by Global Grid Forum, GMA Working Group,
  
• A producer/consumer-based architecture, driven by
  a standard and extensible set of events,
• Allows Grid monitoring tools to interoperate,
• Directory services are needed to support
  information publication and discovery between GMA
  consumers and providers,
• GMA protocols support streaming
  publish/subscribe and query/response models,
• GMA security Public key based X.509
  certificates and SSL connections.

10
Grid Monitoring Architecture (GMA) (3)

• A central repository service (which may
  physically be distributed) is used to bind GMA
  system components together.

11
GridRM

• Extends initial prototype, based on GMA and SNMP.
• Goals
• Security each site responsible for local
  security,
• Redundancy each site provides access to local
  resources,
• Transparency site independent view,
• Scalability monitor multiple sites (using GMA),
• Standard and open source components GMA, Java,
  Servlets, Applets, SQL,
• No installation of additional software on
  monitored resources (SNMP).

12
GridRM Architecture (1)
Two hierarchical layers Local and Global
13
GridRM Architecture (2)
GridRM Local Layer
14
GridRM Architecture (3)
GridRM Gateway
15
GridRM Architecture (4)
GridRM Global Layer
16
Some Issues

• Distributed user session management across
  Gateways required to ensure consistent user view,
  
• Distribution and replication of GMA directory
  services required (SPoF, scalability),
• Remote Gateway location mechanisms,
• Monitoring network connections between Grid sites
  (could use NWS, ICMP).

17
Summary (1)

• GridRM open source, generic, distributed,
  resource-monitoring system for Grid environments.
  
• Composed of a local layer and a global layer
• Local layer operates within an individual Grid
  site GridRM Gateway monitors local site
  resources (SNMP).
• Global layer permits GridRM Gateways to exchange
  monitoring data between Grid sites in a scalable
  and secure manner (GMA),
• Gateways collaborate (at the global GridRM layer)
  to provide a consistent view of resource
  information.

18
Summary (2)

• Underlying complexity hidden behind Web-based
  GUI,
• Users can connect to any GridRM Gateway within
  virtual organisation and see a consistent view,
• On a failure, users can select a different
  Gateway to obtain resource information,
• Subscription based events data passed between
  Gateways only when needed event consumers can go
  off-line unexpectedly and all associated event
  propagation is terminated.

19
GridRM Status

• Current
• Prototype 1, available and working,
• Local layer started (SNMP)
• Limited prototype now available
• Global layer started (GMA)
• Early demonstration August 2002
• Future
• Integration of Local and Global layers (Fall
  2002)
• URL homer.csm.port.ac.uk/projects/research/grid/g
  rid-monitoring/