Monitoring and Information Services

1
Monitoring and Information Services

• Mark L. Green
• CCR, University at Buffalo
• Open Science Grid Consortium Meeting
• July 21, 2005

2
Monitoring and Information Services

• The Monitoring and Information Services Technical
  group is responsible for coordinating the OSG
  activities that relate to dynamic resource and
  information services within the OSG
  infrastructure. These services include
• Information services describing resources and
  other entities within the OSG.
• Dynamic resource monitoring to provide, for
  example, information for scheduling decisions of
  grid applications.
• Tools for troubleshooting services and end-to-end
  debugging of applications and services.
• Services for usage and statistics, and
  infrastructure for higher level services such as
  auditing/accounting.
• The chairs are Mark L. Green and Conrad Steenberg.

3
Monitoring and Information Services

• October - December 2004
• Discussion about MIS core infrastructure with
  other Technical Groups.
• Monitoring applications should present
  information about how they are getting
  information from the resources, what they provide
  as a query analysis interface and have a common
  core infrastructure to build around.
• Develop Schemas from all monitoring services so
  we can eliminate duplicates and start from a
  common subset (GLUE Schema, GRID3, etc.).
• Have our own MIS Schema and add key/value pairs
  as you please.
• The discovery service is a name service,
  providing urls and names of all services.
• UCSD Technical Meeting Preparation
• Plan to present the MIS core infrastructure
  architecture.
• Meet with TeraGrid INCA developers at meeting.

4
Monitoring and Information Services

• October - December 2004 contd
• Preliminary Monitoring and Information Services -
  Core Infrastructure (MIS-CI) jobmanager-mis
  arguments
• Profile
• custom user profiles
• All
• Jobs
• running
• queued
• historical
• GridFTP
• Diskspace
• Accounting
• Policy
• Environment
• Software
• VO
• Statistics
• Security
• Privileges based on authenticated user
  information.

5
Monitoring and Information Services

• October - December 2004 contd
• Investigate if Clarens could also be used for
  hierarchical repository.
• Looking into potential schema issues and how to
  move forward with MIS-CI schema.
• Test plan needed for OSG integration.
• MIS-CI system architecture presented at UCSD
  Technical Meeting.

6
MIS-CI System Architecture

• MIS Compute Element and Storage Element Discovery
  Service
• Lightweight information gathering
• Self monitoring
• Scalable
• Expandable
• Robust and Fault Tolerant
• Timestamps
• API

7
MIS-CI System Architecture

• MIS Compute Element and Storage Element Consumer
  Interface
• Modular and customizable query infrastructure
• Self monitoring
• Scalable
• Consumer profiles
• Consumer throttling
• API

8
MIS-CI System Architecture

• MIS Compute Element and Storage Element Resource
  Repository
• Lightweight with minimal resource loading and
  requirements
• Self monitoring
• Expandable
• Common schema with unique name space
• Virtual organization based aggregation
• Timestamps

9
MIS-CI System Architecture

• MIS Compute Element and Storage Element
  Information Gathering
• Lightweight information gathering
• Self monitoring
• Expandable
• Modular design
• Cron based implementation

10
MIS-CI System Architecture
11
MIS-CI Deployment Plans

• Proof of Concept MIS-CI Deployment
• Pacman Cache
• Grid3Dev
• Mama Cluster
• Grid3
• Acdc Cluster
• Document Management System
• Module management
• Version control
• MIS-CI API
• MIS-CI Online Documentation

12
MIS-CI Deployment Timeline

• Proof of Concept MIS-CI Deployment (GridCat
  Providers) 12/2004
• Pacman Cache
• Grid3Dev 1/2005
• Mama Cluster
• Grid3 2/2005
• ACDC Cluster
• Document Management System 12/2004
• Module management
• Version control
• MIS-CI API 1/2005
• MIS-CI Online Documentation 1/2005
• Proof of Concept Web Repository (ACDC) 1/2005
• Only use MIS-CI for Information Gathering
• Load/Usage Statistics and Scalability reporting

13
Monitoring and Information Services

• January - April 2005
• OSG Monitoring and Information Services
  Requirement solicitation sent out.
• The document management system is setup at url
  https//grid.ccr.buffalo.edu/docmgr_osg_mis/
• A readiness plan for the OSG integration activity
  needs to be developed.
• MIS-CI CVS repository setup at UF Mark, Craig,
  Bockjoo are the current read/writers.
• MIS-CI wrapped in with OSG-INT cache and Pacman 3
  compliant.

14
Monitoring and Information Services

• January - April 2005 contd
• VO specific instances of the discovery service
  are provided at http//discover.gridservice.info.
• Look at making a pacman cache for the GIP and
  creating an authenticated BDII.
• The discovery service can integrate information
  about the MIS-CI grid installations and serve as
  the top level MIS-CI discovery service.
• Currently testing MIS-CI on UBuffalo-CCR-ITB site
  MIS-CI 0.2.6 release candidate for GridCat.

15
Monitoring and Information Services

• January - April 2005 contd
• MDS GIP LDAP Integration Format
• MDS calls GIP and information is returned in LDIF
  format for integration into MDS.
• MIS-CI needs to return information in LDIF format
  and a call added into MDS information gathering
  scripts to incorporate MIS-CI information.
• Identify information deficiencies that the
  Resource Broker requires and resolve them.
• Accounting Activity and MIS-CI discussions are
  scheduled for the end of April.
• TeraGrid Science Gateways discussions start.

16
Monitoring and Information Services

• May - July 2005
• General discussion about OSG Monitoring and
  Information Services minimum requirements.
• MDS developers will helping GIP folks work on XML
  mapping of GLUE Schema 1.2. MDS has providers for
  ldif, XML, and GLUE Schema.
• We need a DBII to collect the gris information.
• Create services VO and Monitoring Services
  Integration (MIS VO)
• Move the burden of monitoring configuration,
  update, and functionality from the novice to the
  experts.
• Current monitoring services should join this VO
  (GridCat, ACDC, MonaLisa, MDS, etc.)
• MIS-CI will fill in the gaps in the GIPs for OSG.

17
Monitoring and Information Services

• May - July 2005 contd
• MIS Current Proposal
• GIP 1.1 ldif
• GLUE Schema 1.1
• MDS 2.4 gris
• BDII Index Service 3.25
• MIS Roadmap Proposal
• GIP 1.2 xml
• GLUE Schema 1.2
• MDS 4
• BDII Index Service 3.25 and/or MDS Index Service
• ACDC Grid Dashboard
• Functionality and tutorial discussion.
• Batch job, fork job, and GridFTP accounting
  online.

18
Monitoring and Information Services

• May - July 2005 contd
• Pre-release VDT cache available for testing of
  the GIP/MDS/GRIS system.
• Decided MIS-CI should form an activity and
  recruit volunteers for development work.
• POLICY_URL variable for resource utilization
  policy added.
• OSG Monitoring and Information Services Minimum
  Requirements Discussion
• The first OSG release will define the minimum
  monitoring and information services as
• The globus top level location is the same as the
  VDT location.
• Top level globus directory/monitoring/grid3-info.c
  onf.
• Forward map for certs to accounts.
• Reverse map for accounts to VOs.
• Interface requirement for minimum requirements is
  ACSII file.

19
Monitoring and Information Services

• May - July 2005 contd
• GRASE VO has determined that six applications are
  reporting correctly to the ACDC Grid Dashboards
  and are generating accounting information.
• Verification of gridftp and batch jobs fine,
  VO/user mapping being verified.
• Kit validation, from the VOs that wish to
  participate in the MIS-OPs service.
• Benefit is that CEs can be validated based on the
  actual VO mapped user, participation is
  completely voluntary.

20
MIS-CI Level of Effort

• MIS-CI Activity Formed
• Center for Computational Research SUNY Buffalo
  Stakeholder
• 1.5 FTE
• Indiana GOC
• 0.5 FTE
• Binghamton
• 1.0 FTE
• LCG/EGEE Grid Deployment
• 0.25 FTE (pending)
• Others
• Estimated requirement 2.0 FTE
• OSG GIP Configuration Recommendations Defined.
• TG may need to form sub groups to focus
  discussions on contributing into the Blueprint
  activity.

21
Monitoring and Information Services

• May - July 2005 contd
• The scope of GIP work is three-fold
• The first is to address how GIP can be integrated
  with other OSG monitoring and information
  services.
• The second is to ease the configuration of GIP
  components.
• The third is to support the interoperability
  between the LCG and OSG.
• Specific goals for this work are to
• Test GIP and provide input to VDT and GIP teams.
• Provide GIP configuration guidance and tools to
  support its interaction/integration with other
  OSG monitoring and information services.
• Define use cases for GIP and standard practices
• Facilitate GIP use in interoperability (LCG and
  OSG) work.

22
Monitoring and Information Services

• May - July 2005 contd
• Action items / effort time line
• 1) Test GIP, and document testing results (5,
  ongoing)
• 2) Evaluate GIP roles with respect to other OSG
  monitoring and information services (10, a few
  months)
• 3) Provide tools to make GIP configuration
  easier, more accurate and useful (10, several
  weeks)
• 4) Provide "typical" GIP use scenarios in which
  OSG resources are involved (5, a few weeks)
• 5) Investigate how GIP is applied to address the
  interoperability between the LCG and OSG (10,
  ongoing)
• 6) Customize GIP to use standards that conform to
  OSG fashion (for example, change LCG_2_4_0 to
  OSG_0_1_6) (5, several days)
• 7) Interact with Condor team about publishing
  Condor policy and dynamic queue status to GIP
  (10, few weeks)

23
Monitoring and Information Services

• May - July 2005 contd
• There is an existing interoperability activity
  where much of the discussion has been made. There
  should be two paths
• 1) interoperability with LCG and
• 2) provide the required information for MIS
  group. Glue schema 2 will potentially merge these
  paths.
• GOC should run a discovery service for the OSG,
  discussion what services will be discovered
  there?
• Try to stay away from the never ending MIS schema
  revision cycle.
• Started work to make GIP configuration easier
  with Perl based automatic configuration of the
  GIP underway.

24
MIS-TG Looking Forward

• Refine MIS-CI System Architecture
• Key components are
• MDS, BDII, R-GMA, GIP, SQL Compliant Database,
  WSRF, MIS Schema,
• hierarchical information transport mechanisms,
• present standardized interface for higher level
  services.
• Extend resource policy definition specification
  and discovery.
• Log file analysis and archival procedures.
• Enforcement and interpretation infrastructure and
  tools.
• Usage, accounting, statistics analysis, and
  metrics reporting infrastructure.
• Interoperate and interoperable with peer grids.
• Coordinated effort with other OSG Technical
  Groups.