Grid Enabling MIMAS Services The GEMS Project

1
Grid Enabling MIMAS Services The GEMS Project

• Keith Cole
• Pascal Ekin
• MIMAS
• University of Manchester

2
Previous Grid-enabling Projects _at_ MIMAS

• SAMD
• Substantive research problem
• Automation of complex workflows
• Grid-enabled data (not OGSA-DAI)
• Parallelised analysis code for HPC system
• GUI developed
• ConvertGrid
• Substantive research problem
• Automation of complex workflow
• Grid-enabled data (OGSA-DAI)
• Grid-enabled application (Convert)
• Grid infrastructure provided by NGS (Oracle
  databases)

3
ConvertGrid Data Visualisation Interface
High average house price sales but low
participation rates
Low average house price sales but high
participation rates
Ten minutes from start to finish

• Relationship between average house price sales
  (Experian) and percentage of 16-19 year olds
  entering university (Neighbourhood Statistics
  Census aggregate statistics)

4
GEMS Grid Enabling MIMAS Services

• Establishing production data grids via the
  National Grid Service (NGS) to support the
  development of an e-infrastructure to support
  research.
• Already an established demand for grid enabled
  access to 2001 Census aggregate data (MOSES
  GeoVue NCeSS Nodes)
• Demonstrate how an existing data service
  infrastructure can be Grid enabled in a standards
  compliant and sustainable way.
• Maximise and build upon the ESRC/JISC investment
  in the establishment of existing data
  infrastructure
• Key tasks
• Connecting the MS SQLServer databases holding the
  2001 Census aggregate data directly to the Grid
  via the NGS
• Grid enabling the current data access system
  (Casweb)

5
GEMS Functionality

• Transform query result into a variety of formats
  (CSV, HTML, etc...) by employing built-in or user
  uploaded XSL Transform scripts
• Inject of table metadata into query results
• Upload query results to a Grid/FTP server
• View SQL generated by user interface for further
  integration into an OGSA-DAI client
• Redirect query results to an grid
  service/OGSA-DAI activity for further processing
• Bulk upload query results to a user
  specified OGSA-DAI enabled database

6
GEMS Architecture
7
GEMS Issues and Challenges

• Inclusion of metadata (Census table names) from
  JDBC sources other than MySQL
• Development of an OGSA-DAI activity which injects
  table-name metadata in SQL server XML query
  results
• Mapping UK e-Science certificates to current and
  future access management protocols
• Athens (current system)
• Shibboleth (to be deployed from September)
• Complexities inherent in Grid computing
• Complex, distributed workflows makes development
  and deployment of services challenging
• Relative immaturity of Grid middleware
• Managing user expectations
• Data Grids do not solve all research problems

8
GEMS II

• Grid-enabling satellite imagery for large scale
  CPU intensive data analysis (e.g. mosaicing or
  classification)
• Demonstrate seamless access between NGS data
  grids and compute grids
• Exploit new 128 CPU NGS data node (Feb 07)
• Start date Jan 07

9
GEMS II - OGC Web Services (OWS)

• OGC's next major advance in the development of
  open specifications that enable geospatial
  interoperability
• OWS involves a wide scope of activities, ranging
  from the acquisition of geospatial data from
  sensors to defining a "common infrastructure" for
  decision support
• OWS is rapidly becoming the de facto standard
  adopted both by the industry and the public sector

10
GEMS II OWS-4 Standards

• Sensor Observation Service (SOS)
• Sensor Planning Service (SPS)
• Sensor Alert Service (SAS)
• Web Processing Service (WPS)
• Catalogue Service (CSW)
• Web Coverage Service (WCS)
• Web Feature Service (WFS)
• Web Map Service (WMS)

GEMS II
11
GEMS II
Landsat 5/7
Registry
Client
Data Service
Grass Processing Service
12
GEMS II OGSA-DAI does not scale to production
services

• OGSA-DAI is not stable large datasets results
  coming from a relational database will crash
  OGSA-DAI, often taking down the instance of
  Tomcat (application server) hosting it
• OGSA-DAI is only partly secure though OGSA-DAI
  supports GSI, transport level security is highly
  inefficient to the point of being unusable for
  returning data results larger than a few
  Kilobytes
• OGSA-DAI workflow
• Chaining OGSA-DAI activities hosted on disparate
  server instances is almost impossible due to the
  inefficiency of the communication channels
• Chaining OGSA-DAI activities hosted on a single
  instance of OGSA-DAI can scale to a few Megabytes
  of date transfer processing of larger datasets
  by the activities is so slow the FTP server
  expecting final results will timeout