OGSA-DAI Usage Scenarios and Behaviour: Determining good practice

1
OGSA-DAI Usage Scenarios and Behaviour
Determining good practice

• Mario Antonioletti
• mario_at_epcc.ed.ac.uk
• EPCC, University of Edinburgh
• http//www.ogsadai.org.uk
• AHM 2004, Data Services and Middleware, 2nd
  September 2004

2
OGSA-DAI

• OGSA-DAI is middleware
• Middleware should be invisible
• Provide additional functionality or hide
  complexity
• Allows uniform access to data resources
• data resources relational and XML databases,
  files,
• Provides an extensible framework
• You can extend functionality - fill any gaps
• We think it works well
• But need feedback
• Recount how OGSA-DAI is being used
• Some background first

3
Basic Operational Model
DAISGR
Client
4
Why OGSA-DAI?

• Why use OGSA-DAI over JDBC?
• Language independence at the client end
• Do not need to use Java
• Platform independence
• Do not have to worry about connection technology,
  drivers, etc
• Can handle XML resources
• Can embed additional functionality at the service
  end
• Transformations
• Third party delivery
• etc
• Avoiding unnecessary data movement
• Provision of Metadata is powerful
• Usefulness of the Registry for service discovery
• Dynamic service binding process

5
More Complex Behaviour
GDS
And there's a lot more that you can do
6
Usage Patterns
Retrieve
Update/Insert
Pipeline
Data Q - Query D - Delivery S - Status R -
Result U - Update I - Data id
7
Activities are the drivers

• Express a task to be performed by a GDS
• Three broad classes of activities
• Statement
• Transformations
• Delivery
• Extensible
• Easy to add new functionality
• Does not require modification to the service
  interface
• Extension operate within the OGSA-DAI framework
• Functionality
• Implemented at the service
• Work where the data is (do not require to move
  data back)

8
OGSA-DAI Deck
9
Building Applications

• Activities are grouped together
• Perform document
• Data can flow between activities
• Optimisation
• Avoids multiple message exchanges
• Can deliver to other GDSs
• Prerequisite for data integration
• Base middleware for projects requiring data
  access
• Some capability for data integration
• That is the theory now for the practice
• OGSA-DAI being adopted by a number of projects

10
Who is Using OGSA-DAI?
N2Grid (http//www.cs.univie.ac.at/institute/index
.html?project-8080)
Bridges (http//www.brc.dcs.gla.ac.uk/projects/bri
dges/)
BioSimGrid (http//www.biosimgrid.org/)
INWA (http//www.epcc.ed.ac.uk/projects/inwa/)
BioGrid (http//www.biogrid.jp/)
AstroGrid (http//www.astrogrid.org/)
eDiaMoND (http//www.ediamond.ox.ac.uk/)
OGSA-DAI (http//www.ogsadai.org.uk)
GEON (http//www.geongrid.org/)
myGrid (http//www.mygrid.org.uk/)
MCS (http//www.isi.edu/deelman/MCS/)
ODD-Genes (http//www.epcc.ed.ac.uk/oddgenes/)
OGSA-WebDB (http//www.gtrc.aist.go.jp/dbgrid/)
GridMiner (http//www.gridminer.org/)
FirstDig (http//www.epcc.ed.ac.uk/firstdig/)
GeneGrid (http//www.qub.ac.uk/escience/projects.p
hpgenegrid)
IU RGRBench (http//www.cs.indiana.edu/plale/proj
ects/RGR/OGSA-DAI.html)
11
Project classification
12
Projects using OGSA-DAI

• These projects form a list of case studies
• Need to capture requirements
• How OGSA-DAI is being used
• Where it succeeds and where it fails
• Other issues that arise
• An on-going process
• Only time to outline salient points from a couple
  of projects
• More detail in the paper
• but this only gives a top level overview
• On-going process
• Solicit more
• If you have more then please get in touch

13

• e-Digital MammOgraphy National Database
• Built a prototype of a national database of
  mammographic images in support of the UK Breast
  screening programme
• Employ Grid technologies to facilitate this
  process
• Mike Brady gave a keynote that went over the
  details

14
(No Transcript)
15

• eDiaMoND Findings
• OGSA-DAI provides a flexible framework
• Dynamically configure the system through
  discovery
• Activities can operate with different levels of
  granularity
• Federation can introduced at various levels
• Upgrading from R3 ? R4 broke some things
• Low level XML issues
• Good documentation on how to extend the framework
• Extended Activities to access IBM DB2 Content
  Manager

16
INWA Objectives

• Innovation Node Western Australia
• Informing Business Regional Policy
  Grid-enabled fusion of global data and local
  knowledge
• Project
• Run from Nov 2003 - Aug 2004
• Involved 10 partners (6 UK 4 Australia)
• Aim
• Data mine commercially sensitive data
• Security an absolute MUST
• Employ Grid technologies
• Need access to data and computational resources
• Demonstrator using
• OGSA-DAI
• Incorporate data resources
• Sun DCG's TOG (Transfer-queue Over Globus)
• Handle job submission to analyse micro array data

17
INWA
18
INWA Lessons Learned

• Performing Data Integration
• TimeZone date problems
• Security issues
• Bugs in
• JavaCoG in GT3
• OGSA-DAI could not switch security for Grid data
  transfers
• TOG had no security option
• All of these have been fixed
• Middleware not mature enough for commercial
  deployment

19
ODD-Genes

• OGSA-DAI Demo for Genetics
• Collaboration between
• EPCC
• Scottish Centre for Genomic Technology and
  Informatics (GTI)
• Human Genetics Unit (HGU)
• ODD-Genes demonstrates
• Perform high-speed batch analysis of microarray
  data on the Grid
• Browse the results of previous analyses stored in
  a database
• View data from arbitrary databases as HTML
• Discover related databases on the Grid
• Perform coupled queries on newly-discovered
  databases to provide a richer analysis of gene
  data

20
ODD-Genes Actors
1. Client 2. EPCC is an example of a
computational resource. 3. HGU is an example
of a data repository.
21
ODD-Genes Findings

• Data discovery perceived to be very important
• Map data views time -gt spatial locations
• Discovery of new resources
• Transparency to data access
• _at_HGU had an XML database
• _at_GTI had a relational database
• Deploy OGSA-DAI and not worry about databases
• Issues
• Registry maintenance policy
• Semantics of the discovery process
• Groups working the same area but different
  schemas, no generic metadata (schemas were the
  effective metadata)
• Provides an additional tool for researchers

22
Other Projects

• AstroGrid
• Identified (and fixed) a number of bugs
• Passed on requirements
• FirstDig
• Identified a number of bugs
• Have contributed a data browser to OGSA-DAI
• GeneGrid
• Interfacing Perl through an OGSA-DAI service to
  access biological databases
• Requirement for file support
• EdSkyQuery-G
• Collaboration between OGSA-DAI Eldas
• Based on SkyQuery project by John Hopkins
  University, Baltimore, USA

23
More on Projects

• MSc at Edinburgh looking at data integration
  scenarios
• Benchmarking OGSA-DAI
• Investigating capabilities
• GDSActivity allowing perform documents to be
  executed at other GDSs
• Identifying further requirements for data
  integration - control flow
• sequence
• flow
• Question as to whether such capabilities should
  be included in OGSA-DAI or OGSA-DAI should
  interface with other workflow languages
• MSc at Edinburgh looking at C bindings to the
  OGSA-DAI CTK
• For language independance need to provide more of
  these
• Perl, Python Eiffel!!
• GridMiner
• Have a really cute logo
• Have a member of that team currently at NeSC

24
Conclusions

• Still early days
• Standardisation process not stabilising quickly
  enough
• Infrastructure still developing and prone to
  change
• OGSA-DAI acting as an enabler
• Showing people what can be done
• However is it cracking a nut with a sledge
  hammer?
• Usage patterns are similar
• Call for people to work together to solve similar
  problems
• Problems that are not OGSA-DAI specific
• Metadata, Time zones, security,
• Data discovery perceived to be important
• Is this in the scope of what OGSA-DAI should be
  doing?
• Need to talk to users and gather war stories
• http//www.ogsadai.org.uk/projects
• On going process

25
Further Information

• The OGSA-DAI Project Site
• http//www.ogsadai.org.uk
• The DAIS-WG site
• http//cs.man.ac.uk/grid-db
• OGSA-DAI Users Mailing list
• users_at_ogsadai.org.uk
• General discussion on grid DAI matters
• Formal support for OGSA-DAI releases
• http//www.ogsadai.org.uk/support
• support_at_ogsadai.org.uk
• OGSA-DAI training courses