Bridging WSRF services and the EGEE middleware

1
Bridging WSRF services and the EGEE middleware

• Pedro Andrade - CERN, Geneva

2
Agenda

• Digital Libraries and the DILIGENT Project - 45
• QA
• DILIGENT Process Management Service Example -
  10
• QA
• DILIGENT DIS and Keeper Service Example - 15
• QA
• Open Discussion

3
Outline

• Digital Library Community
• DILIGENT Project
• User Communities
• Architecture
• Technical Solution
• Conclusions

4
Outline

• Digital Library Community
• DILIGENT Project
• User Communities
• Architecture
• Technical Solution
• Conclusions

• Evolution
• DLs today
• Requirements

5
Evolution
knowledge management
2005 -
user producerconsumer
digital texts repository centralized catalogue
search service
1996
user consumer
many virtual organizations
few large institutions
6
DLs today

• The demand for DLs has recently grown
  considerably
• DLs are perceived as a necessary instrument to
  support communication and collaboration among the
  members of communities of interest
• The maintenance and management of a DL requires
  high investments and specialized personnel,
  multimedia handling requires high computational
  resources

7
DLs today

• Many of the organizations that demand the usage
  of Digital Libraries are small, distributed, and
  dynamic
• DLs are often used to support temporary
  activities
• Courses (training courses, technology transfer
  courses, etc)
• Exhibitions, Conferences
• Projects
• DL services are required by many application
  domains
• e-Health, e-Learning, e-Government, e-Science
• Corporate Knowledge Management

8
Requirements

• Live documents
• New information objects
• Pollution map
• Table summarizing data from millions of
  satellite measures
• Graph reporting an analytical trend of certain
  information extracted from a vast amount of data

9
Requirements

• The creation and handling of these documents
  require
• Access to many different, large, heterogeneous
  information sources
• Use of specialized services
• High computing power
• Small virtual dynamic organizations usually do
  not have the instruments for satisfying these
  requirements
• Controlled resources sharing is the only possible
  solution
• Information sources
• Services
• Computers and storage
• Technical staff

10
Outline

• Digital Library Community
• DILIGENT Project
• User Communities
• Architecture
• Technical Solution
• Conclusions

• Objective
• DILIGENT impact
• Project numbers
• Partners
• Activities

11
Objective

• Develop a Digital Library test-bed
  infrastructure that allows members of dynamic
  virtual research organizations to create
  on-demand transient digital libraries based on
  shared computing, storage, multimedia, multi-type
  content and application resources

12
DILIGENT Impact

• The DILIGENT service infrastructure will
  dramatically change the DL development model used
  by distributed and dynamic organisations and
  communities
• Using DILIGENT, these organisations and
  communities will be able to setup their own DL
• Even if they do not have enough financial and
  human resources
• When and for the time they need it
• Defining their own specific requirements
• Exploiting Grid-based services
• Profiting form an vast amount of storage and
  computational resources
• Sharing resources in a collaborative and
  efficient way

13
Project Numbers

• Duration 36 Months
• Start/end date September 2004 August 2007
• Effort 1024 p/m
• Cost 9.546.561
• EU funding 6.300.000

15
24
61
14
Partners

• Commercial Companies
• ENG - Engineering Ingegneria Informatica SpA
• FAST - Fast Search Transfer ASA
• 4D SOFT - 4D SOFT Software Development
• RAI - Italian National Broadcaster

• Universities/Research Institutes
• FhG/IPSI - Fraunhofer-Gesellschaft zur Förderung
  der angewandten Forschung e.V.
• SNS - Scuola Normale Superiore di Pisa
• UNIBAS - University of Basel
• UoA - University of Athens
• USG - University of Strathclyde
• International Organizations
• CERN - European Organization for Nuclear Research
• ESA - European Space Agency

• Administrative Coordinator
• ERCIM - European Research Consortium for
  Informatics and Mathematics
• Scientific Coordinator
• CNR-ISTI - Institute of Information Science and
  Technology of the Italian National Research
  Council

15
Activities
16
Outline

• Digital Library Community
• DILIGENT Project
• User Communities
• Architecture
• Technical Solution
• Conclusions

• DILIGENT user communities
• ImpECt user community
• Arte user community

17
DILIGENT User Communities
Collect requirements and validate the built
technology

• Earth Science Domain
• Represented by ESA
• Goal Improve accessibility, interoperability and
  usability of environmental data, models, tools
  and algorithms

• Cultural Heritage Domain
• Represented by SNS and RAI
• Goal Stimulate collaborative multidisciplinary
  scientific research

18
ImpECt User Community

• Data
• Automatically generated, annotated and relatively
  large
• Wide variety of content types (maps, satellite
  data, etc)
• Services
• Complex, taylored image processing
• Composed services
• ...
• Use cases
• Dynamic report generation
• Organization of conferences
• Handling of information in case of environmental
  accidents

19
ARTE User Community

• Data
• Retrodigitized, manually annotated and relatively
  small
• Image, video material (from RAI) and (structured)
  text
• Services
• Feature extraction for improving retrieval
• Composed services (processes management)
• ...
• Use cases
• Multidisciplinary research working on images and
  videos
• Courses, Workshop
• Exhibition Catalogue

20
Outline

• Digital Library Community
• DILIGENT Project
• User Communities
• Architecture
• Technical Solution
• Conclusions

• Design Process
• Overview
• Services description

21
Design Process
User Activities
Base Technology Analysis
Functional Specification
Testing Strategy and Plan
Service Specification
Experimentation
Architectural Specification
Service Design
22
Overview
23
DL Creation and Management
24
DL Creation and Management

• Objective provide the functionality necessary to
  aggregate and manage a collection of resources
  for efficient usage under the control of a
  Virtual Digital Library system

25
DL Creation and Management

• DILIGENT Information System (DIS)
• Aggregation and storage of information about
  DILIGENT resources
• Provide details information about the resources
  state through query and subscription interfaces
• Integration and exploitation of gLite information
  service
• Triggered actions and information caching
• Keeper
• Dynamic services deployment
• Distributed hosting node management
• Deployed services monitoring

26
DL Creation and Management

• VDL Generator (VDL)
• Users/communities support
• Dynamic composition and configuration of DLs
• Dynamic Virtual Organization Support (DVOS)
• Robust and flexible security framework
• Advanced authentication and authorization model
• Notification support
• Dynamic aggregation of resources and users/groups
• Broker Matchmaker (BMM)
• Optimal resources allocation

27
DL Creation and Management
28
Content and Metadata Management
29
Content and Metadata Management

• Objective provide access and storage of
  information objects, management of related
  metadata, encryption and decryption of
  information objects for security reasons, and
  information object annotations management.

30
Content and Metadata Management

• Content Management
• Distributed content storage support
• Advanced replication mechanism
• Flexible storage model
• Support for arbitrary object types
• content, metadata, features, collections,
  archives, indexes, etc
• Support for object relationships
• contains, characterizes, extracted-from,
  indexed-by, etc
• Integration of 3rd party data sources (archive
  migration)
• Metadata Management
• Efficient support for arbitrarily structured
  metadata and transparent support for multiple
  metadata schemas and mappings

31
Content and Metadata Management

• Content Security
• Deals with the specialties of protecting
  multimedia content
• Include authenticity, integrity and
  confidentiality
• Grid-enabled watermarking algorithms for
  different media types
• Annotation
• Provides interactive functionality for the
  management of manually authored, subjective, and
  context-dependent metadata items about DILIGENT
  information objects, primarily multimedia
  documents.

32
Content and Metadata Management
33
Content and Metadata Management
34
Index and Search Management
35
Index and Search Management

• Objective design and implement an open,
  scalable, highly-featured, distributed search
  engine enhanced by the usage of various types of
  indices.

36
Index and Search Management

• Index Service
• Build and consume inverted and forward indices on
  metadata for search performance enhancement
• Search Service
• Locate content by searching in metadata and
  extracted features
• Optimisation in search execution
• Support Full Text, Similarity Based and Exact
  Match searches
• Feature Extraction
• Process documents to extract feature descriptors

37
Index and Search Management

• Personalization Service
• Provide personalized information retrieval
• Manage user profiles
• Content Source Description and Selection
• Filter content sources to maximize search
  relevance/cost ratio
• Automatically generate source description
• Data Fusion
• Fuse (merge) the ranked content result sets into
  a single ranked result set

38
Index and Search Management
39
Process Management
40
Process Management

• Objectives
• Use existing services as building blocks for more
  complex, value-added workflows (processes/compound
  services)
• Dynamic scheduling of service requests
• Define structured processes to be executed on the
  Grid
• Efficient usage of the Grid resources available

41
Process Management

• Design and Verification
• Check whether a process is defined correctly
• Provide a graphical UI for specification and
  analysis of processes
• Execution and Reliability
• Distributed execution process on the Grid
• Avoid centralized workflow engine - avoid
  bottleneck
• Dynamic allocation of resources
• Automate process
• Optimisation
• Structural process modifications to maximise
  parallelism of execution
• Optimise Grid resources utilisation

42
Process Management
43
Application Specific Functionality
44
Application Specific Functionality

• Objective Integrate DILIGENT elements under an
  application-specific context by adding all the
  custom elements expected. It also aims to prove
  the concept of DILIGENT as a versatile and
  powerful open DLMS able to capture specific
  community needs.

45
Application Specific Functionality

• Portal
• Hosts the graphical user interface (UI) of the
  core services provided by the DILIGENT platform
• Every user interface element is a portlet or a
  part of a portlet
• The Portlet Hosting Platform
• support for all JSR168 portlet containers
• currently using uPortal and GridShpere
• optionally support of Web Services for Remote
  Portlets (WSRP)
• Visualization
• Provides application-specific visualization
  services for the DILIGENT user communities

46
Outline

• Digital Library Community
• DILIGENT Project
• User Communities
• Architecture
• Technical Solution
• Conclusions

• Key concepts
• EGEE in DILIGENT
• Globus in DILIGENT
• Status and Plans
• Examples

47
Key Concepts

• The DILIGENT design is service oriented in order
  to provide as many reusable components as
  possible for other e-applications that could be
  created on top of the basic DILIGENT
  infrastructure
• DILIGENT system is design and relies on the OGSA
  concepts of Grid Services and its related
  implementations

48
Key Concepts

• DILIGENT is going to be built by integrating
  Digital Library services on the infrastructure
  developed by the Enabling Grids for E-sciencE
  project (EGEE)
• DILIGENT enhances existing Grid services with the
  functionality needed to support the complex
  services interactions required to build, operate
  and maintain transient virtual digital libraries

49
Key Concepts

• gLite
• WMS
• SE
• LFC
• R-GMA
• VOMS
• GT4
• Java WS-Core
• WS-MDS
• WS AuthN AuthZ

User-Community Specific Applications
50
EGEE in DILIGENT

• DILIGENT has a WP dedicated to collaborate and
  provide feedback to the EGEE project
• Many kinds of interactions
• Participation in several technical meetings and
  tutorials/trainings
• Work with EGEE Technical Coordination Group
  workgroups
• Contribution in EGEE conferences, User Forum, etc
• Subscription to EGEE mailing lists
• Feedback was given about
• EGEE middleware and infrastructures (only PPS for
  now)
• DILIGENT requirements, status and plans

51
EGEE in DILIGENT

• DILIGENT adopts the EGEE middleware as the grid
  middleware solution
• The DILIGENT application will be composed by
• services provided by the DILIGENT project
• services provided by the gLite distribution
• Both DILIGENT and gLite services will be deployed
  on the same infrastructure
• DILIGENT exploits the gLite components
• DILIGENT services are designed to include or wrap
  pieces of gLite software

52
EGEE in DILIGENT
Data Management SE/LFC/FTS
Job Management WMS/CE/WN
Job Management WMS/CE/WN
Data Management SE/LFC
Job Management WMS/CE/WN
Security VOMS
Information System RGMA
53
EGEE in DILIGENT

• As soon as the DILIGENT middleware reaches a
  stable point, DILIGENT will
• join the EGEE production infrastructure

54
Globus in DILIGENT

• DILIGENT has adopted many OASIS specifications
• WSRF, WSN, WS-Addressing and WS-Security
• Most DILIGENT services are compliant with these
  specifications
• The following WSRF and WSN specifications are
  being used
• WS-ResourceProperties (WSRF-RP)
• WS-ResourceLifetime (WSRF-RL)
• WS-BaseFaults (WSRF-BF)
• WS-ServiceGroup (WSRF-SG)
• WS-BaseNotification (WSN)
• WS-Topics (WSN)

55
Globus in DILIGENT

• DILIGENT hosting nodes must be equipped with a
  service container capable to host and support
• Classical Web services - compliant with basic WS
  specifications
• WSRF Web services - compliant with WSRF
  specifications
• The following GT4 components are used
• Java WS Core (as hosting environment for the
  DILIGENT services)
• WS-MDS
• Credential Management
• WS AuthN and AuthZ

56
Globus in DILIGENT
Java WS-Core
Credential Management WS AuthN and AuthZ
WS-MDS
57
Examples

• Implementations examples will be given in the
  next presentations about
• DL Creation and Management
• DIS
• Keeper
• Process Management

58
Status and Plans
release candidate Final
release candidate Beta
release candidate Alfa
release Final
release Beta
release Alfa
development started
Functionalities
Jan 2006
Jul 2006
May 2006
Nov 2006
Jan 2007
Mar 2007
May 2007
Jul 2007
Nov 2005
Mar 2006
Sep 2006
Sep 2005
59
Outline

• Digital Library Community
• DILIGENT Project
• User Communities
• Architecture
• Technical Solution
• Conclusions

60
Conclusions

• DILIGENT services provide state-of-the-art,
  reusable, scalable, persistent and dynamic
  software components empowered by the
• Compliance with OGSA standards
• Adoption of gLite as the grid middleware
• Large-scale infrastructure provided by EGEE
• DILIGENT test-bed provides an highly efficient
  shared DL infrastructure for distributed
  communities which
• Allows on-demand shared usage of DL Resources
  (computing, storage, content, services)
• Supports multi-type content and application
• Is validated in two areas - Cultural Heritage and
  Earth Science

61
Thank you !!!

• Questions ??
• http//www.diligentproject.org
• pedro.andrade_at_cern.ch

62
Backup Slides
63
DILIGENT Infrastructures

• DILIGENT has 2 independent infrastructures up and
  running all gLite services (r1.4.1)
• Development infrastructure
• Testing infrastructure
• Infrastructures are geographically distributed,
  linking 6 sites in Athens, Budapest, Darmstadt,
  Pisa, Innsbruck and Rome

• Running gLite experimentation tests since July
  2005
• Data upload and job performance
• gLiteWrappers
• Storage Broker
• WMS Broker

64
Technical Solution cont.

• EGEE gLite Middleware
• Architecture and Planning
• Design of grid services
• Global security architecture
• OGSA and WSRF specifications Java WSCore
• WS-Resource (Stateful Web Service)
• WS-ResourceProperties (View of the Web Service
  State)
• WS-Addressing (WS-Resource access mechanisms)
• Aggregator Framework (WS-ResourceProperties
  Collector)