RealityGrid: An Integrated Approach to Middleware through ICENI

1
RealityGrid An Integrated Approach to Middleware
through ICENI

• Prof John Darlington
• London e-Science Centre,
• Imperial College London, UK

2

• Positioning
• Grids transparent mapping of complex
  applications onto distributed machinery

Routinely for practising scientists c.f. heroic
HPC
3

• Requirements
• Simple application construction
• Automatic mapping to appropriate machines
• Automatic scheduling of activities
• Simple support for user interaction
• Simple support for collaboration
• Straightforward middleware installation and
  maintenance

4

• No half-way house need complete solution
• Requires automated use of considerable
  knowledge/intelligence previously provided
  manually
• Requires complete set of interoperable services
  for whole task
• Not necessarily monolithic single middleware
  solution (c.f. heroic middleware)

5

• For Reality Grid
• Efficient execution of LB3D code in Grid
  environment
• Integrated support for collaborative steering and
  visualisation

6
The application Pipeline

• Deployment
• Getting code and data to the resource
• Execution
• Running the code
• Steering
• Results Analysis
• Visualisation
• Either after completion or real time

7
ICENI Imperial College e-Science Network
Infrastructure
http//www.lesc.ic.ac.uk/iceni/
8
ICENI Strands
Usability
Deployment
ICENI
Service Oriented Architecture
Component Programming Model
Workflow Guided Scheduling
Role Based Access Security
Semantic Adaptation
9
Deep TrackTackle fundamental issues within Grids

• Focus on aspects relevant to RG scientists
• Running jobs selecting resources
• Staging and managing data
• Collaborative steering and visualisation
• Controlled sharing of resources, data knowledge
• Export solutions to other Grid activities
• Promote best practice through RG experience
• Lead develop relevant grid standards

10
Development Infrastructure

• Project Website
• mailing lists
• Daily build
• Regression tests
• On success binaries updated
• Regenerated JavaDoc
• Deployment tests
• CVS
• Code split across multiple repositories modules
• Documentation, manuals user guides
• ICENI Open Source License (Extended SISSL)

• Java builds on Solaris W2K
• Daily deployments on
• rhea (solaris_at_DoC)
• dirac (IRIX_at_UCL)

• Controlled open access to CVS source code
• Multiple repositories with
• defined release tags
• development branches

• Evolving 150 page manual
• Installation Configuration
• Deployment Usage
• Developer Contributors

11
Activity over the last year

• Use ICENI to launch LB3D
• Select which LB3D instance to use
• Consider machine availability basic performance
• Wrapping LB3D as a binary component
• Visualise steer LB3D through ICENI
• Integration of fast track file-based steering
• ICENI testbed
• Simplify deployment through Webstart
• Wizards to simplify configuration

12
Resource Discovery Initiation
13
Dynamic Discovery Composition
Deployed application
Application
Visualisation Server
14
Collaborative Visualisation Steering Integrated
with ICENI Driven Access Grid!
Service Oriented Architecture
Dataset B
Dataset A B
Dataset A
15
Installation Mechanism and Control Centre

• Client Requirements
• JRE 1.4.2
• Java Web Start (inc.)
• Internet Access

The ICENI Control Centre now has an installation
wizard that encapsulates configuration
execution for standard actions.
16
Wrapping up Legacy Code
Legacy code can quickly be made available to the
ICENI architecture
17
Story so far within Grids

• Can submit run jobs
• But dont necessarily know when they will run
• Collaborative visualisation steering
• Need to co-ordinate multiple resources
• Require predictability guaranteed execution

18
Effective Interaction within the Grid
Application
Execution
Fabric
19
Service Architecture
Reservation Service
Scheduler
Application Service
Reservation Engine
Performance Store
Launcher
20
Scheduling Framework
Scheduling Framework

• Listen out for services
• Launcher Services
• Reservation Services
• Performance Services

21
Performance Repository Framework
22
Performance Events

• Events fired whenever ICENI components start or
  ports are accessed
• Used to gather performance information about
  currently running application
• Events contain data relating to
• Time application
• Source component type, location resource
• Event type start or port
• Events are serialised objects
• Can be XML documents

23
Collection of Performance Results
Linear Equation Source
Linear Equation Solver
Time Event 1200
Linear Equation Source Start 1204
Send out Equations 1203
Linear Equation Solver Start 1205
Receive Equations 1212
..
Display Vector Results
24
Fully exploit meta-data to infer temporal view of
workflow
From the data flow and performance database infer
the temporal workflow and thereby which component
must be executed where and when. Reservations
need to obtained from the grid fabric.
25
Launching Service
Launching Framework
26
Launching Service

• Have a generic job submission system
• This is being developed into an independent Web
  Service (GridSAM)

27
Evolution of ICENI
28

• Summary
• Have demonstrated that transparent mapping is
  possible
• See the demo

29
Future work

• Expansion of ICENI testbed with performance
  driven scheduling reservations
• Use of simple WS to start jobs
• Prototype in advanced development
• Integration of service based steering

BASIC APPLICATION
30
Acknowledgements

• Director Professor John Darlington
• Research Staff
• Nathalie Furmento, Stephen McGough, William Lee
• Jeremy Cohen, Marko Krznaric, Murtaza Gulamali
• Asif Saleem, Laurie Young, Jeffrey Hau
• David McBride, Ali Afzal
• Support Staff
• Oliver Jevons, Sue Brookes, Glynn Cunin, Keith
  Sephton
• Alumni
• Steven Newhouse, Yong Xie, Gary Kong
• James Stanton, Anthony Mayer, Angela OBrien
• Contact
• http//www.lesc.ic.ac.uk/ ? e-mail
  lesc_at_ic.ac.uk