The Ibis Project: Simplifying Grid Programming

1
The Ibis ProjectSimplifying Grid Programming
DeploymentHenri Bal, Jason Maassen, Rob van
Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel
Jacobs,Frank Seinstra, Roelof Kemp, Kees
VerstoepVrije Universiteit Amsterdam
2
The Promise of the Grid

• Efficient and transparent (i.e. easy-to-use)
  wall-socket computing over a distributed set of
  resources Sunderam ICCS2004, based on
  Foster/Kesselman

3
Reality Problems of the Grid

• Performance scalability
• Heterogeneous
• Low-level changingprogramming interfaces
• writing deploying grid applications is hard

• Connectivity issues
• Fault tolerance
• Malleability

!
Wide-Area Grid Systems
User
4
The Ibis Project

• Goal
• drastically simplify grid programming/deployment
• Grids as promised

5
Approach

• Minimal assumptions about execution environment
• Virtual Machines (Java) deal with heterogeneity
• Use middleware-independent APIs
• Different programming abstractions
• Designed for dynamic/hostile grid environments
• Modular and flexible can replace Ibis components
  by external ones

6
Global picture
7
Grid programming

• Programming models
• Message passing (RMI, MPJ)
• Divide-and-conquer (Satin)
• IPL (Ibis Portability Layer)
• Java-centric run-anywhere library, sent with
  application
• Can handle fault-tolerance, malleability
• SmartSockets
• Solve connectivity problems automatically
  (firewalls, NAT, addressing problems)

8
Grid deployment

• JavaGAT Java Grid Application Toolkit
• Make applications independent of underlying grid
• Used for file copying, resource discovery, job
  submission monitoring, user authentication
• API is currently standardized (SAGA)
• Zorilla P2P system
• Job management, gossiping,clustering, flood
  scheduling

9
Ibis applications

• e-Science (VL-e)
• Brain MEG-imaging
• Mass spectroscopy
• Multimedia content analysis
• Video processing
• Many HPC applications
• SAT-solver, N-body, grammar learning, .

10
European users

• D-Grid Workflow engine for astronomy
• U. Erlangen grid file system
• INRIA ProActive on Ibis RMI
• U. Patras Jylab scientific computing
• UPC Barcelona Grid Superscalar
• HITACHI Peta-scale data management

11
DAS-3
272 nodes(AMD Opterons) 792 cores 1TB
memory LAN Myrinet 10G Gigabit
Ethernet WAN (StarPlane) 20-40 Gb/s OPN
See CCGrid08 session 11 (Verstoep)
12
Gene sequence comparison in Satin (on DAS-3)
Speedup on 1 cluster
Run times on 5 clusters

• 78 efficiency on 5 clusters (with 1462
  WAN-msgs/sec)

13
Multimedia content analysis

• Analyzes video streams to recognize objects
• Extract feature vectors from images
• Describe properties (color, shape)
• Data-parallel task
• Compute on consecutive images
• Task-parallelism on a grid

See CCGrid08 SCALE challenge sessions 1216
14
High Resolution Video Processing

• Realtime processing of CineGrid movie data
• 3840x2160 (4xHD) _at_ 30 fps 1424 MB/sec
• Multi-cluster processing pipeline
• Using DAS-3, StarPlane and Ibis

15
Summary

• Goal Simplify grid programming/deployment
• Key ideas in Ibis
• Virtual machines (JVM) deal with heterogeneity
• High-level programming abstractions (Satin)
• Handle fault-tolerance, malleability,
  connectivity problems automatically (Satin,
  SmartSockets)
• Middleware-independent APIs (JavaGAT)
• Modular

16
Future work?

• Clouds as promised (CCGrid09 Shanghai)
• Exaflop as promised (CCGrid10 Melbourne)
• Cold fusion as promised
• Stock market prediction as promised
• Indiana Jones 5 as promised (2032?)

17
More information

• Ibis can be downloaded from
• http//www.cs.vu.nl/ibis
• Papers
• Satin PPoPP07, SmartSockets HPDC07,
  Gossiping HPDC07, JavaGAT SC07,MMCA IEEE
  Multimedia07, CCGrid08,DAS-3/StarPlane
  CCGrid08
• Ibis tutorials
• CCGrid08