Une librairie de communication portable et adaptative pour rseaux hautdbit

1
(No Transcript)
2
Plan

• Introduction
• computation, clusters, high-performance
  communications
• Framework
• PM², RPC, communication interfaces
• Proposal
• Madeleine 2
• Implementation and evaluation
• results on top of TCP, VIA, SCI
• Conclusion

3
Introduction

• What do academic scientist and industrial
  engineers need
• complex computations on huge data
• high-performance computation machines clusters
  of workstations
• distributed parallel programming environments
  for clusters PM² distributed RPC-based
  multithreaded environment
• efficient communication support suited for such
  environments

4
RPC

• RPC
• Remote Procedure Call
• PM² basis paradigm
• A communication library for RPC should meet the
  following requirements
• portability
• efficiency
• zero-copy data transfer

5
Low Level Interfaces

• Characteristics
• direct network access
• high efficiency
• low portability
• Examples
• BIP
• Fast-Messages
• VIA
• SCI

6
High-Level Interfaces MPI

• MPI - Message Passing Interface
• widely used in parallel computation
  applications
• efficient portable implementations MPI-CH
• available on top of BIP and FM
• But
• interface is not flexible enough for efficient
  RPC support

7
Madeleine

• Characteristics
• communication library used by PM²
• portability
• efficiency
• dedicated to RPC paradigm
• But
• heavily message passing oriented
• suboptimal support of recent protocols SCI,
  VIA, ...
• limited interface

8
Madeleine II
9
User / Madeleine 2 Contract

• Interface of Madeleine 2
• semantical specification of data transfer
  options
• Consequences
• semantically adequate with user requirements
• more power to the library
• more liberty of movement
• more efficiency

10
Example
Example of a program using the Madeleine 2
communication library Sending side

Receiving side . . .
. . . len strlen("Madeleine
2") conn
begin_packing(chan, remote) conn
begin_unpacking(chan) pack(conn, len,
sizeof(int), unpack(conn,
len, sizeof(int), send_CHEAPER,
receive_EXPRESS)
send_CHEAPER, receive_EXPRESS)
str malloc(len
1) pack(conn, "Madeleine 2", len 1,
unpack(conn, str, len 1,
send_CHEAPER, receive_CHEAPER)
send_CHEAPER, receive_CHEAPER) end_packing(con
n)
end_unpacking(conn) . . .
. . .
11
Communication Channels

• Multi-channel support
• separate independant communication channels
• important for easy modular application
  development
• Multi-protocol support
• one protocol per channel
• optimal use of available resources

12
Adaptative Mechanisms

• Objective
• support of protocols with multiple paradigms
• dynamic paradigm selection for each data
  block
• Solution
• use of a choice function criteria
• size
• pack/unpack mode
• anything else if needed

13
Multi-Paradigm Structure

• Connection
• contains one or more links
• Link
• symbolizes a given data transfer method
• allows full use of protocol dependant
  characteristics and optimizations
• data ordering transparently enforced by Mad2s
  high level generic layer
• increased efficiency

14
Buffer Management

• Two kinds of buffers
• dynamic buffers
• static buffers
• Responsabilities of the pack function
• allocation of dynamic buffers
• static buffer filling
• buffers steering to adequate links
• buffer group management for burst data
  transfer

15
(No Transcript)
16
Implementation

• Features of the implementation of Madeleine 2
• drivers available on top of
• VIA message passing RDMA write
• SCI remote write, DMA support planned
• TCP
• SBP
• MPI and BIP support to be ready soon
• 100 operational environment

17
Performance on top of TCP
The following tests were realized using 2 PC
Pentium II / 450MHz interconnected by a
Fast-Ethernet network
18
Performance on top of VIA
The following tests were realized using 2 PC
Pentium II / 450MHz interconnected by a
Fast-Ethernet network
19
Performance on top of SCI
The following tests were realized using 2 PC
Pentium II / 450MHz interconnected by a SCI
network
20
Dynamic Paradigm Selection
21
Conclusion

• Madeleine 2
• more functionalities multi-channel,
  multi-protocol, multi-paradigm
• increased overall efficiency
• full operational system
• support for SCI, VIA, TCP, SBP
• first high-performance adaptative communication
  library

22
Future Work
23
Multi-Cluster Support

• Objectives
• network-heterogeneous multi-cluster support
• metacomputing capabilities
• Features
• inter-cluster channel
• user-friendly configuration scheme
• script-based applications and clusters
  description
• one-step multi-cluster application
  compilation and launch
• support for protocol specific loaders

24
Example
Example of cluster and application description
files for Madeleine 2 Cluster description
Application description cluster
popc application gauss
hosts popc0--11 executable mad_gauss
domain popc-private path
"/pm2/mad_applis/" gateways domain
ens-lyon host popc0 clusters adapters
name popc hosts popc0--3,
alias TCP0 protocol tcp,
name sci hosts sci0--4 alias
TCP1 protocol tcp suffix "-t", channels
alias BIP protocol bip
alias Control adapter
sci.TCP0, popc.TCP0,
alias Calc
adapter sci.SCI,
popc.BIP
25

• Clusters of cluster
• dynamic protocol selection
• multi-protocol routing
• architecture and network heterogeneity support
• Computation/communication scheduling
• optimized communication management
• multiple reception policies support