The Madeleine Communication Library

1
The MadeleineCommunicationLibrary

• Olivier Aumage
• LIP, ENS Lyon

Amsterdam March 2002
2
Introduction

• Madeleine

3
Madeleine, PM2
Application
Interface
PM2
DSM-PM2
Iso-malloc
Marcel
Madeleine
Net-Toolbox
Toolbox
4
Madeleine, PM2
Application
Interface
PM2
DSM-PM2

• RPC management

Iso-malloc
Marcel
Madeleine
Net-Toolbox
Toolbox
5
Madeleine, PM2
Application
Interface
PM2
DSM-PM2

• RPC management

Iso-malloc
Marcel
Madeleine
Net-Toolbox
Toolbox

• Communication management
• Message-passing paradigm
• Session management
• Generic communication interface

6
Madeleine, PM2
Application
Interface
PM2
DSM-PM2

• RPC management

Iso-malloc
Marcel
Madeleine
Net-Toolbox
Toolbox

• Thread management
• Thread migration
• SMP support
• Scheduler activations
• Synchronization
• Event polling

• Communication management
• Message-passing paradigm
• Session management
• Generic communication interface

7
Madeleine, PM2
Application
Interface
PM2
DSM-PM2

• RPC management

Iso-malloc
Marcel
Madeleine
Net-Toolbox
Toolbox

• Lists
• Hash
• Dyn. arrays
• Arguments
• String
• Fast mem. alloc.
• Macros

• Thread management
• Thread migration
• SMP support
• Scheduler activations
• Synchronization
• Event polling

• Communication management
• Message-passing paradigm
• Session management
• Generic communication interface

• Auxiliary communication
• TCP support

8
Madeleine, PM2
Application
Interface
PM2
DSM-PM2

• Distributed-sharedMemory

• RPC management

Iso-malloc

• Iso-address memory allocator

Marcel
Madeleine
Net-Toolbox
Toolbox

• Lists
• Hash
• Dyn. arrays
• Arguments
• String
• Fast mem. alloc.
• Macros

• Thread management
• Thread migration
• SMP support
• Scheduler activations
• Synchronization
• Event polling

• Communication management
• Message-passing paradigm
• Session management
• Generic communication interface

• Auxiliary communication
• TCP support

9
Madeleine, PM2
Application
Interface
PM2
DSM-PM2

• Distributed-sharedMemory

• RPC management

Iso-malloc

• Iso-address memory allocator

Marcel
Madeleine
Net-Toolbox
Toolbox

• Lists
• Hash
• Dyn. arrays
• Arguments
• String
• Fast mem. alloc.
• Macros

• Thread management
• Thread migration
• SMP support
• Scheduler activations
• Synchronization
• Event polling

• Communication management
• Message-passing paradigm
• Session management
• Generic communication interface

• Auxiliary communication
• TCP support

10
Madeleine

• Generic communication interface
• Network support
• Session management
• Efficiency
• Portability
• Functionalities
• Simplicity

11
Objectives

• Required characteristics

12
Objectives

• PM2 multithread environment
• High performance clusters
• Support for RPC-like communications
• Efficiency
• Reactivity

Node 1
Network
Node 2
13
Objectives - networks

• Adaptivity
• Multi-paradigm network interfaces
• VIA message passing, remote DMA
• SCI shared memory, DMA
• Static-buffer based network interfaces
• SBP
• Multi-mode network interfaces
• BIP short/long messages
• Exhaustivity
• Multi-protocol support
• Multi-adapter support

14
Objectives multi-clusters

• Multi-cluster exploitation
• Fast intra-cluster links
• Fast inter-cluster links
• Nework-level heterogeneity

15
Madeleine

• Interface

16
Interface
17
Packing - Unpacking

• Commands
• Mad_pack(cnx, buffer, len, pack_mode,
  unpack_mode)
• Mad_unpack(cnx, buffer, len, pack_mode,
  unpack_mode)
• Modes

18
Send
Send_SAFER
Send_LATER
Send_CHEAPER
Pack
19
Send
Send_SAFER
Send_LATER
Send_CHEAPER
Pack
Modification
20
Send
Send_SAFER
Send_LATER
Send_CHEAPER
Pack
Modification
?
End_packing
Transmitted version
21
Receive
Receive_EXPRESS
Receive_CHEAPER
Unpack
22
Receive
Receive_EXPRESS
Receive_CHEAPER
Unpack
Après Unpack
Data is available
Data availability???
23
Receive
Receive_EXPRESS
Receive_CHEAPER
Unpack
Après Unpack
Data is available
Data availability???
End_packing
Data is available
24
Example
Send
Receive
int n
int n
char s NULL
char s "Hello, World !"
p_mad_connection_t cnx
p_mad_connection_t cnx
25
Example
Send
Receive
int n
int n
char s NULL
char s "Hello, World !"
p_mad_connection_t cnx
p_mad_connection_t cnx
26
Example
Send
Receive
int n
int n
char s NULL
char s "Hello, World !"
p_mad_connection_t cnx
p_mad_connection_t cnx
cnx mad_begin_unpacking(channel)
cnx mad_begin_packing(channel, dest)
27
Example
Send
Receive
int n
int n
char s NULL
char s "Hello, World !"
p_mad_connection_t cnx
p_mad_connection_t cnx
cnx mad_begin_unpacking(channel)
cnx mad_begin_packing(channel, dest)
n strlen(s) 1
mad_unpack(cnx, n, sizeof(int),
mad_pack(cnx, n, sizeof(int),
send_CHEAPER,receive_EXPRESS)
send_CHEAPER, receive_EXPRESS)
28
Example
Send
Receive
int n
int n
char s NULL
char s "Hello, World !"
p_mad_connection_t cnx
p_mad_connection_t cnx
cnx mad_begin_unpacking(channel)
cnx mad_begin_packing(channel, dest)
n strlen(s) 1
mad_unpack(cnx, n, sizeof(int),
mad_pack(cnx, n, sizeof(int),
send_CHEAPER,receive_EXPRESS)
send_CHEAPER, receive_EXPRESS)
s malloc(n)
29
Example
Send
Receive
int n
int n
char s NULL
char s "Hello, World !"
p_mad_connection_t cnx
p_mad_connection_t cnx
cnx mad_begin_unpacking(channel)
cnx mad_begin_packing(channel, dest)
n strlen(s) 1
mad_unpack(cnx, n, sizeof(int),
mad_pack(cnx, n, sizeof(int),
send_CHEAPER,receive_EXPRESS)
send_CHEAPER, receive_EXPRESS)
s malloc(n)
mad_unpack(cnx, s, n,
mad_pack(cnx, s, n,
send_CHEAPER,receive_CHEAPER)
send_CHEAPER, receive_CHEAPER)
30
Example
Send
Receive
int n
int n
char s NULL
char s "Hello, World !"
p_mad_connection_t cnx
p_mad_connection_t cnx
cnx mad_begin_unpacking(channel)
cnx mad_begin_packing(channel, dest)
n strlen(s) 1
mad_unpack(cnx, n, sizeof(int),
mad_pack(cnx, n, sizeof(int),
send_CHEAPER,receive_EXPRESS)
send_CHEAPER, receive_EXPRESS)
s malloc(n)
mad_unpack(cnx, s, n,
mad_pack(cnx, s, n,
send_CHEAPER,receive_CHEAPER)
send_CHEAPER, receive_CHEAPER)
mad_end_unpacking(cnx)
mad_end_packing(cnx)
31
Madeleine

• Architecture

32
Architecture

• Modular approach
• Buffer Management Modules (BMM)
• Transmission Modules (TM)

Interface
BMM
BMM
Buffermanagement
TM
TM
TM
Network management
Network
33
Buffers

• Generic buffer management layer
• Virtual buffers
• Static
• Dynamic
• Buffer groups
• Aggregation
• Splitting

34
Networks

• Network management layer
• Data transfers
• Send, receive
• Aggrated transfers
• Transmission mode selection
• Selection function

35
Adaptivity
Interface
Pack
Buffer Management
?
Network Management
36
Madeleine

• Multi-cluster support

37
Multiplexing

• Channels
• Network
• Set of nodes
• Set of point to point connections

38
Real Channels

• One-to-one mapping to physical networks
• Partially cover the configuration

39
Virtual Channels

• Cover the whole configuration
• Built on top of real channels

Virtual
TCP
TCP
40
Functioning

• Automatic multi-network forwarding support
• MTU negociation
• Static routes
• Multi-threaded handling
• Generic approach

41
Bandwidth preservation

• A single copy
• Same buffer used for reception and forwarding

LANai

• Pipeline
• Simultaneous receive and send operations

42
Integration

• Generic forwarding transmission module
• Limitation of code traversal on gateways

Interface
BMM
BMM
Buffermanagement
Generic TM
TM
TM
TM
Networkmanagement
Network
43
Madeleine

• Polling support

44
Polling support

• Interaction with the Marcel thread scheduler
• Request aggregation
• Channel level aggregation
• Low level request
• Support is not reentrant
• Polling frequency
• Coarse-grained contrôl
• Timer, yields, idle
• No fine-grained network-specific polling frequency

45
Marcels polling support
Process
Network
Node
Process
LANai
Marcel
Thread
Process
46
Polling support To Do list

• Request handling
• Multi-level polling
• Better forwarding support on gateways
• Polling vs. interruptions
• Automatic switch mechanism between blocking and
  non-blocking network listening methods
• Polling frequency
• Request priorities
• Higher polling priority for efficient networks

47
Madeleine

• Session management

48
Session management

• Startup
• Modular approach
• Flexibility
• Scalability
• Two modules
• Madeleine
• Communication
• Léonie
• Session control

49
Léonie

• Sessions
• Multi-cluster configurations
• Unified process spawning
• Grouped process spawning
• Support for network-specific launchers (e.g.
  bipload)
• Support for optimized process launchers
• Network
• Internal tables building
• Information directory
• Virtual channel routing tables
• Scheduling
• NIC initialization, channel setup

50
Configuration Structure
Madeleine
Léonie
51
Madeleine

• Some results

52
Myrinet
53
Myrinet
54
SCI
55
SCI
56
Multi-cluster
57
Conclusion

• The Madeleine communication library
• Portability
• Linux, Solaris, Aix
• x86, Sparc, Alpha, PowerPC
• Heterogeneous cluster support
• Efficiency
• Multi-protocol support
• BIP, SISCI, VIA, SBP, MPI, TCP, UDP
• Dynamic transfer mode selection
• Multi-cluster support
• Automatic message forwarding on gateways

58
On-going and future work

• Various improvements
• Dynamicity
• Fault tolerance
• The GRID
• Madeleine and the GRID

59
Madeleine

• and the GRID

60
Madeleine Grid Component

• Multiprotocol communication device for
• Nexus
• MPICH
• Provide cluster-level communication support
• Generic
• Efficient

61
Structure
Nexus
Nexus/Madeleine module
Message Passing module
TCP module
Other modules
TCP protocol
MPL protocol
INX protocol
MAD SCI protocol
MAD TCP protocol
Madeleine
MPL Library
INX Library
Sockets
SCI
TCP
62
Latency
63
Bandwidth
64
MPICH/Madeleine
MPI API
Generic part (collective operations,
context/group management, ...)
ADI
Generic ADI code, datatype management, request
queues management
ProtocolInterface
CH_MAD device inter-node communication polling
loops eager protocol rendez-vous-protocol
SMP_PLUG device intra-node communication
CH_SELF device self communication
Madeleine heterogeneity management
TCP
SISCI
BIP
Fast-Ethernet
SCI
Myrinet
65
Latency
66
Bandwidth
67
GRID-RMI Project

• Environment for code coupling applications
• French Research Department supported project
• Involves many French research teams
• Madeleine used as the communication layer basement

68
Project Architecture
Simulation Code Coupling
C3D
Plants growing
ProActivePDC
OpenCCM
Do!
PaCO
GK
MPI
DSM Mome
Java VM
CORBA
PadicoTM
Madeleine
Marcel
69
(No Transcript)
70
(No Transcript)
71
Scheduler Activations

• Support for blocking syscalls
• and interrupts
• Vincent Danjean LIP, ENS Lyon

72
Interrupts et system

• Scheduler activation Anderson et al. 91
• Idea bidirectionnal cooperation between two
  schedulers (user and kernel level)
• User level scheduler syscalls
• Kernel level scheduler upcalls !
• Upcall
• Tell the application about kernel events
• Activations ( virtual processors)
• As many running activations as the number of
  processors
• Kernel controls creation/destruction of
  activations

73
Idea

• Blocking syscall / 2 processors

74
Idea

• Blocking syscall / 2 processors

75
Idea

• Blocking syscall / 2 processors

76
Idea

• Blocking syscall / 2 processors

77
Idea

• Blocking syscall / 2 processors

78
Idea

• Blocking syscall / 2 processors

79
Idea

• Blocking syscall / 2 processors

80
Idea

• Blocking syscall / 2 processors

81
Activation improvements

• Extension of Andersons model
• Kernel/Application independance
• Activation number not bounded
• Support for every blocking syscalls
• Better upcalls management
• new, block, preempt, unblock
• Optimization
• Pool of ready activations in the kernel

82
(No Transcript)
83
Aggregation

• Sequential

Flush
Flush
TM1
TM1
TM2
84
Aggregation

• Half-sequential

Main
TM 1
TM 2
85
Symmetry

• Asymmetrical mode

Flush
Flush
Flush
Send
Receive
86
Symmetry modes

• Asymmetrical mode

Flush
Flush
Flush
Send
Receive

• Symmetrical mode

Flush
Flush
Flush
Flush
87
Special cases

• Send_LATER / Receive_CHEAPER

Main
Pack
Pack
TM 1
TM 2
88
Special cases

• Send_LATER / Receive_EXPRESS
• Actual send is delayed until the call of
  End_packing

89
(No Transcript)