Portals

1
Portals
2
History

• Lightwieght kernel research for MPP (SUNMOS and
  Puma) -gt Portals 2.0
• Address need for and leverage NIC capability for
  zero-copy into app addr space without kernel
  buffering
• Data structures to specify how data should be
  sent/received, without application involvement,
  i.e. on behalf of the application
• Basic building blocks for range of higher-level
  message-passing layers
• one and two sided, also integrate network
  communication, remote file IO and interprocess
  communication
• Attempt for port to Linux resulted in Portals 3.0

3
Portals overview

• Connectionless, but protected, inorder, reliable
  communication
• e.g., different processes on behalf of same
  application no need to set up explicit
  point-to-point -gt improves scalability
• Remote operation does not specify actual address
  (typical in one-sided models)
• target is in charge of managing local memory,
  determining how to respond to a message
• Portals data structures can reside at different
  levels user, kernel, NIC -gt whichever is most
  appropriate

4
Portals addressing

• Process ID gt node, Portal table
• Portal ID gt memory buffer ID
• List of match elements
• Memory descriptors for each element
• Memory region and (optional) event queue per
  descriptor
• Both match elements and descriptors have flags
  for different behaviors

• Match list
• Source node ID
• Source process ID
• Job ID
• User ID
• 64 match bits
• 64 ignore bits
• wildcards

5
Address Translation

• API specifies that unlinking a MD (ME) frees up
  its resources
• The application doesnt need to participate in
  registering/deregistering resources

6
Access control

• Access control list of pairs of process
  identifiers and table indices
• Still assumes trustworthy senders
• Could use authentication mechanisms, but load on
  receiver -gt easy to get denial of service
  attacks.
• cookie -gt capabilities, management harder

7
Message transmission

• Initiators and targets, exchange messages, events
  are associated with message exchange
• 4 types of messages put request,
  acknowledgement, get request, reply
• Put/get requests initiated by remote nodes, so
  address translation must be performed
• Ack/reply generated by target in response to an
  initiator, reuse initiator provided data

8
Message formats
ack
put
get
reply
9
Events
10
Benefits

• Decouple communication and computation
• Zero copy, OS bypass and application bypass
• MPI progress
• Traditional MPI implementation, and other upper
  level protocols require that the
  application/messaging layer participate in
  communication in order for the communication to
  make progress
• Alternatives include interrupts and context
  switch overheads dedicated threads
• Portal data structures specify behavior and
  communications related tasks in general

11
Portals for Cray XT3

• Joint Sandia/Cray work
• Lightweight OS on compute nodes -gt Catamount
• Applications have single computational thread
• Specialized IO thread drives communications
• gt must have communication API which permits that
  the two are decoupled
• Linux on IO/Login/Service nodes
• Specialized NIC SeaStar
• 3D Torus interconnect

12
SeaStar NIC
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)