The Totem System

1
The Totem System

• Jan. 6, 2001
• Bum-Ho Kim
• CDSN Lab.
• Information Communications University

2
Introduction

• Totem
• Logical token-passing ring
• Single-ring protocol
• Reliable totally ordering
• Configuration change service
• Multiple-ring protocol
• Global total ordering
• Network topology maintenance service
• Process group interface
• Deliver group membership service

3
Single-Ring Protocol (1)

• Token
• Circulate around the ring
• Only processor holding the token can broadcast a
  message
• Total ordering
• Rapid detection of faults
• Effective flow control

4
Single-Ring Protocol (2)

• Message Ordering
• Agreed delivery
• Sequence number field in the token
• Total order on message
• Provides a single sequence of message sequence
  numbers for all messages
• Detecting gaps requesting retransmission in
  retransmission requesting field in token

5
Single-Ring Protocol (3)

• Message Ordering
• Safe delivery
• All-received-upto field
• After a full token rotation, processor can
  determine all processors have received all
  messages with lower sequence numbers
• Reclaim the buffer space

6
Single-Ring Protocol (4)

• Local Configuration Services
• Consensus
• Reducing the membership until consensus is
  reached in bounded time
• First Configuration Change message
• Introducing a transitional configuration of
  reduced size
• Indicating that the agreed and safe delivery
  guarantees apply only to the transitional
  configuration
• the remaining messages of the old configuration
  are delivered
• Second Configuration Change message
• Introducing new regular configuration

7
Multiple-Ring Protocol (1)

• Model
• Multiple LANs interconnected by gatways
• Each gateway connects a pair of LANs, but a pair
  of LANs is connected directly by at most one
  gateway
• Combine sequence numbers and timestamps
• Provide a global total order of messages across
  multiple interconnected LANs
• Network partitioning remerging, and processor
  failure and recovery

8
Multiple-Ring Protocol (2)

• Services
• Reliable Ordered Delivery
• Total Ordering Lamport timestamp
• Deliver messages and topology changes in
  timestamp order
• Messages with the same timestamp are delivered in
  the order of source ring id
• Topology Maintenance Services
• Topology Change messages are generated by the
  multiple-ring protocol as a result of a
  configuration change

9
Multiple-Ring Protocol (3)

• Reliable Ordered Delivery
• A gateway broadcasts a forwarded message
• Provides the message with a new sequence number
  to be reliably delivered on a ring
• The timestamp remains unchanged to be delivered
  in the same total order on all rings

10
Multiple-Ring Protocol (4)

• Reliable Ordered Delivery
• Delivery in Agreed Order
• Processor has the recv_msgs list for all of the
  rings
• If message is the lowest timestamp message and
  the recv_msgs list for all of the rings are
  non-empty
• Guarantee Vector messages
• A gateway periodically generates broadcast
• A processor is not delayed for long by the lack
  of a message
• Contains a guarantee vector

11
Multiple-Ring Protocol (5)

• Reliable Ordered Delivery
• Delivery in Safe Order
• guarantee_vector
• Highest timestamp of any message received from
  the corresponding ring
• All entries in the column of the guarantee_array
  for the source ring must be at least equal to the
  timestamp of the message
• All of the processors and gateways have received
  the message

12
Multiple-Ring Protocol (6)

• Example

13
Multiple-Ring Protocol (7)

• Topology Maintenance Services
• Each gateway maintains a data structure
• Current view of the network topology
• Handling of Configuration Changes
• Configuration change is Reported by a
  Configuration Change message generated by
  single-ring protocol
• The gateways analyze the Configuration Change
  message to determine its effect on the network
  topology
• Topology Change messages are generated by the
  multiple-ring protocol as a result of a
  configuration change

14
Multiple-Ring Protocol (8)

• Topology Maintenance Service
• Handling of Configuration Changes
• Topology Change messages and Configuration Change
  messages are delivered to the process group layer
• To forwards information about membership to the
  application
• All processors agree on logical time for topology
  change and on messages delivered before and after
  the topology change
• All messages are processed in timestamp order

15
Flow Control

• Flow control
• Performance is limited by input buffer overflow
  under high loads
• Token provide accurate information
• The number of message transmitted during the
  previous token rotation
• Limit transmissions
• To ensure that input buffers seldom overflow

16
Conclusion

• Totem System
• Ring-based
• Extended Virtual Synchrony
• Recovery, Partition, Remerge
• Reliable Ordered Delivery
• Multiple LANs

17
References

• Totem System
• Y. Amir, L. E. Moser, P. M. Melliar-Smith, D. A.
  Agarwal, and P. Ciarfella, The Totem Single-Ring
  Ordering and Membership Protocol, ACM
  Transactions on Computer Systems 13, 4 (Nov.
  1995), 311-342.
• D. A. Agarwal, L. E. Moser, P. M. Melliar-Smith,
  and R. Budhia, Reliable Ordered Delivery Across
  Interconnected Local-Area Networks, Proceedings
  of the International Conference on Network
  Protocols, Tokyo, Japan (Nov. 1995), 365-374.
• L. E. Moser, P. M. Melliar-Smith, D. A. Agarwal,
  R. K. Budhia, and C. A. Lingley-Papadopoulos,
  Totem A Fault-Tolerant Multicast Group
  Communication System, Communications of the ACM,
  April 1996.