Lightweight Causal and Atomic Group Multicast

1
Lightweight Causal and Atomic Group Multicast

• Kenneth Birman, André Schiper, Pat Stephenson
• August 1991

Presented by Eric Engineer Assisted by Justin
Singer
2
Multicast

• What is multicast?
• Transmission of a single message to a select
  group of recipients
• More efficient than multiple unicasts
• Allows for unknown destination delivery

3
Example Distributed Database
Client
Client
Account DB
Account DB
Account DB
identical servers
4
Example Distributed Database

• Distribution allows for
• Replication
• Cooperative algorithms
• Fault tolerance

5
Example Distributed Database

• Communication concerns
• What if there is message loss?
• What if messages delivered in different order?
• What if a sender crashes after some, but not
  all, deliveries?
• What if failure is detected by some nodes before
  others?

6
Example Distributed Database

• Desired Features
• Reliability
• Ordering
• Broadcast/Multicast
• Delivery atomicity
• Failure atomicity

7
Key Concepts

• Process groups
• Ordering
• Vector time

8
Key Concept Process Groups

• Set of processes working together to exploit
  distribution
• Members join and leave dynamically
• Failure ? departure from all groups

9
Key Concept Ordering

• Message reception vs. delivery
• Types of ordering
• FIFO
• Causal
• Total

10
Key Concept FIFO Ordering

• Example
• A client process multicasts the following
  messages
• Create a new record in the database
• Update the contents of the record
• Delivery order matters record must first exist
  to be updated!

11
Key Concept FIFO Ordering

• Definition
• If a process multicasts a message m before it
  multicasts a message m, then no process delivers
  m unless it delivered m.

12
Key Concept Causal Ordering

• Example
• Messages on a distributed newsgroup
• Client posts message on server
• Other client posts a response
• Each server must deliver the first message before
  the second otherwise, response will be out of
  context!

13
Key Concept Causal Ordering

• Based on happens before relation (?)
• In a process, if event A is executed before event
  B, then A?B
• If A is event of sending a message and B is event
  of receiving the message, then A?B
• If A?B and B?C, then A?C

14
Key Concept Causal Ordering

• Definition
• If the multicast of a message m causally precedes
  the multicast of message m, then no correct
  process delivers m unless it has already
  delivered m

15
Key Concept Total Ordering

• Example
• Consider two concurrent, unrelated multicasts
• User deposits money in account
• Bank updates interest earned
• Causal ordering allows unrelated messages to be
  delivered in any order. What if a consistent
  sequence ensures correctness?

16
Key Concept Total Ordering

• Atomic multicast provides total ordering ?
  concurrent messages are delivered in the same
  order at all destinations
• Definition
• If processes p and q both deliver messages m and
  m, then p delivers m before m iff q delivers m
  before m

17
Key Concept Vector Time

• Problem
• Ordering requires some notion of time
• Solution
• Vector clock
• Serves as a logical timestamp

18
Key Concept Vector Time
Vector Time (VT) Vector that counts number of
messages sent by each process in a group
P0 P1 P2
Pn-1

• One counter per process in a group

19
Key Concept Vector Time
time
20
Key Concept Vector Time
21
Protocols

• CBCAST ? causal ordering
• ABCAST ? total ordering
• Assumption
• Reliable FIFO reception provided by transport
  layer

22
Protocols Design Evolution

• One group, fixed membership
• Membership changes
• Handling failures
• (Multiple groups)

23
CBCAST

• Problem
• Must preserve
• Safety causality is never violated
• Liveness messages eventually get delivered
• Solution
• Delay messages on a queue (ordered by vector
  time) and then deliver

24
CBCAST (one group, fixed membership)

• Message m is multicast by process p0
• Process p1 compares its timestamp with that of
  the message m
• p1 delays delivery of m until
• Its counter for p0 is exactly one less than that
  of the message
• Its counters for all other processes are greater
  than or equal to those of the message

25
CBCAST (one group, fixed membership)
Process1
Process0
Process0 multicasts a message
Message
Process2
far away
26
CBCAST (one group, fixed membership)
Message delivered to Process1 its clock is
updated
Process1
Process0
Message
Process2
27
CBCAST (one group, fixed membership)
Process1 multicasts a response
Process1
Process0
Response
Process2
Message0
28
CBCAST (one group, fixed membership)
Process1
Process0
delay queue
Response
Process2
Message
Response delayed since there is an outstanding
message
29
CBCAST (one group, fixed membership)
Process1
Process0
delay queue
Response
delivery
Process2 updates its timestampallowing for
delivery of Response
Process2
30
ABCAST

• Problem
• Desire a total ordering consistent across all
  processes
• Solution
• Assign one process in the group as token-
  holder and allow it to establish a canonical
  ordering for others to use

31
ABCAST (one group, fixed membership)
VT(m0)VT(m1)VT(m2)
message1
message0
message2
ABCAST
Token-holder process
other process
32
ABCAST (one group, fixed membership)
VT(m0)VT(m1)VT(m2)
message2
message0
message1
message1
? delay queues ?
message0
message2
Token-holder process
other process
Marked undeliverable
33
ABCAST (one group, fixed membership)
message0
message1
message2
Token-holder process
other process
Marked undeliverable
34
Membership Changes

• Need to dynamically add/remove processes from a
  group such that changes are atomic
• Definitions
• View List of a groups members
• View sequence Directly prior view differs only
  by one member
• Stability A multicast m is stable if it has
  been received by all intended recipients

35
Membership Changes Flush

• On definition of a new view
• Use deterministic rule to designate one member of
  the new view as flush coordinator Pc
• No new multicasts initiated
• All processes in new view (other than Pc) wait
  until all multicasts have stabilized and then
  send a flush message to Pc

36
Membership Changes Flush

• A process does not initiate new multicasts until
  it has received flush messages from all other
  members of new view
• When it has received flush messages from all the
  others, Pc multicasts its own flush
  messageindicating its safe to send to new view

37
Handling Failures

• Delivery, addressing atomicity, and ordering
  threatened when failures occur
• Retransmit unstable messages
• Refine flush protocol
• ABCAST ordering when token-holder fails ?
  deterministic ordering rule
• use message-id

38
Performance

• CBCAST cost grows roughly linearly with size of
  destination group

39
Performance

• ABCAST runs at nearly half the speed of CBCAST
• Conclusions

40
Performance

• Most of time spent in transport layer
• (b/c of FIFO ordering?)

41
Conclusions

• Importance of multicast to distributed systems
• Key issues atomicity, ordering, and failure
• Vector clocks, CBCAST, ABCAST
• Effective algorithm
• Does not solve ABCAST for multiple groups