Synchronization

1
Synchronization
Distributed Computing

• Dr. Yingwu Zhu

2
Topics to Discuss

• Physical vs. Logical Clocks
• Lamport Clocks
• Lamport Vector Clocks
• Mutual Exclusion Algorithms
• Election Algorithms

3
Synchronization? Whats for?

• Temporal ordering of events produced by
  concurrent processes
• Synchronization between senders and receivers of
  message
• Msg m1 from process P to Q is sent before or
  after msg m2 from process Q?
• Coordination of joint activity
• Serialization of concurrent access for shared
  objects (e.g., access to a shared printer)

4
An Ideal World

• All machines clocks are perfectly synchronized,
  synchronization is really easy!

5
Clock Synchronization Example

• In centralized systems, no problem for the above
  Make program
• In distributed systems, when each machine has its
  own clock, an event that occurred after another
  event may nevertheless be assigned an earlier
  time.

6
Logical vs. Physical Clocks

• Logical clock keeps track of event ordering
• Among related (causal) events
• Do not care the real time where events occurred
• Physical clock keeps time of day
• Consistent across systems

7
Physical Clock (timer) in Computers

• Real-time Clock CMOS clock (counter) circuit
  driven by a quartz oscillator
• battery backup to continue measuring time when
  power is off
• OS generally programs a timer circuit to generate
  an interrupt periodically
• e.g., 60, 100, 250, 1000 interrupts per
  second(Linux 2.6 adjustable up to 1000 Hz)
• Programmable Interval Timer (PIT) Intel 8253,
  8254
• Interrupt service procedure adds 1 to a counter
  in memory

8
Physical Clock Problems

• Getting two systems to agree on time
• Two clocks hardly ever agree
• Quartz oscillators oscillate at slightly
  different frequencies
• Clocks tick at different rates
• Create ever-widening gap in perceived time
• Clock Drift
• Difference between two clocks at one point in
  time
• Clock Skew

9
Clock Drift
Frequencies of perfect, slow and fast clocks
10
Dealing with Drift

• Assume we set computer to true time
• Not good idea to set clock back
• Illusion of time moving backwards can confuse
  message ordering and software development
  environments

11
Dealing with Drift

• Go for gradual clock correction
• If fast
• Make clock run slower until it synchronizes
• If slow
• Make clock run faster until it synchronizes
• Clock synchronization, e.g., Linear compensation
  function

12
Compensating for a fast clock
13
Getting Accurate Time

• Attach GPS receiver to each computer
• 1 msec of UTC (Universal Coordinated Time)
• Attach WWV radio receiver
• Obtain time broadcasts from Boulder or DC
• 3 msec of UTC (depending on distance)
• Attach GOES receiver
• 0.1 msec of UTC
• Not practical solution for every machine
• Cost, size, convenience, environment

14
Practical Clock Synchronization

• NTP (Network Time Protocol)
• Berkeley algorithm

15
Clock Synchronization Network Time Protocol (NTP)

• Synchronize from another machine
• One with a more accurate clock
• Machine/service that provides time information
• Time server (w/ WWV receiver)

16
Clock Synchronization NTP
Assumption latency A?B and B?A is same, and good
estimate! Offset of A to B theta T3 - (T2-T1)
(T4-T3) / 2 Delay estimate delta (T2-T1)
(T4-T3) / 2, keeps the minimum one! Adjust
gradually e.g., to slow down, add a smaller time
for each interrupt
17
Clock Synchronization The Berkeley Algorithm

• The time server is active, polling every machine
  periodically for their time
• Based on responses, it computes an average time
  and tell every machine to adjust their clocks
• Used scenarios No machine has a WWV receiver
• All machines agree on the same time, but not
  necessarily the real time

18
The Berkeley Algorithm (1)

• The time daemon asks all the other machines for
  their clock values.

19
The Berkeley Algorithm (2)

• The machines answer.

20
The Berkeley Algorithm (3)

• The time daemon tells everyone how to adjust
  their clock.

21
Logical Clocks

• In a classic paper (1978), Lamport showed
• Although clock synchronization is possible, it
  need not be absolute
• If two processes do not interact, it is not
  necessary that their clocks be synchronized!
• More importantly, the processes should agree on
  the order in which events occur! This matters!

22
Logical Clocks

• Assign sequence numbers to messages
• All cooperating processes can agree on order of
  events
• vs. physical clocks time of day
• Assume NO central time source
• Each system maintains its own local clock
• No total ordering of events
• No concept of happened-when

23
Lamports Logical Clocks (1)

• The "happens-before" relation ? can be
  observed directly in two situations
• If a and b are events in the same process, and a
  occurs before b, then a ? b is true.
• If a is the event of a message being sent by one
  process, and b is the event of the message being
  received by another process, then a ? b
• Happens-before is transitive if a ? b and b? c
  then a? c

24
Logical clocks concurrency

• Assign clock value to each event
• if a?b then clock(a) lt clock(b)
• since time cannot run backwards
• If a and b occur on different processes that do
  not exchange messages, then neither a ? b nor b?a
  are true
• These events are concurrent

25
Lamports Logical Clocks (1)

• (a) Three processes, each with its own clock. The
  clocks run at different rates.

Lamport clocks Counters or Sequence numbers
26
Lamports Logical Clocks (2)

• (b) Lamports algorithm corrects the clocks (by
  adding 1).

27
Lamports Logical Clocks (3)

• Figure 6-10. The positioning of Lamports logical
  clocks in distributed systems.

28
Lamports Logical Clocks (4)

• Updating the local counter Ci for process Pi
• Each process maintains a local counter
• Before executing an event Pi executes Ci ? Ci
  1.
• When process Pi sends a message m to Pj, it sets
  ms timestamp ts (m) equal to Ci after having
  executed the previous step.
• Upon the receipt of a message m, process Pj
  adjusts its own local counter as Cj ? maxCj ,
  ts (m), after which it then executes the first
  step and delivers the message to the application.

29
Lamports algorithm

• Each message carries a timestamp of the senders
  clock
• When a message arrives
• if receivers clock lt message timestamp, set
  system clock to (message timestamp 1)
• else do nothing
• Clock must be advanced between any two events in
  the same process

30
Lamports algorithm

• Algorithm allows us to maintain time ordering
  among related events
• Partial ordering

31
Summary

• Algorithm needs monotonically increasing software
  counter
• Incremented at least when events that need to be
  timestamped occur
• Each event has a Lamport timestamp attached to it
• For any two events, where a?b C(a) lt C(b)

32
Example Totally Ordered Multicasting

• Updating a replicated database and leaving it in
  an inconsistent state.
• Totally Ordered Multicasting
• all msgs are delivered in the same order to each
  receiver!
• Can be implemented by Lamports logical clocks
  (multicast messages and acks, msg queue ordered
  by timestamp, msg delivered to app if it is the
  first and acks from all nodes are received)

33
Problems

• Identical timestamps two events could be
  concurrent
• Detect causal relations
• If C(e) lt C(e), cannot conclude that e?e
• Looking at Lamport timestamps, cannot conclude
  which events are causally related
• Solution use a vector clock

34
Vector Clocks (1)

• Concurrent message transmission using logical
  clocks.
• Trcv(m1) lt Tsnd(m2), but m1 and m2 are concurrent
• Lamport clocks do not capture causality!

35
Vector Clocks (2)

• Vector clocks are constructed by letting each
  process Pi maintain a vector VCi with the
  following two properties
• VCi i is the number of events that have
  occurred so far at Pi. In other words, VCi i
  is the local logical clock at process Pi .
• If VCi j k then Pi knows that k events have
  occurred at Pj. It is thus Pis knowledge of the
  local time at Pj .
• If VCa lt VCb then event a ? event b

36
Vector Clocks (3)

• Steps carried out to accomplish property 2 of
  previous slide
• Before executing an event Pi executes VCi i
  ? VCi i 1.
• When process Pi sends a message m to Pj, it sets
  ms (vector) timestamp ts (m) equal to VCi after
  having executed the previous step.
• Upon the receipt of a message m, process Pj
  adjusts its own vector by setting VCj k ?
  maxVCj k , ts (m)k for each k, after which
  it executes the first step and delivers the
  message to the application.

37
Enforcing Causal Communication

• Figure 6-13. Enforcing causal communication.

• Causally ordered multicasting weaker than
  totally ordered multicasting if 2 msgs are not
  related to each other, we do not care in which
  order they are delivered to apps.
• Assume clocks are adjusted only when
  sending/receiving msgs. Sending by
  incrementingthe item in the VC by 1 receiving
  only by adjusting to max for all components in
  VC. A msg from process i is delivered to apps
  only the following 2 conditions are met
• 1) ts(m)i VCj i 1
  2) ts(m)k lt VCjk for all k ! i

38
Mutual ExclusionA Centralized Algorithm

• (a) Process 1 asks the coordinator for permission
  to access a shared resource. Permission is
  granted.
• (b)Process 2 then asks permission to access the
  same resource. The coordinator does not reply.
• (c) When process 1 releases the resource, it
  tells the coordinator, which then replies to 2.

39
Mutual ExclusionA Centralized Algorithm

• Simple 3 messages request, grant, release
• Downsides
• Simple point of failures
• Performance bottleneck

40
Mutual Exclusion A Distributed Algorithm

• The requestor broadcasts a message containing the
  requested resource, process id, and logical time
• Three different cases
• If the receiver is not accessing the resource and
  does not want to access it, it sends back an OK
  message to the sender.
• If the receiver already has access to the
  resource, it simply does not reply. Instead, it
  queues the request.
• If the receiver wants to access the resource as
  well but has not yet done so, it compares the
  timestamp of the incoming message with the one
  contained in the message that it has sent
  everyone. The lowest one wins. (Lamports clock
  vector to implement tm)

41
Mutual Exclusion A Distributed Algorithm

1. Two processes want to access a shared resource at
   the same moment.
2. Process 0 has the lowest timestamp, so it wins.
3. When process 0 is done, it sends an OK also, so 2
   can now go ahead.

42
Mutual Exclusion A Distributed Algorithm

• Message complexity 2(n-1) per entry
• Magnify the single point of failure problem in
  centralized algorithms (n points)
• Group membership is known
• Bottleneck each machine handles same load, but
  machines may be heterogeneous

43
Mutual Exclusion A Token Ring Algorithm

• (a) An unordered group of processes on a network.
  (b) A logical ring constructed in software.

44
Mutual Exclusion A Token Ring Algorithm

• Problems
• Lost tokens, how to detect them?
• Process failures, how to detect them?

45
Election Algorithms

• Many distributed systems require one process to
  act as coordinator/initiator, or perform some
  special role
• Elect one to fit into that role
• In general, election algorithms attempt to locate
  the process with the highest process number as
  the coordinator
• Traditional alg. assumes message passing is
  reliable network topology does not change

46
Election Algorithms

• The Bully Algorithm
• P sends an ELECTION message to all processes with
  higher numbers.
• If no one responds, P wins the election and
  becomes coordinator.
• If one of the higher-ups answers, it takes over.
  Ps job is done.

47
The Bully Algorithm (1)

• (a) Process 4 holds an election.
• (b) Processes 5 and 6 respond, telling 4 to stop.
  
• (c) Now 5 and 6 each hold an election.

48
The Bully Algorithm (2)

• (d) Process 6 tells 5 to stop.
• (e) Process 6 wins and tells everyone.

49
Election A Ring Algorithm

• Figure 6-21. Election algorithm using a ring.

After discovering crash of the old coordinator,
some process initiate the ELECTION message
circulating the ring (containing process numbers
whose processes saw the message) Then, the
COORDINATOR message is circulating again,
containing all the members, the process with the
highest number is the new coordinator