Distribution Seminar

1
Distribution Seminar

• Chien-Liang Fok
• liang_at_cse.wustl.edu

2
Topic One

• Clocks, Order, and Mutual Exclusion in a
  Distributed System
• Leslie Lamport, "Time, Clocks, and the Ordering
  of Events in a Distributed System",
  Communications of the ACM, 21(7), pp. 558-565,
  July 1978.

3
What is a Distributed System?

• A collection of processes communicating through
  message passing.

4
Representation of a Process

• A process is a sequence of events
• It may not be possible to tell which of two
  events in different processes occurred first.
• Ex occurs before Ey is only a partial order

P1
P2
Pm
5
The a?b relation

• a?b denotes a occurs before b
• We want to develop an algorithm that uses this
  relation to determine a total ordering of events
  in our system
• Real clocks are not easily synchronized
• Must define truth of a?b w/o using physical
  clocks

6
Truth of A ? B

• A?B is true when
• A occurs before B in the same process
• A and B in different processes A sends message
  while B receives it
• A?C and C?B

A




P1

B



P2
7
Concurrency

• Events a and b are concurrent iff
• This implies that a cannot causally affect b and
  vice versa
• Note that for any event, e, ?(e?e) is true
• ? is an irreflexive partial ordering on the set
  of all events in the system

?(A?B) ? ?(B?A)
8
Concurrency (Continued)

• Given the space time diagram on right
• P1?P2, Q1?Q3
• P1?Q1, Q3 ? P3
• ?(P2?Q2) ? ?(Q2?P2)
• P2 and Q2 are concurrent!

P3
Q3
Q2
P2
Q1
P1
Time
Process P
Process Q
Event
Sent Message
9
Logical Clocks

• Assign a number to each event such that if a?b
  then C(a) lt C(b) where C(x) denotes the
  number assigned to event x.
• This is known as the clock condition
• Note the converse is not true

10
Clock tick

• Occurs between events on a process
• Let a and b be two events such that c(a)4 and
  c(b)7
• Ticks 5, 6, and 7 occur between a and b
• A tick line connects like-times between two or
  more processes
• Must be between any two events on a process
• Every message line must cross a tick line

11
Tick Lines Visually Rep.
Sent Message
Event
Tick Line
12
Clock Implementation

• How to implement a clock that satisfies the clock
  condition??
• Add a register to each process
• Increment it after each event
• Augment each message with a timestamp Tm
• When received message, increase the clock
  registers value to be greater than both its
  current value and Tm

13
Obtaining Total Order

• Using system of clocks, total ordering of all
  events in the system is easy
• Order events by the time they occur
• Break ties by using an arbitrary ordering of
  processes
• Let a?b be true if c(a)ltc(b) or c(a)c(b) and as
  process is preferred over bs
• The ? relation is NOT unique!

14
Using Total Order to Solve theMutual Exclusion
Problem

• Only one process can be granted the resource at a
  time
• Processes must be granted the resource in the
  order they requested for it
• Every process that is granted the resource will
  eventually release it

15
Mutual Exclusion Algorithm (1/4)

• Implement the system of clocks
• Each process has a clock register
• Augment each process with a request queue rq
• Initialize all rqs to contain message t0p0
• t0 is smaller than all clock registers
• p0 is the process initially granted the resource

16
Mutual Exclusion Algorithm (2/4)
pi
Acknowledge
Request

• pi requests resource by sending tmpi to every
  other process and putting tmpi in rqi
• When pj receives tmpi, add to rqj and send
  acknowledgement to pi

17
Mutual Exclusion Algorithm (3/4)
pi
Request

• pi releases resource by sending a request to
  everyone and removing tmpi from rqi
• When pj receives the request, it removes tmpi
  from rqj

18
Mutual Exclusion Algorithm (4/4)

• pi is granted the resource when both
• tmpi ? all other requests in rqi
• pi received a message from everyone else with
  time stamp greater than tm
• Note this can be determined locally

19
Vulnerabilities

• Two vulnerabilities
• If just one process deadlocks, the whole system
  dies
• Anomalous behavior if system model does not match
  real-life model

20
Anomalous Behavior
Hay!! Request it!
User B
User A

• User bs message may have a lower time stamp than
  a

21
Addressing Anomalous Behavior

• Introduce strong clock condition, a?b
• For any two events a and b in the system,
  if a?b then c(a)ltc(b)
• This is not supported by our clock system!

22
Solution to Anomalous Behavior

• Use real clocks, let
• Ci(t) reading of clock i at time t
• ? maximum clock error rate, sec off/sec
• ? maximum error between clocks, sec
• ? maximum message transmission delay
• To prevent anomalous behavior, ensure
• Ci(t ?) - Cj(t) gt 0
• For the above to be true,
• ?/(1-?) ? ?
• See end of paper for derivation/proof.

23
Topic 2

• Vector Clocks
• Friedemann Mattern, Virtual Time and Global
  States of Distributed Systems.
• Cosnard M. et al. (Eds) Proc. Workshop on
  Parallel and Distributed Algorithms,
  North-Holland / Elsevier, pp. 215-226, 1989
• Reprinted in Z. Yang, T.A. Marsland (Eds.),
  "Global States and Time in Distributed Systems",
  IEEE, 1994, pp. 123-133.)

24
Motivation

• Previously, partially ordered events were mapped
  into a total order
• Two simultaneous events are treated as if one
  occurred first
• Useful information is lost
• Linear ordering of events is sometimes not good
  enough

25
The Nature of Time

• Time has the following properties
• Transitivity
• Irreflexivity
• Linearity
• Eternity
• goes on forever
• Density
• can be divided into infinitesimally small units

26
The BIG Question

• Can we develop a virtual clock system that
  preserves more of the systems timing properties?
• Lamports algorithm does not preserve causal
  independence

27
Vector time

• Suppose a deity can observe all clocks and store
  their values in a vector

Process P
Process Q
Process R
0 0 1
0 1 1
1 1 1
1 2 1
1 2 2
1 3 2
2 3 2
28
Goal

• Create algorithm so each process gets an optimal
  approximation of this vector

29
Technique

• Replace the register in each process with a
  vector.
• Size of vector number of processes
• Update value in vector of own time like usual
• When receive message, update estimated time of
  other process based on
• current time
• timestamp vector within message

30
Using our Technique
1 0 0
2 0 0
Process P
0 1 1
0 2 1
1 3 1
Process Q
0 2 2
0 0 1
Process R
0 0 1
0 1 1
1 1 1
1 2 1
1 2 2
1 3 2
2 3 2
Ideal
31
Things to Note

• Each process has perfect local information
• Given two time vectors u, v
• u v iff ?i u i v i
• u lt v iff (u v) ? (u ? v)
• u v iff ?(u lt v) ? ?(v lt u)
• u v and u lt v are partial orders
• u v is the concurrent relation
• It is NOT transitive!!

32
Cuts
Process P
Process Q
Process R
Consistent Cut
Inconsistent Cut
Sent Message
Event
Cut Event
Cut Line

• A timing diagrams with consistent cuts can be
  modified so the cuts are straight lines.

33
Time at a Cut
Process P
Process Q
Process R
Cut 1
Cut 2
Sent Message
Event
Cut Event
Cut Line

• The time vector of a consistent cut is the time
  of each individual process at the cut

34
Key Point

• For any set of events
• the lattice of consistent cuts
• the lattice of possible time vectors
• Thus, two events e and e are causally related
  iff TimeVector(e) lt TimeVector(e)

35
Applications

• When do we care which event in a set of
  concurrent events occurred first??
• Distributed debugging
• Must consider causal relationship between events
  in the system
• Performance analysis
• Vector clocks allows one to calculate potential
  concurrency

36
Topic 3

• Global Snapshot
• K. Mani Chandy and Leslie Lamport, "Distributed
  Snapshots Determining Global States of
  Distributed Systems", ACM Transactions on
  Computer Systems, 3(1), pp. 63-75, February 1985.

37
Motivation

• Knowing the global state of the system is useful

38
Restriction

• Cannot pause the system
• Cannot prevent system from performing regular
  processing

39
Problem

• Given the restrictions, it is impossible to
  obtain global state
• Cannot halt system
• System is constantly changing
• What can we do?

40
Goal

• Find a possible global state
• Can determine value of stable system properties
• deadlock
• Termination
• By definition if a possible global state
  satisfies a stable property, the system will
  forever satisfy this property.

41
The Model
Channel

• Processes send and receive messages through
  channels

42
Global State

• Consists of
• Process states
• Whats the state of the process?
• Channel states
• What messages are being sent through it?

43
The Algorithm

• Sender process p
• Record state
• Send marker on all outgoing channels
• Receiver process q
• If q has not recorded state
• record state
• record channel state as empty
• If q has recorded state
• record channel state as all messaged received
  after recording, but before receiving marker

44
The Algorithm (2/2)

• Assuming strongly connected graph, all process
  will have recorded
• Process state
• Channel state for each incoming channel
• Consolidate these records to form global snapshot
• The paper proves this snapshot is a reachable
  from one that actually existed.
• Enough to prove stable properties of system

45
Conclusions

• Virtual clocks are necessary since physical
  clocks are not easily synchronized
• Vector clocks improve upon Lamports virtual
  clock since it preserves causal independence
• Global snapshots of possible states is useful to
  determine stable properties of the system

46
Event Definition

• Process is a total ordering of events
• en ltp, s, s, M, cgt
• p the process doing the event
• s initial state
• s final state
• m message sent
• c the channel used

P1
47
Changing Global State

• Given
• Global state S
• Event e ltp, s, s, M, cgt
• e changes S when
• Process p is in state s
• Channel c contains message M at head