Global States in a Distributed System

1
Global States in a Distributed System

• By John Kor and Yvonne Cheng

2
Initial Problem Example

• Garbage Collector
• Frees up memory which is no longer in use
• Checks if a reference to memory still exists
• What about in a distributed system

3
Initial Problem Example (contd)

• A distributed system consists of multiple
  processes
• Each process is located on a different computer
• No sharing of processor or memory

4
Initial Problem Example (contd)

• Each process can only determine its own state
• Problem How do we determine when to garbage
  collect in a distributed system?
• How do we check whether a reference to memory
  still exists?

5
System Model

• A distributed system consists of multiple
  processes
• Each process is located on a different computer
• Each process consists of events
• An event is either sending a message, receiving a
  message, or changing the value of some variable
• Each process has a communication channel in and
  out

6
Our Garbage Collection Problem

• In order to test whether a certain property of
  our system is true, we cannot just look at each
  process individually
• A snapshot of the entire system must be taken
  to test whether a certain property of the system
  is true
• This snapshot is called a Global State

7
Definition

• The global state of a distributed system is the
  set of local states of each individual processes
  involved in the system plus the state of the
  communication channels.

8
Determinism

• Deterministic Computation
• At any point in computation there is at most one
  event that can happen next.
• Non-Deterministic Computation
• At any point in computation there can be more
  than one event that can happen next.

9
Deterministic Computation
10
Non-Deterministic Computation
11
Determinism

• Deterministic computation
• A local event would reveal everything about the
  global state!
• The process will know other process state
• Non-Deterministic computation
• Because of branching, a local event cannot reveal
  what the next step will be

12
Simple Algorithm

• Create a new process that collects the states of
  every other process
• Every process will save their state at an
  arbitrary time and send it to this new process

13
Advantages

• Very simple
• Easy to implement

14
Problems?

• Based on the assumption that all processes work
  on a synchronized global clock
• Wrong assumption!

15
Problems (contd)

• State recorded by p

p
q
m
16
Problems (contd)
p
q
m

17
Problems (contd)

• State recorded by q

p
q

m
18
Problems (contd)

• Global state recorded

p
q
m
m
19
Another view
p
m
q
20
Another view

• Process p has no record of sending m
• Process q HAS record of receiving m
• Problem?
• Global state does not show p sending m, therefore
  there is confusion as to where m came from
• Breaks the Consistency concept

21
Consistency

• A global state is consistent if it could have
  been observed by an external observer
• If e ? e , then both e and e must reside within
  the same state
• For a successful Global State, all states must be
  consistent

22
Solution

• Need to develop an asynchronous algorithm
• Cannot depend on a clock
• Must ensure consistency in all global states

23
Assumptions

• Distributed system Finite set of processes and
  channels described by graph
• Processes
• Set of states, initial state, set of events
• Channels
• FIFO, error-free, infinite buffers, arbitrary but
  finite delay

24
PART 2

• Presented By Yvonne

25
Idea of a global state recording algorithm

• each process records its own state
• the two processes incident by one channel
  cooperate in recording the channel state

26
Challenge

• No global clock
• Need a meaningful result
• Superimposed on underlying computation

27
Meaningful The notion of Consistency

• it could have been observed by an external
  observer
• All feasible states are consistent

28
An Example
q
p
Sp0
Sp1
Sp2
Sp3
p
m2
m1
m3
q
Sq0
Sq1
Sq2
Sq3
29
A Consistent State?
q
p
Sq1
Sp1
Sp0
Sp1
Sp2
Sp3
p
m2
m1
m3
q
Sq0
Sq1
Sq2
Sq3
30
Yes
q
p
Sq1
Sp1
Sp0
Sp1
Sp2
Sp3
p
m2
m1
m3
q
Sq0
Sq1
Sq2
Sq3
31
A Consistent State?
q
p
Sq3
Sp2
m3
Sp0
Sp1
Sp2
Sp3
p
m2
m3
m1
q
Sq0
Sq1
Sq2
Sq3
32
Yes
q
p
Sq3
Sp2
m3
Sp0
Sp1
Sp2
Sp3
p
m2
m3
m1
q
Sq0
Sq1
Sq2
Sq3
33
An inconsistent State
q
p
Sq3
Sp1
Sp0
Sp1
Sp2
Sp3
p
m2
m1
m3
q
Sq0
Sq1
Sq2
Sq3
34
Conducting algorithm Using An Example

• Processes p and q
• Channels c and c
• Token t

p
q
c
c
35
An Example

• p records its state

p
q
c
t
c
36
An Example

• q, c, and c record their states

p
q
c
t
c
37
An Example

• The composite global state!

p
q
c
t
t
c
38
An Example

• n number of messages sent along c before ps
  state is recorded
• n number of message sent along c before cs
  state is recorded

p
q
c
c
39
An Example

• - Reason of inconsistency nltn

p
q
c
t
n 0
c
p
q
c
t
n 1
c
40
Similar scenario

• c is recorded when the token is at process p.
• p sends the token through channel c, and the
  states of c, p, and q are recorded.
• The recorded global state no tokens in the
  system.
• The reason of inconsistency ngtn

41
Conclusion from the example

• A consistent global state
• requires
• n n

42
Similar Conclusion

• m number of messages received along c before
  qs state is recorded
• m number of messages received along c before
  cs state is recorded
• To be consistency mm

43
Some other equations
n gt m

• m number of messages received along c before
  cs state is recorded
• n number of messages sent along c before cs
  state is recorded
• m number of messages received along c before
  ps state is recorded
• n number of messages sent along c before ps
  state is recorded
• n n
• m m

n gt m
44
Other Fact

• The state of channel c that is recorded must be
  the sequence of messages sent along the channel
  before the senders state is recorded, excluding
  the sequence of messages received along the
  channel before the receivers state is recorded.
• Two cases
• nm c is empty
• ngtm c must be the (m1)stnth messages sent
  by p along c

45
Put All TogetherA brief sketch of the algorithm

• p sends a marker message along all its outgoing
  channels after it records its state and before it
  sends any other messages.
• On receipt of a marker message from channel c
• else
• state ( c ) messages received on c since it
  had recorded its state excluding the marker.
• if p has not recorded its state
• record the state
• state ( c ) EMPTY

46
Chandy and Lamport Algorithm

• Features
• Does not promise us to give us exactly what is
  there
• But gives us consistent state!!

47
Algorithm in Action
Sp0
Sp1
Sp2
Sp3
p
m1
m2
m3
q
Sq0
Sq1
Sq2
Sq3
48
Algorithm in Action
q records state as Sq1 , sends marker to p
Sp0
Sp1
Sp2
Sp3
p
m1
m2
m3
q
Sq0
Sq1
Sq2
Sq3
49
Algorithm in Action
p records state as Sp2, channel state as empty
Sp0
Sp1
Sp2
Sp3
p
m1
m2
m3
q
Sq0
Sq1
Sq2
Sq3
50
Algorithm in Action
q records channel state as m3
Sp0
Sp1
Sp2
Sp3
p
m1
m2
m3
q
Sq0
Sq1
Sq2
Sq3
51
Algorithm in Action
Recorded Global State ((Sp2, Sq1), (0,m3) )
Sp0
Sp1
Sp2
Sp3
p
m1
m2
m3
q
Sq0
Sq1
Sq2
Sq3
52
Algorithm in Action
Recorded Global State ((Sp2, Sq1), (0,m3)
) Computation may not even have passed
through the state recorded!
Sp0
Sp1
Sp2
Sp3
p
m1
m2
m3
q
Sq0
Sq1
Sq2
Sq3
53
What have we recorded

• The recorded consistent state can be anything!

54
Properties of the recorded global state

• Si global state when the algorithm starts
• Sj global state when the algorithm finishs
• S state recorded by the algorithm
• Then
• S is reachable from Si
• Sj is reachable from S

55
S Is reachable from Si
Si
Sj
56
Sj Is reachable from S
Si
Sj
57
Still what good is it?

• Stable Properties
• A property Y is called a stable property iff for
  all states S reachable from S
• Y(S) -gt Y(S)

58
Detection of Stable Properties

• Outcome false
• while ( outcome false )
• determine Global State S
• outcome Y (S)

59
Checkpoint

• S serves as a checkpoint
• On a failure, restart the computation from S
• Problem!
• Not able to restore to Sj

Si
S
Sj
60
Solution Publishing

• A Broadcast medium
• A central recorder process records all the
  messages received by each process
• Processes record their states at their own time
  and send it to the recorder

61
Determining Global State

• Recorder can construct global state from
• Checkpointed States of all processes
• Plus
• Messages recorded since last checkpoint

62
Problems

• Publishing keeps track of all messages received
  by each process
• Expensive!
• Solution
• recorder takes checkpoint of process p at time t
• deletes all messages recd by p before t.

63
Comparison
64
Conclusion

• Global State detection difficult in Distributed
  Systems
• Snapshot algorithm may not give an actual state
  but is very helpful in detecting Stable
  Properties
• Publishing gives an asynchronous way of
  determining global states but is unscalable