PROCESS AND PROCESSORS IN DISTRIBUTED SYSTEM

1
Chapter 4

• PROCESS AND PROCESSORS IN DISTRIBUTED SYSTEM
• (p.169p.222)

2
4.1 Threads

• If the file server had multiple threads of
  control, a second thread could run while the
  first one was sleeping.

3

• The analogy, thread is to process as process is
  to machine, holds in many ways. All the threads
  share the same address space, the same set of
  open files, child processes, timers, and signals,
  etc.

4
Fig. 4-1. (a) Three processes with one thread
each. (b) One process with three threads.
5
Fig. 4-2. Per thread and per process concepts.
6
4.1.2 Thread usage

• Consider the file server example
• The dispatcher threads reads incoming requests
  for work from the system mailbox. After examining
  the request, it chooses an idle worker thread and
  hands it the request.

7

• If there is absence of threads, the server is
  idle while waiting for the disk and does not
  process any other request.

8

• A third possibility is to run the server as a big
  finite-state machine. The finite-state machine
  approach achieves high performance through
  parallelism, but uses nonblocking calls and thus
  is hard to program.

9

• Fig. 4-3. Three organization of threads in a
  process.
• Dispatcher/worker model.
• (b) Team model. (c) Pipeline model.

10
Fig. 4-4. Three ways to construct a server.
11

• In team model case, a job queue can be
  maintained, with pending work kept in the job
  queue. With this organization, a thread should
  first check the job queue before looking in the
  system mailbox.

12

• In pipeline model case, the first thread
  generates some data and passes them on to the
  next one for processing. It is appropriate for
  producer-consumer problem.

13

• Threads are frequently useful for clients. One
  thread is dedicated full time to waiting for
  signals. When a signal comes in, it wakes up and
  processes the signal. Thus using thread can
  eliminate the need for user-level interrupts.

14

• A properly designed program that uses threads
  should work equally well on a single CPU that
  timeshares the threads or on a true
  multiprocessor, so the software issues are pretty
  much the same either ways.

15
4.1.3 Design issues for thread packages

• A set of primitives (e.g. library calls) which is
  available to the user relating to thread is
  called a threads package.

16

• With a static design, the choice of how many
  threads there will be is made when the program is
  written or when it is compiled.
• A more general approach is to allow threads to be
  created and destroyed dynamically during
  execution.

17

• Two alternatives are used for synchronization o
  threads mutex and condition variable. The
  difference between mutexes and condition
  variables is that

18

• 
  mutexes are used for short-term locking, mostly
  for guarding the entry to critical regions.
  Condition variables are used for long-term
  locking waiting until a resource becomes
  available.

19
lock mutex check data structures
while (resource busy) wait (condition
variable) mark resource as busy Unlock
mutex
lock mutex mark resource as free Unlock
mutex Wakeup (condition variable)
Fig. 4-5. Use of mutexes and condition variables.
20

• Local variables and parameters do not cause any
  trouble, but variables that are global to a
  thread but not global to the entire program can
  cause difficulty.

21
Fig 4-6. Conflicts between threads over the use
of a global variable.
22

• Various solutions to this problem are possible
• To prohibit global variables altogether.
• To assign each thread its own private global
  variables.

23

• New library procedures can be introduced to
  create, set, and read these thread-wide global
  variables

24

• create-global(bufptr)
• set-global(bufptr, buf)
• bufptrread-global(bufptr)

25
4.1.4 Implementing a threads package

• There are two ways to implement a threads
  package in user space and in the kernel.

26
Fig. 4-8. (a) A user-level threads package. (b) A
threads packaged managed by the kernel.
27

• User-level threads have some advantages
• Thread switching is faster than trapping to the
  kernel.
• It allows each process to have its own customized
  algorithm.

28

• It scales better, since kernel treads invariably
  require some table space and stack space in the
  kernel, which can be a problem if there are a
  very large number threads.
• All calls to a run time system procedure are
  lesser cost than a call that is implemented as
  system calls.

29

• User-level threads have some major problem
• It is the problem of how blocking system calls
  are implemented. Since letting the thread make
  the system calls is unacceptable, it will stop
  all the threads.

30

• One solution is that the system calls could all
  be changed to be nonblocking
• Another solution is to tell in advance if a call
  will block. This approach requires rewriting
  parts of the system call library, is inefficient
  and inelegant, but there is little choice.

31

• Another problem is that if a thread starts
  running, no other thread in that process will
  ever run unless the first thread voluntarily
  gives up the CPU.
• One possible solution is to have run-time system
  request a clock signal (interrupt).

32

• For application where the threads block often in
  a multithread file server, these threads are
  constantly making system calls. It will need for
  constantly checking to see if system call is safe.

33
4.1.5 Threads and RPC

• Bershad et al. (1990) have proposed a new scheme
  that makes it possible for a thread in one
  process to call a thread in another process on
  the same machine much more efficiently than the
  usual way.

34

• The
  call is made in such a way that the arguments are
  already in place, so no copying or marshalling is
  needed.

35

• Another technique is widely used to make remote
  RPCs faster as well. When a server thread blocks
  waiting for a new request, it really does not
  have any important context information.

36

• When a new message comes in to the servers
  machine, the kernel creates a new thread
  dynamically to service the request.

37

• 
  This method has several major advantages over
  conventional RPC no context has to be saved, no
  context has to be restored, time is saved by not
  having to copy in coming messages.

38
4.1.6 An example threads package

• In OSFs software, it has a total of 51
  primitives (library procedures) relating to
  threads that user programs can call. They are
  classified into seven categories.

39

• The first category deals with thread management.
• The second group allows user to create, destroy,
  and manage templates for threads, mutexes, and
  condition variables.

40

• The third group deals with mutexes.
• The fourth group deals with the calls relating to
  condition variables.

41

• The fifth group manipulates the per-thread global
  variables.
• The sixth group deals with the selected calls
  relating to killing threads.

42

• The last group deals with the selected scheduling
  calls.

43
4.2.1 The Workstation Model

• Diskless workstations are popular at universities
  and companies for several reasons
• Less expensive and fast speed

44

• Ease of maintenance
• Less noisy
• Symmetry and flexibility

45
Fig. 4-10. A network of personal workstations,
each with a local file system.
46

• Diskful workstations can be used in one of at
  least four ways
• Paging and temporary files
• Paging, tempory files, and system binaries

47

• Paging, tempory files, system binaries, and file
  caching
• Complete local file system

48

• The advantages and disadvantages of disk usage on
  workstations
• Easy to understand
• Fixed amount of dedicated computing power

49

• Guaranteed response time
• Two problems cheaper
  processor chip ? personal multiprocessor idle
  workstation uses it

50
Fig. 4-12. A registry-based algorithm for finding
and using idle workstations.
51

• The flaws to use idle workstations such as rsh
  machine command
• Put the full burden of keeping track of idle
  machines on the user
• Different remote process is running on requested
  idle machine

52

• The research on idle workstations has centered on
  solving the problems described in(4)

53

• How is idle workstation found? Server driven
  registry file on server or broadcast message and
  maintain registry on each machine. Client
  driven request and wait it reply.

54

• How can remote process be run transparently?
  Marking programs run on
  remote machines as though they were running on
  their home machines is possible, but it is
  complex and tricky business.

55

• What happens if the machines owner comes back?
• Kill off the intruding process
• Give a fair warning
• Migrate the process to another machine.

56

• When the process is gone it should leave the
  machine in the same state at it found it, to
  avoid disturbing the owner.

57
4.2.3 The Processor Pool Model
Fig. 4-13. A system based on the processor pool
model
58

• What happens when it is feasible to provide 10 or
  100 times as many CPUs as there are active users?
  One solution is to use a personal multiprocessor.
  But
  it is an inefficient design. An alternative
  approach is to construct a processor pool model.

59

• The motivation for the processor pool idea comes
  from taking the diskless workstation idea a step
  further. All the computing power is converted
  into idle workstations that can be accessed
  dynamically.

60

• The biggest argument for centralizing the
  computing power in a processor pool comes from
  queueing theory.

61
Fig. 4-14. A basic queueing system.
62

• Since dividing the processing power into small
  servers is a poor match to a workload of randomly
  arriving requests, a few servers are busy even
  overloaded, but most are idle.

63

• 
  It is this wasted time that is eliminated in the
  processor pool model, and the reason why it gives
  better overall performance.

64

• The queueing theory result is one of the main
  arguments against having distributed systems at
  all. But there are also arguments in favor of
  distributed computing. Cost is one of them.
  Reliability and fault tolerance are factors as
  well.

65

• A possible compromise is to provide each user a
  personal workstation, and to have a processor
  pool in addition. Interactive work can be done on
  workstations, but all noninteractive processes
  run on the processor pool.

66
4.3.1 Processor Allocation Models

• Processor allocation strategies
• Nonmigratory
• Migratory

67

• Criteria to select some allocation algorithm
• CPU utilization
• Response time

68
Fig.4-15. Response times of two processes on two
processors.
69

• Response ratio it is define as the amount of
  time it takes to run a process on some machine
  divided by how long it would take on some
  unloaded benchmark processor.

70
4.3.2 Design Issue for Processor Allocation
Algorithm

• Deterministic algorithms are appropriate when
  everything about processor behavior is known in
  advance or a reasonable approximation is
  obtainable. Heuristic algorithms are used in
  systems where the load is completely
  unpredictable.

71

• Centralized algorithms collect all the
  information in one place allows a better decision
  to be made, but is less robust and can put a
  heavy load on the central machine.

72

• In practice, most actual distributed systems
  settle for heuristic, distributed, suboptimal
  solutions due to the difficulty of obtaining
  optimal ones.

73

• When a process is about to be created and if that
  machine is too busy, the new process will be
  transferred somewhere else or to be gotten rid of
  it. This issue is called transfer policy.

74

• Once the transfer policy has decided to get rid
  of a process, the location policy has to figure
  out where to send it. The location policy cannot
  be local. It needs information about the load
  elsewhere to make a intelligent decision.

75

• The
  information can be collected by sender or
  receiver doing initiation.

76
Fig. 4-16. (a) A sender looking for an idle
machine. (b) A receiver looking for work to do.
77
4.3.3 Implementation Issues for Processor
Allocation Algorithm

• Measuring load is required in the design issues,
  but is not simple. Several approaches are
  provided
• Count the number of process.

78

• Count only processes that are running or ready.
  Most put only a small load on the system.
• Fraction of time the CPU is busy Set up a timer
  and let it interrupt the machine periodically.

79

• Another implementation issue is how overhead is
  deal with. Few do, mostly because it is not easy.

80

• Next implementation consideration is complexity
• Pick a machine at random and just sends the new
  process there.

81

• Pick a machine at random and sends it a probe
  asking if it is underloaded overloaded.
• Probe k machines to determine their exact loads.
  The process is then sent to the machine with the
  smallest load.

82
4.3.4 Example Processor allocation algorithm

• A graph-theoretic deterministic algorithm
  The goal is to find the
  partitioning that minimizes the network traffic
  while meeting all the constraints.

83
Fig. 4-17. Two ways of allocating nine processes
to three processor.

• Disadvantage It requires complete information in
  advance.

84

• A centralized algorithm A positive
  score of usage table indicate that the
  workstation is a net user of system resources,
  where as a negative one means that it needs
  resources. A zero score is neutral.

85
Fig. 4-18. Operation of the up-down algorithm.
86

• A hierarchical algorithm If the
  manager receiving the request thinks that is has
  too few processors available, it passes the
  request upwards in the tree to its boss.

87

• 
  If the boss cannot handle it either, the request
  continues propagating upward until it reaches a
  level that has enough available workers at its
  disposal.

88
Fig. 4-19. A processor hierarchy can be modeled
as an organizational hierarchy.
89

• When a process is created, the machine on which
  it originates sends probe message to a
  randomly-chosen machine, asking if its load is
  below some threshold value.

90

• 
  If so, the process is send there. If not, another
  machine is chosen for probing. If no suitable
  host is found within N probes, the algorithm
  terminates and the process runs on the
  originating machine.

91

• When a process finishes, the system check to see
  if it has enough work. If not, it picks some
  machine at random and asks it for work. If that
  machine has nothing to offer, a second, and then
  a third machine are asked.

92

• 
  If no work is found with N probes, the receiver
  temporarily stops asking, does any work it has
  queued up, and tries again when the next process
  finishes.

93

• A bidding algorithm Each
  processor advertises its approximate price. A
  process wants to startup a child process. The set
  of processor whose service it can afford is
  determined.

94

• 
  From this set, it computes the best candidate.
  It generates a bid and sends the bid to its first
  choice. Processors collect all the bids sent to
  them, and make a choice, presumably by picking
  the highest one.

95
4.4 Scheduling in Distributed Systems

• It is difficult to dynamically determine the
  interprocess communication patterns.

96
Fig. 4-20. (a) Two jobs running out of phase with
each other. (b) scheduling matrix for eight
processors, Scheduled allocated slots.
97

• Ousterhout (1982) proposed coscheduling concept,
  which takes interprocess communication patterns
  into account while scheduling to ensure that all
  members of a group run at the same time.

98

• 
  For example, four process that must communicate
  should be put into slot3, on processors 1, 2, 3,
  and 4 for optimum performance.

99
4.5 Fault Tolerance

• Component faults
• Transient faults
• Intermittent fault
• Permanent fault

100

• The goal of designing and building fault-tolerant
  systems is to ensure that system as whole
  continues to function correctly, even in the
  presence of faults.

101

• If some component has a probability p of
  malfunctioning in a given second of time, the
  probability of it not failing for k consecutive
  seconds and then failing is p(1-p)k. Mean time to
  failure

102

• System failures
• Fail-silent faults
• Byzantine faults
• Dealing with Byzantine faults is going to be much
  more difficult than dealing with fail-silent ones.

103

• A system that has the property of always
  responding to a message within a known finite
  bound if it is working is said to be synchronous.
  A system not having this property is said to be
  asynchronous.

104

• 
  Hence, asynchronous systems are going to be
  harder to deal with than synchronous ones.

105

• Three kinds of redundancy are possible
• Information redundancy extra bits are
  added to allow recovery from garbled bits.

106

• Time redundancy an action is performed and it is
  performed again if need be.
• Physical redundancy extra equipment is added to
  tolerate the malfunction of component.

107

• There are two ways to organize extra processors
• Active replication
• Primary backup

108

• A system is said to be k fault tolerant if it can
  survive faults in k components and still meet it
  specifications. For example, k1 processors is
  enough if the processors fail silently, while
  2k1 processors are needed if the processors
  exhibit Byzantine failures.

109

• Consider the simple protocol of figure 4-22 in
  which write operation is depicted
• A primary crashes before doing the work.

110

• A primary crashes after doing the work but before
  sending the update.
• A primary crashes after step 4 but before step 6.

111
Fig. 4-22 A simple primary-backup protocol on a
write operation.
112

• What happens if the primary has not crashed but
  is merely slow?

113

• The general goal of distributed agreement
  algorithms is to have all the nonfaulty processor
  reach consensus on some issue, and do that within
  a finite number of step.

114

• Are messages delivered reliably all the time?
• Can processes crash, and if so, fail-silent or
  Byzantine?
• Is the system synchronous or asynchronous?

115

• Two-army problem Even with
  nonfaulty processors, agreement between even two
  processes is not possible in the face of
  unreliable communication.

116

• Byzantine general problem Lamport proved that in
  a system with m faulty processors, agreement can
  be achieved only if 2m1 correctly functioning
  processors are present, for a total of 3m1.

117

• Fischer proved that in a distributed system with
  asynchronous processors and unbounded
  transmission delays, no agreement is possible if
  even one processor is faulty.

118
Fig.4-23. The Byzantine generals problem for 3
loyal generals and traitor. (a) The generals
announce their troop strengths (in units of 1K).
(b) The vectors that each general assembles based
on (a). (c) The vectors that each general
receives in step 2.
119
Fig. 4-24. The same as Fig. 4-23, except now with
2 loyal generals and one traitor.