CS 582 / CMPE 481 Distributed Systems

1
CS 582 / CMPE 481Distributed Systems

• Fault Tolerance

2
Class Overview

• Introduction
• Definition
• failure classification
• failure semantcs
• Failure Model
• Fault Tolerant Approaches
• state machine
• primary-backup

3
Introduction

• Definition
• system is considered faulty once its behavior is
  no longer consistent with its specification
  Schneider
• Separation property of distribution systems lead
  to partial failure property
• components that one component depends on may fail
  to respond due to various reasons
• system or network failure
• system or network overload

4
Introduction (cont)

• Failure classification Cristian
• omission failure
• a server omits to respond to an input
• timing failure (performance failure)
• a server responds correctly but not in time
  (early or late)
• response failure
• a server responds incorrectly
• value failure incorrect output
• state transition failure incorrect state
  transition
• crash failure
• a server repeatedly fails to respond to inputs
• failure management depends on server state at
  restart
• amnesia crash no record of state at crash reset
  to initial state
• partial amnesia crash partially recorded
• pause crash restart in state before crash
• halting crash no restart

5
Introduction (cont)

• Failure semantics
• description of the ways in which a service may
  fail
• recovery actions depends on the likely failure
  behavior of the server when its failure is
  detected
• designer should ensure that the behavior of the
  server conforms to a specified failure semantics
• e.g. network with omission/time failure semantics
• need to guarantee detection of message corruption
  such as checksum
• stronger failure semantics costs more in general
• adequacy of failure semantics would require
  preliminary stochastic analyses

6
Failure Model

• Representative faulty behavior
• Byzantine failures
• system exhibits arbitrary and malicious behavior
  which may collude with other systems
• fail-stop failures
• when system fails, it changes to a state that
  allows others to detect its failure and then stops

7
Fault-Tolerant Approaches

• Fault tolerance
• can detect a fault and either fail predictably or
  mask the fault from users
• hiding the occurrence of errors in system
  components and communications
• incorporate redundant processing component to
  achieve fault tolerance
• k-resilient/fault-tolerant
• a set of systems satisfies its specification if
  no more than k systems become faulty
• k is chosen based on statistical measures of
  system reliability
• Byzantine failure 2k1
• fail-stop failure k1

8
Fault-Tolerant Approaches (cont)

• Two approaches to support fault tolerance (fault
  masking)
• hierarchical masking
• hierarchical failure and recovery management
• error detection in layered communication
  protocols
• various levels of error abstraction in OS
• group failure masking
• state-machine approach
• primary-backup approach
• Fault tolerance support can be done
• Hardware
• stable storage
• Software
• replicated servers

9
Stable Storage

• Group masking at disk block level
• duplicate a careful storage service
• has only omission failure semantics
• corruption errors are masked by converting them
  to omission failures
• stable block
• unit of storage
• represented by two careful disk blocks containing
  the same data
• operation semantics
• operation invariant
• no more than one of the pair is bad
• the pair has most recent data except during write
• read one of pair
• write one by one with guarantee of success of
  first write
• recovery good block is copied to bad one

10
State-Machine Approach

• State machine specification
• state variables commands
• state variables
• encode states of state machine
• Commands
• implemented by deterministic program
• execution of command is atomic with respect to
  other commands
• modify state variables and/or produce some output
• assumptions on ordering of requests made by
  clients
• O1 requests issued by a single client to a state
  machine sm are processed by sm in the order they
  were issued
• O2 if request r issued by client c could have
  caused a request r issued by client c, then
  state machine processes r before r

11
State-Machine Approach (cont)

• Fault-tolerance state machine
• Assumptions
• k fault-tolerant state machine can be implemented
  by replicating state machine and running a
  replica on each of processors in a distributed
  system
• if all replicas start in the same initial state
  and execute the same requests in the same order,
  each replica will do the same task and produce
  the same result
• if each failure can affect at most one replica,
  result for k fault-tolerant state machine is
  obtained by combining result of replicas
• degree of replication
• Byzantine failure 2k 1
• failstop failure k 1

12
State-Machine Approach (cont)

• Requirements for k fault-tolerant state machine
• all replicas receive and process the same
  sequence of request
• agreement every non-faulty replica receives
  every request
• specify interaction behavior of a client with
  state machine replicas
• relaxed for read-only request in failstop failure
• order every non-faulty replica processes
  requests it receives in the same relative order
• specify behavior of state machine replicas in
  term of how to process requests from clients
• relaxed for commutative requests

13
State-Machine Approach (cont)

• Agreement requirement
• to satisfy agreement requirement, state-machines
  should support a message broadcasting protocol
  which conforms to
• IC1 all non-faulty processors agree on the same
  value
• IC2 if sender of request is non-faulty, then all
  non-faulty processors use its value as the one on
  which they agree
• message broadcasting protocol is called Byzantine
  agreement protocol or reliable broadcast protocol

14
State-Machine Approach (cont)

• Order requirement
• to implement order requirement requires
• assignment of unique identifier to each message
• stability (a request is ready to be delivered
  once all the previous requests have been
  delivered) test
• logical clock-based
• only for failstop failures
• unique id assignment logical clock
• LC1 timestamp is incremented after each event at
  p
• LC2 upon receipt of a message with timestamp t,
  process p resets its timestamp Tp to max(Tp, t)1
• stability test
• a request is stable at replica smi if a request
  with larger timestamp has been received by smi
  from every client running on a non-faulty
  processor
• messages between a pair of processors are
  delivered in the order sent
• processor p detects that a failstop process q has
  failed only after p has received qs last message
  sent to p

15
Primary-Backup Approach

• Primary-backup requirements
• Pb1 there is at most one server whose state
  satisfies a condition being a primary
• Pb2 each client maintains a server identity to
  which client can send a message
• Pb3 if a client request arrives at a server that
  is not a primary, then that request is not
  enqueued
• Pb4 there exist fixed value k and d such that
  the service behaves such that all server failures
  can be grouped into at most k intervals of time
  with each interval having length at most d

16
Primary-Backup Approach (cont)

• Simple primary-backup protocol
• assumption
• one primary server p1 and one backup server p2
• operations when p1 receives a request
• processes the request and updates its state
• send update info to p2 (state update message)
• send response to client without waiting for ack
  from p2
• spec conformance
• Pb1 (p1 has not crashed) (p2 has not received
  a message from p1 for Td)false
• Pb2 client sends a message to p1
• Pb3 requests are not sent to p2 until after p1
  has failed
• Pb4 (1, T4d)-bofo server