Chapter 16 Recovery and Fault Tolerance

1
Chapter 16 Recovery and Fault Tolerance

• Introduction
• Transaction recovery
• Fault tolerance
• Hierarchical and group masking of faults
• Summary

2
16.4 Hierarchical and group masking of faults

• there are two common approaches to masking
  faults
• - hierarchical failure masking and group
  failure masking
• Hierarchical masking
• the server at the higher level attempts to mask
  the faults at the lower level
• - the faults can be entirely hidden
• e.g. request/reply protocol can mask omission
  failure by retransmitting messages
• - when a lower-level failure cant be masked,
  it is converted to a higher
• level exception
• e.g. request/reply protocol can mask a
  servers crash failure by reporting
• an exception to its client

• Group failure masking
• a service can be made fault tolerant by
  implementing it as a group of servers
• each of which runs on a different computer
• - group masking hides the failure of
  individual members by a group
• mechanism
• it can hide performance failure
• it can hide value failure

3

• to ensure that each member of a group of
  servers can provide an equivalent
• service, each member starts in the same
  initial state and executes the same
• requests in the same order
• - there are two requirements on the servers
  program
• the program must be deterministic
• each operation must be atomic with respect to
  all the other operations

• Closely synchronized group
• all the members execute all of the requests
  immediately after receiving them
• - a fault-tolerant state machine can be
  implemented as a closely
• synchronized group of servers
• an idealized form of server program whose
  operations are defined
• to be both deterministic and atomic
• executes requests according to causal ordering
• - requirements for the state machine can be
  relaxed
• read-only requests need not be executed by all
  the members
• the number of replies needed by the client
  depends on the failure
• semantics of the servers
• can be further relaxed if the operations of the
  servers are commutative
• - approaches to the implementation of a
  fault-tolerant state machine
• atomic multicast ensures that all the members
  receive the same requests
• totally-ordered atomic multicast ensures that
  they receive them in same order
• causally ordered multicast can be used to
  satisfy the requirement for causality

4

• loosely synchronized group
• loosely synchronized group may be an attractive
  alternative in some environment
• - the primary server is used so long as it
  performs correctly
• - and other backup(or stand-by) servers are
  available to take over
• when it fails
• the primary server deals with all the client
  requests by executing
• them and returning replies
• one or more backup servers log the requests
• if the primary fails, one of the backup servers
  executes the requests
• in its log and then take over
• the primary occasionally sends a checkpoint of
  its state to backups
• - a loosely synchronized group uses less of
  the available computing resources
• - but recovery takes longer for a loosely
  synchronized group
• - and would not be acceptable for
  applications requiring a guaranteed
• response time
• - primary server/stand-by server arrangement
  cant be used for
• Byzantine failures
• - group masking of faults can be done at the
  hardware level as well as software
• e.g. Tandem system

5

• Stable storage
• an example of group masking at the disk block
  level
• - to ensure that any essential permanent
  data will be recoverable
• - uses group masking by duplicating a
  careful storage service
• a careful storage service has only omission
  failure semantics
• Read operation uses checksum to convert value
  failures to omission failures
• each stable block is represented by two careful
  disk blocks
• Read operation reads one of the pair of stable
  blocks
• enable stable storage Read operation to mask
  omission failures
• - a stable storage service guarantees that
  the following invariant is maintained
• Not more than one of the pair of blocks is bad
• if both are good, they both have the most
  recent data, except during
• the execution of a Write operation
• - when a stable storage server is restarted
  and the recovery procedure
• is invoked, each pair of stable blocks
  will be in one of the following states
• both good and the same
• both good and different
• one good, one bad
• - the recovery does the following in each
  the above cases

6

• Primary and backup servers
• primary/backup servers arrangement can provide
  fault-tolerance for
• most applications which can tolerate some
  delay during recovery
• - backup server is relatively inactive during
  the normal operation of
• the primary
• - enables backup server computer to be used
  for other productive work
• - example Auragen, a fault tolerant version
  of a distributed UNIX
• designed for a transaction-processing
  environment
• checkpointing is relatively infrequent and
  achieved transparently
• each primary has a backup on a different
  computer
• when the primary fails, the backup reads in the
  checkpoint
• and executes the same messages that were
  executed by the primary
• when it catches up it takes over as primary
• a request message from a primary to another is
  always sent to three
• destinations the senders backup, the
  receiving primary and its backup

B
A
B
A
7

• each request is a three-way totally ordered
  atomic multicast
• each server performs an appropriate action on
  receipt of a message
• Role Action
• Primary Execute the operation requested and
  return a reply
• Backup Save the message in a log for
  subsequent use in recovery
• Senders Count the number of messages since
  the last checkpoint by
• backup the sender

• a primary and backup are synchronized from time
  to time to prevent
• the backup from getting too far behind the
  primary
• - the primary performs a checkpoint by saving
  the state of its data items
• - the primary informs its backup that it has
  done the checkpoint
• - whereupon the backup deletes the messages
  in its log
• - and sets the count of messages sent by the
  primary to zero

• synchronization is done automatically
• - at specified time intervals
• - and whenever the backups message log
  exceeds some maximum length

8
Chapter 17 Security

• Cryptography
• Authentication and key distribution
• Digital signatures
• Case study Kerberos
• Logics of authentication

9
17.1 Introduction
Threats

• some threats to security in distributed systems
  are obvious
• - leakage the acquisition of information by
  unauthorized recipients
• - tampering the unauthorized alteration of
  information(including programs)
• - resource stealing the use of facilities
  without authorization
• - vandalism interference with the proper
  operation of a system without
• gain to the perpetrator

Methods of attack

• The methods by which security violations can be
  perpetrated include
• - eavesdropping obtaining copies of
  messages without authority
• - masquerading sending or receiving
  messages using the identity of
• another principal without their authority
• - message tampering intercepting messages
  and altering their contents
• before passing them on to the intended
  recipient
• - replaying storing messages and sending
  them at a later date

10
Scenarios
Server
(a) Client
Request message
Reply message
Server
(b)
(c) Client
11
Security requirements for client-server systems

• security policies must be adopted to guard
  against the above threats to
• the security in distributed systems
• - a set of rules or decisions derived from
  an individuals or organizations
• attitude toward security
• - to ensure appropriate levels of security
  for the activities that are
• performed in the system
• secure the channels of communication used, to
  avoid eavesdropping
• design clients and servers to view each other
  with mutual suspicion
• - Its a vice to trust all, and equally
  a vice to trust none
• ensure that communication is fresh in order to
  avoid security
• violations through the replay of
  messages
• appropriate security mechanisms can be employed
  for enforcing the
• security polices
• - based on the use of the following
  techniques
• cryptography functions
• authentication protocol
• access control mechanism

12
Security services
- Privacy preventing unauthorized release
of information - Authentication verifying
identity of the remote participant -
Integrity making sure message has not been
altered
security
Security Security mechanism services
Privacy Authentication Message
integrity
Cryptography algorithms
Authentication protocol (e.g., Kerberos)
Access control mechanism (e.g., access control
list in UNIX)
Secret Public Message key
key digest (e.g., DES) (e.g., RSA)
(e.g., MD5)
13
Cryptography

• encryption of messages plays three major roles
  in the implementation
• of secure distributed systems
• - used to conceal private information where
  its exposed in parts of the system
• - used in support of mechanisms for
  authenticating communication between
• pairs of principals
• - used to implement a mechanism known as a
  digital signature

Authentication mechanism

• the means by which the identities of servers
  and clients are reliably established
• - centralized systems password check at the
  start of each interactive check
• - distributed system based on the
  possession of encryption keys
• a principal who possesses the appropriate
  secret encryption key has the
• identity that it claims
• - authentication mechanisms for distributed
  systems takes the form of an
• authentication service
• require a key distribution service generating,
  storing and distributed all of
• the encryption keys
• rely on the use of encryption to guarantee
  security

14
Access control mechanism

• ensuring that access to resources is available
  only to that subset of
• users that are currently authorized to do so
• - access control mechanism occur in
  non-distributed systems and distributed
• systems
• - to allow each user to maintain some
  private files and to share them
• in a controlled manner
• - e.g., UNIXs access control list for
  restricting access to individual files

15
17.2 Cryptography

• applying some rule to encrypt a message
• - the rule transforms the message from plain
  text to cipher text
• - it cant be understood by anyone except the
  intended recipient
• - the transformations are defined with two
  parts a function and a key
• the function defines an encryption algorithm
• its result is heavily dependent on the value of
  the key
• - MK a text M encrypted with an agreed
  encryption function and a key K
• - the key-based scheme relies on the secure
  distribution and storage of keys
• key distribution service
• - two main classes of encryption techniques
  secret-key public-key

Secret-key encryption

• a message is encrypted by applying an agreed
  encryption function to
• the plain text with a secret key
• - decryption is achieved by applying the
  inverse function to the cipher
• text using the same key
• - the encryption and decryption functions
  need not be secret

16
- both sender and recipient must possess the
encryption function(or its inverse) and a
shared key
Sender A Recipient B
1. Acquire K 2. receive Mk 3. F-1(K,Mk)--gtM
1. Acquire K 2. F(K,M)--gtMk 3. Send Mk
Mk

• a key size and encryption function must be
  selected with care
• - secure against attempts to break it
• attacks designed to discover M givenMk
• or to discover K given any specific message M
  and its encrypt form MK
• so larger keys and more complex encryption
  functions must be used
• - Data Encryption Standard(DES) encryption
  algorithm is the most
• widely-used secret-key encryption method

17
Secret-key (DES)

• developed by IBM and subsequently adopted as a
  US national standard
• 64-bit key(56-bits 8-bit parity)
• - its encryption function maps a 64-bit plain
  text into a 64-bit encrypted
• output using a 56-bit key
• 19 stages
• - 3 key-independent transpositions
• 1st stage key-independent transpositions on the
  64-bit plain text
• last stage the exact inverse of the
  transposition in 1st stage
• stage prior to the last exchanges the
  left-most 32 bits with the
• right- most 32 bits
• - 16 key-dependent stages(or 16 rounds)
• each stage takes two 32-bit inputs and produces
  two 32-bit outputs
• the left output is simply a copy of the right
  input
• the right output is the bitwise XOR of the left
  input and a function
• of the right input and the key for this stage
• in each of the 16 stages, a different key is
  used
• the function and the generation of the key in
  each stage not shown
• see Tanenbaum ltComputer Networksgt,
  third edition

18
16 rounds Each round
Ki
F

• decryption with the same key run the steps in
  the reverse order
• therere no known instances of successful
  attacks against DES

19
Public-key encryption

• eliminates the need for trust between the
  communication parties
• - avoids the need for the transmission of
  secret keys between principals
• based on the difficulty of finding factors of
  large numbers
• - determination of factors of large number is
  very computationally intensive
• this method is based on two separate well-known
  functions, E and D
• and two separate keys Ke and Kd for encryption
  and decryption
• - each potential recipient of a message makes
  a pair of keys, Ke and Kd
• keeps the decryption key Kd as secret
• Ke can be made known publicly for use by anyone
  who wants to communicate
• - its based on the use of a one-way function
  to define the relation
• between the two keys
• for one-way function f(X) Y its hard to
  determine X from Y

Sender A Recipient B
Request Ke
Public key database
Ke
Ke
1. Compute Ke,Kd 2. Publish Ke 3. receive
Mke 4. D(Kd,Mke)--gtM
1. Acquire Ke 2. E(Ke,M)--gtMke 3. Send Mke
Mke
20
Public-key (RSA)

• designed by Rivest, Shamir and Adelman, and
  widely used
• no flaws have been found in it, despite
  extensive investigations
• how to find a key pair e,d
• - choose two large prime numbers, P and
  Q(each greater than 10100), and
• form NP Q and Z(P-1) (Q-1)
• - For d choose any number that is relatively
  prime with Z
• two numbers are relatively prime if they have
  no common factor
• greater than one
• - To find e solve the equation
• e d 1 mod Z
• That is, e d is the smallest element in the
  series Z1, 2Z1, 3Z1...
• that is divisible by d
• - the plain text is divided into equal blocks
  of length k bits (2kltN)
• the function for encrypting a single block of
  plain text M
• E(e,N,M) Me mod N
• the function for decrypting a block of
  encrypted text c
• D(d,N,c) cd mod N

21

• functions E and D are mutual inverses for all
  values of P (0lt Plt N)
• E(D(x)) D(E(x)) x
• Ke lte,Ngt and Kd ltd,Ngt
• the publication of lte,Ngt doesnt compromise the
  secrecy of d
• - any attempt to determine d requires
  knowledge of original prime numbers P and Q
• - these can only be obtained by the
  factorization of N
• - its time-consuming

Key distribution

• for the secret-key
• - both sender and receiver must possess a
  shared secret key
• - and each must be convinced that the key is a
  secret known only to the other
• - authentication servers can be used to
  distribute the secret keys

• for public-key
• - the recipient of public keys must be sure
  that the keys are authentic
• - this can be met in two ways
• using a key distribution server
• using conventional database methods, but each
  key must be
• distributed with a certificate

22
Comparison of secret- and public-key cryptography

• security
• - both are secure with suitable keys and
  encryption algorithms
• convenience
• - public-key encryption can be more
  convenient to implement
• performance
• - secret-key encryption algorithms are much
  faster
• RSA public-key encrypt 0.5 103(bits per
  second)
• decrypt 32 103(bits per second) DES
  secret-key encrypt/decrypt 400 103 (bits per
  second)

Current practice

• the use of cryptography at the application
  level
• - PEM (Privacy Enhanced Mail)
• uses both public and secret-key encryption
• mail users publish their public-keys with
  their mail addresses
• PEM program generates a one-shot secret key to
  encrypt a sending
• message using DES algorithm
• the secret key is encrypted in recipients
  public key and appended
• to the encrypted message
• - PGP (Pretty Good Privacy)

23
17.3 Authentication and key distribution

• authentication and secure distribution of keys
  are best provided in a
• single service
• - based on an authentication server that
  supplies secret keys to clients
• - Needham and Schroeders model does so
• - the model describe two protocols
• the first using secret keys
• the second using public keys
• - Kerberos authentication service used Needham
  and Schroeder with secret keys

Needham and Schroeder with secret keys

• the authentication server S maintains a table
  containing a name and a
• secret key for each principal known to the
  system
• - the secret key is used only to authenticate
  client processes and the
• authentication server
• - and used to transmit message securely
  between client processes and
• authentication server
• - a secret key is the equivalent of the
  password used to authentication
• users in centralized systems

24
The Needham - Schroeder secret-key authentication
protocol
Header Message Notes
1. A--gtS A,B,NA A requests S to
supply a key for communication with B 2. S--gtA
NA,B,KAB, S returns a message encrypted
in As secret key containing
KAB,AKBKA a newly-generated key KAB, and a
ticket encrypted in Bs
secret key.
The nonce NA demonstrates that the message was

sent in response to the preceding one 3.
A--gtB KAB,AKB A sends the ticket
to B 4. B--gtA NBKAB B decrypts the
ticket and uses the new key KAB to
encrypt
another nonce NB 5. A--gtB NB-1KAB
A demonstrates to B that it was the sender of the
previous

message by returning an agreed
transformation of NB
A Name of principal for the process
initiating communication B Name of principal for
As communication partner KA As secret key KB
Bs secret key KAB Secret key for communication
between A and B NA A nonce generated by
A MK Message M encrypted in key K
25
Needham and Schroeder with public keys

• public keys must be distributed by a trusted
  key distribution server

PKA As public key PKB Bs public
key PKS Servers public key SKS Servers secret
key
26
17.4 Digital signatures

• used to enable the recipient to verify that
• - the claimed originator of a document is
  the real originator
• - that the document has not subsequently
  been altered
• - and the originator will not be able to
  repudiate the document subsequently
• Notation
• - message M can be signed by a principal A
  by encrypting a copy
• of M in a key KA and attaching it to a
  plain-text copy of M and As identifier
• signed document ltM, A,MKAgt

• to reduce the size of digital signature for
  large documents, a digest
• function D is used to produce a
  characteristic value- message digest
• - e.g. MD5 used in secure mail

Message Digest

• Cryptographic checksum
• - just as a regular checksum protects the
  receiver from accidental
• changes to the message
• - a cryptographic checksum protects the
  receiver from malicious
• changes to the message

27
Message Digest

• One-way function
• - given a cryptographic checksum for a
  message, its virtually
• impossible to figure out what message
  produced that checksum
• - its not computationally feasible to find
  two messages that hash to
• the same cryptographic checksum

• Relevance
• - if you are given a checksum for a message,
  and you are able to
• compute exactly the same checksum for that
  message
• - then its highly likely that this message
  produced the checksum you
• were given

Digital signatures with public keys
Header Message Notes
1. A--gtB M, A A sends the original
message and the signature to B
D(M)KAprivate 2. B--gtS A
B requests As public key from S 3. S--gtB
A,KApublic S supplies As public key . B
uses it to decrypt the
signature received in
Message 1 and compare it with a
newly-computed
value for D(M)
28
Digital signatures with secret keys

• a principal A sends a signed message M to
  another principal B using secret keys

Header Message Notes
1. A--gtS A,D(M) KA A computes D(M),
the digest of the message and
encrypts D(M)with
As secret key and sends it to the
authentication
server. 2. S--gtA A, D(M),t KS The
Server makes a signed and dated certification of
As signature on the document. It
does this by making a
composite text
containing As name, D(M) and a timestamp

t and encrypts it with its secret key. It sends
the resulting
certificate back to A 3.
A--gtB M,A,D(M),tKS A sends the original
message and the certificate in a
message to B 4. B--gtS B,A,D(M),tKS B
saves a copy of the message and the certificate
and
then sends the certificate to the
authentication server
for decrypting 5. S--gtB
A,D(M),tKB The server decrypts the
certificate. It then uses Bs secret key

to encrypt the resulting plain text and sends it
to B, where it
is decrypted.
29
17.4 Case studies Kerberos

• an authentication protocol based on Needham
  and Schroeder secret-
• key protocol
• - developed and widely used at MIT
• - both DCE and AFS 3.0 include an
  integrated implementation

Architecture
Kerberos Key Distribution Center
1. Request for TGS ticket 2. TGS ticket
Authentication database
authen- tication service A
Ticket- granting service T
Client C
Server S
3. Request for server ticket 4.
Server ticket
Login session setup Server session
setup DoOperation
5. Service request
Server function
6. Service reply
30

• Kerberos deals with three kinds of security
  objects
• - Session key a secret key randomly
  generated by Kerberos and issued
• to a client for use when communicating with
  a particular server
• a session key between client C and server S KCS
• - Ticket used to verify that the sender has
  been recently authenticated
• by Kerberos
• has a fixed period of validity starting at time
  t1 and ending at time t2
• a ticket for client C to access server S
  C,S,t1,t2,KCSKS gt ticket(C,S)KS
• - Authenticator used to prove the identity
  of the sender and the currency
• of any communication with a server
• an authenticator constructed by client C to be
  authenticated by server S
• C,tKCS gt auth(C)KCS

• Needham and Schroeder secret-key protocol is
  followed quite closely, Why?
• - both use encrypted ticket
• the receiver can use it to figure out the
  session key
• - both use nonces
• to guard against replay of old message
  intercepted in the network
• and the reuse of old tickets found
  lying in the machine memory
• to apply a lifetime to tickets, enabling the
  system to revoke users rights

31
Protocol

• Notation
• A Name of Kerberos authentication service
• T Name of Kerberos ticket granting service
• C Name of client
• n A nonce
• t A timestamp
• t1 Starting time for validity of ticket
• t2 Ending time for validity of ticket

• Login session setup

Header Message Notes
1. C--gtA C, T, n Client C requests
Kerberos authentication server A to
supply a
ticket for communication with the ticket
granting service T 2. A--gtC KCT,nKC,
A returns a message containing a ticket
encrypted in Ts
ticket(C,T)KT secret key and a session key
for C to use with T. The
containing inclusion of the nonce n
encrypted in KC shows that
C,T,t1,t2,KCT the message comes from the
recipient of Message 1, who must
know KC
32

• Server session setup

Header Message Notes
3. C--gtT auth(C)KCT, C requests
the ticket-granting server T to supply a
ticket(C,T)KT,S,n ticket for
communication with another server S
4. T--gtC KCS,nKCT, T
checks the ticket. If it is valid T generates a
new ticket(C,S)KS
random session key KCS and returns it with a
containing ticket
for S(encrypted in the servers secret key
C,S,t1,t2,KCS KS)

• Request service

Header Message Notes
5. C--gtS auth(C)KCS, Client C sends
the ticket to S with a newly-generated
ticket(C,S)KS, authenticator for C and a
request. The request would be
request,n encrypted in KCS if secrecy of
the data is required 6. S--gtC nKCS
(optional) S sends the nonce to C, encrypted in
KCS.
33
Diagrammatic view of the Kerberos protocol
obtain Kerberos session key , once per login
session 1. C--gtA C, T, n 2. A--gtC
TGS session key,n ticket(C,T) obtain
server ticket , once per client-server session
3. C--gtT auth(C) ticket(C,T)
server, n 4. T--gtC CS session key, n
ticket(C,S) issue server request 5.
C--gtS auth(C) ticket(C,S) Service
request, n authenticate server(optional) 6.
S--gtC n response
Encryption keys
KCT TGS session key
KCS session key
KC Cs private key
KT Ts private key
KS Servers private key
34
Login with Kerberos

• a user logs in to a workstation
• - login program sends users name to the
  Kerberos authentication service
• - the latter replies with a session key and
  a nonce encrypted in the
• users password and a ticket for the TGS
• - the login program prompts the user to type
  his password
• - and decrypts the session key and the nonce
• - it checks the nonce and stores the session
  key with the ticket
• - a login session is then started

Accessing typical servers with Kerberos

• a user needs to access a new service, e.g., to
  login to a remote computer
• - rlogin program obtains a ticket from the
  Kerberos ticket-granting
• service for access to the rlogind network
  service
• - it sends the ticket with a new
  authenticator to rlogind process
• - rlogind program decrypts the ticket with
  the rlogin services secret key
• - and checks the validity of the ticket
• - rlogind program then uses the session key
  to decrypt the authenticator
• - and checks that the authenticator is
  fresh(can be used only once)
• - a remote login session is established for
  that user on the remote machine

35
Implementation of Kerberos

• Kerberos is implemented as a server that runs
  on a secure machine
• - uses the DES encryption algorithm
• - maintains authentication databases
• Users passwords are known only to the user and
  the Kerberos
• authentication servers
• Services have secret keys that are known only
  to Kerberos ticket-
• granting servers and the servers that
  provide the services

• to make Kerberos service scalable
• - the world is divided into separate domains
  of authentication authority,
• called realms
• - each with its own Kerberos server
• - Within a single realm, there can be several
  authentication servers
• all of which have copies of the same
  authentication database
• the authentication database is replicated by a
  simple master-slave
• technique
• - most principals are registered in just one
  realm
• - but the Kerberos ticket-granting servers
  are registered in all of the realms

• Kerberos ticket-granting service is introduced
  separately, why?
• - More safe - More efficient and
  friendly - more scalable and flexible

36
Thats the whole lecture, thanks for your
attention!