An Efficient Lookup Algorithm in Large Scale GRID Environment

1
An Efficient Lookup Algorithm in Large Scale
GRID Environment
2002. 7. 12.
Computer Software Laboratory, ETRI
Han Namgoong, Incheol Jeong, Dukjoo Son nghan,
jic_at_etri.re.kr
2
1. Background

• World of Contents
• More than 106 scattered same digital contents
• music files, web pages, selling items,
• More than 108 internet connected users
• every internet applications use lookup service
• (ex) DNS(domain name server), router table etc.
• How do you find the data efficiently?
• less than 1 sec. response time?

3
2. Problem
Source Robert Morris, MIT
Lookup is very important - no control of
join/leave
4
3. Related Works

• Centralized lookup (Napster)

Source Robert Morris, MIT
O(N) state and a single point of failure - should
register/query to central server
5
3. Related Works

• Flooded queries (Gnutella)

Source Robert Morris, MIT
worst case O(N) messages per lookup
6
3. Related Works

• Routed queries(Freenet and Chord)

Source Robert Morris, MIT
Chord emphasizes efficiency and simplicity
- Efficient O(log(N)) messages per
lookup - Scalable O(log(N)) state
per node
7
3. Related Works (Chord)

• Key identifier SHA-1(key)
• Node identifier SHA-1(IP address)
• Both are uniformly distributed and exist in the
  same ID space

How to map key IDs to node IDs?
8
3. Related Works (Chord)
Lookup(my-id, key-id) - Basic n my
successor if my-id lt n lt key-id call
Lookup(id) on node n // next hop else return
my successor // done Correctness depends only
on successors
9
3. Related Works (Chord)

• Finger table allows log(N)-time lookups

Finger i points to successor of n2i
10
3. Related Works (Chord)

• finger table of 3 nodes and interval

11
3. Related Works (Chord)
Lookup(my-id, key-id) look in local finger
table for highest node n s.t. my-id
lt n lt key-id if n exists call Lookup(id)
on node n // next hop else return my
successor // done
O(log(n)) complexity
12
3. Related Works (Chord)

• What is happening if nodes fail?

Failures might cause incorrect lookup
13
3. Related Works (Chord)

• solution of multiple nodes failures
• build and keep successor list
• Each node knows r immediate successors
• After failure, will know first live successor
• Correct successors guarantee correct lookups
• Guarantee is with some probability

• Lookup(my-id, key-id)
• look in local finger table and successor-list
• for highest node n s.t. my-id lt n lt key-id
• if n exists
• call Lookup(id) on node n // next hop
• if call failed,
• remove n from finger table
• return Lookup(my-id, key-id)
• else return my successor // done

14
4. Not Covered Problems in Chord

• multiple key values for GRID environment
• available resources CPU, disk space, printer,
  etc.
• different attributes of available resources
• amount(CPU allowed time, disk space,..)
• start/end time of availability of resources
• .
• (read only)data availability?
• only one node fails we wait the recovery of that
  node
• strictly sequential access ? low performance
• network partition
• partition occurs in the right before and just
  after the successor list

15
5. Extension of Chord

• multiple key values for GRID environment
• available resources CPU, disk space, printer,
  etc.
• different attributes
• amount(CPU allowed time, disk space,..)
• start/end time of availability of resources

• multiple signature keys depending on resources
  types
• key identifier SHA-1(multiple signature keys)
• node identifier SHA-1(IP address)
• both are uniformly distributed and exist in the
  same ID space

16
5. Extension of Chord

• (read only)data availability?
• only one node fails we wait the recovery of that
  node
• strictly sequential access ? low performance
• network partition
• partition occurs in the right before and just
  after the successor list

?
?
17
5. Extension of Chord

• solution ? make replica
• evenly distribution in the same ID space
• ? in 2 replica case expose the key to the n,
  n(1/3)n, n(2/3)n

Key 5
K5
Node 110
N110
K20
K80
replica 1
K80
replica 2
Circular 7-bit ID space
N32
N90
K80
1. high rate of data availability ? 2 replica
one node failure probability 0.05 ? 2 replica
provides 99.98 availability 2. network partition
? 2 replica we get unavailability of desired
data only when more than 2/3 nodes fails
18
5. Extension of Chord

• penalty by this extension
• constant times of existing Chord algorithm
• look up cost(worst case) O(log(n) k)
  O(log(n))
• join cost O(log(n) k) O(log(n))
• leave cost O(log(n) k) O(log(n))
• route info. in each node O(log(n) k)
  O(log(n))
• ? k is the number of replica

19
6. Conclusion

• same performance as Chords
• Efficient O(log(N)) messages per lookup
• N is the total number of nodes
• Scalable O(log(N)) state per node
• Robust survives massive failures
• solve network partition
• provide high availability
• primarily depends on the number of replica
• not covered by our approach
• practical deployment and experiment
• use of locality and(or) pattern of data

20

• Thank You