Symmetric Replication in Structured PeertoPeer Systems

1
Symmetric Replication in Structured
Peer-to-Peer Systems

• Ali Ghodsi, Luc Onana Alima, Seif Haridi

2
Scope of the Talk

• Where to place replicas in a DHT?

3
Background

• DKS System (http//dks.sics.se/)
• Were initially replicating on successor-lists
  (Similar to Chord/Koorde)
• Turned out to have several problems

4
Outline

• Background
• Proposed scheme symmetric replication
• Extensions
• Related Work Summary

5
What is Replication on the Successor-List?

• In one sentence
• Replicate node ns data to its f closest
  neighbors for replication degree f

6
Successor-list Example Chord

• A Chord-alike system, with nodes 0, 3, 5, 10, 11

0
15
1
2
14
13
3
12
4
11
5
6
10
7
9
8
7
Successor-list Example Chord

• With a replication factor of f1

Data 12, 13, 14, 15, 0
0
15
1
2
14
Data 1, 2, 3
13
3
12
4
Data 11
Data 4, 5
11
5
6
10
7
9
8
Data 6, 7, 8, 9, 10
8
Successor-list Example Chord

• With a replication factor of f2

Node 11 11
Data 12, 13, 14, 15, 0
0
15
1
Node 0 12, 13, 14, 15, 0
2
14
Data 1, 2, 3
13
3
Node 10 6, 7, 8, 9, 10
12
4
Data 11
Node 3 1, 2, 3
Data 4, 5
11
5
6
10
Node 5 4, 5
7
9
8
Data 6, 7, 8, 9, 10
9
Successor-list Example Chord

• With a replication factor of f3

Node 10 6, 7, 8, 9, 10
Node 11 11
Data 12, 13, 14, 15, 0
Node 11 11
0
15
1
Node 0 12, 13, 14, 15, 0
2
14
Data 1, 2, 3
13
3
Node 5 4, 5
Node 10 6, 7, 8, 9, 10
12
4
Node 0 12, 13, 14, 15, 0
Data 11
Node 3 1, 2, 3
Data 4, 5
11
5
Node 3 1, 2, 3
6
10
Node 5 4, 5
7
9
8
Data 6, 7, 8, 9, 10
10
Successor-list Example Chord

• With a replication factor of f4

Node 5 4, 5
Node 10 6, 7, 8, 9, 10
Node 11 11
Data 12, 13, 14, 15, 0
Node 10 6, 7, 8, 9, 10
Node 11 11
0
15
1
Node 0 12, 13, 14, 15, 0
2
14
Data 1, 2, 3
13
3
Node 3 1, 2, 3
Node 5 4, 5
Node 11 11
Node 10 6, 7, 8, 9, 10
12
4
Node 0 12, 13, 14, 15, 0
Data 11
Node 3 1, 2, 3
Data 4, 5
11
5
Node 0 12, 13, 14, 15, 0
Node 3 1, 2, 3
6
10
Node 5 4, 5
7
9
8
Data 6, 7, 8, 9, 10
11
Problem Dynamism

• What happens in case of leaves or failures?
• f4

Node 5 4, 5
Node 10 6, 7, 8, 9, 10
Node 11 11
Data 12, 13, 14, 15, 0
Node 10 6, 7, 8, 9, 10
Node 11 11
0
15
1
Node 0 12, 13, 14, 15, 0
2
14
Data 1, 2, 3
13
3
Node 3 1, 2, 3
Node 5 4, 5
Node 11 11
Node 10 6, 7, 8, 9, 10
12
4
Node 0 12, 13, 14, 15, 0
Data 11, 10, 9, 8, 7, 6
Node 3 1, 2, 3
Data 4, 5
11
5
6
10
7
9
8
12
Problem Dynamism

• What happens in case of leaves or failures?
• f4

Node 5 4, 5
Node 11 11,10,9,8,7,6
Data 12, 13, 14, 15, 0
Node 11 11,10, 9, 8, 7, 6
0
15
1
Node 0 12, 13, 14, 15, 0
2
14
Data 1, 2, 3
13
3
Node 3 1, 2, 3
Node 11 11
Node 5 4, 5
12
4
Node 0 12, 13, 14, 15, 0
Data 11, 10, 9, 8, 7, 6
Node 3 1, 2, 3
Data 4, 5
11
5
6
10
7
9
8
13
Problem Dynamism

• Need to move items to restore replication degree!

Node 5 4, 5
Node 11 11,10,9,8,7,6
Data 12, 13, 14, 15, 0
Node 5 4, 5
Node 11 11,10, 9, 8, 7, 6
0
15
1
Node 0 12, 13, 14, 15, 0
2
14
Data 1, 2, 3
Get items
13
3
Node 3 1, 2, 3
Node 11 11,10, 9, 8, 7, 6
Data 4, 5
Node 5 4, 5
12
4
Node 0 12, 13, 14, 15, 0
Data 11, 10, 9, 8, 7, 6
Node 3 1, 2, 3
Data 4, 5
11
5
6
10
7
9
8
14
Problem Dynamism

• Need to move items to restore replication degree!

Node 3 1, 2, 3
Node 5 4, 5
Node 11 11,10,9,8,7,6
Data 12, 13, 14, 15, 0
Node 5 4, 5
Node 11 11,10, 9, 8, 7, 6
0
15
1
Node 0 12, 13, 14, 15, 0
2
14
Data 1, 2, 3
Data 1, 2, 3
13
3
Get items
Node 3 1, 2, 3
Node 11 11,10, 9, 8, 7, 6
Node 5 4, 5
12
4
Node 0 12, 13, 14, 15, 0
Data 11, 10, 9, 8, 7, 6
Node 3 1, 2, 3
Data 4, 5
11
5
6
10
7
9
8
15
Problem Dynamism

• Need to move items to restore replication degree!

Node 3 1, 2, 3
Node 5 4, 5
Node 11 11,10,9,8,7,6
Data 12, 13, 14, 15, 0
Node 5 4, 5
Node 11 11,10, 9, 8, 7, 6
0
15
1
Node 0 12, 13, 14, 15, 0
2
14
Data 1, 2, 3
13
3
Node 0 12,13,14,15,0
Node 3 1, 2, 3
Data 12, 13, 14, 15, 0
Node 11 11,10, 9, 8, 7, 6
Get items
Node 5 4, 5
12
4
Node 0 12, 13, 14, 15, 0
Data 11, 10, 9, 8, 7, 6
Node 3 1, 2, 3
Data 4, 5
11
5
6
10
7
9
8
16
Problem Dynamism

• Need to move items to restore replication degree!

Node 5 4, 5
Node 11 11,10,9,8,7,6
Data 12, 13, 14, 15, 0
Node 11 11,10, 9, 8, 7, 6
0
15
1
Node 0 12, 13, 14, 15, 0
2
14
Data 1, 2, 3
13
3
Node 3 1, 2, 3
Node 11 11,10, 9, 8, 7, 6
Node 5 4, 5
12
4
Node 0 12, 13, 14, 15, 0
Get items
Data 11, 10, 9, 8, 7, 6
Node 3 1, 2, 3
Data 4, 5
Data 10, 9, 8, 7, 6
11
5
6
10
7
9
8
17
Problem Dynamism

• Every join/leave/failure requires
• ?(f) messages, to restore a replication degree of
  f
• Need coordination of the restoration
• What if the coordinator fails?

18
Problem Performance Security

• How do we contact 2nd, 3rd, 4th replica?
• Have to contact the 1st replica!
• The 1st replica is
• Bottleneck
• Can fail
• Can be malicious (distributed voting useless)

19
Outline

• Background
• Proposed scheme symmetric replication
• Extensions
• Related Work Summary

20
Symmetric Replication

• Goal of Symmetric Replication
• Simple algorithms
• Each join/leave should require O(1) messages
• Enable concurrent requests
• Load-balancing without a bottleneck
• Increase security
• Use distributed voting
• Add proximity information
• Route through closest replica

21
Symmetric Replication

• Idea
• For replication degree f, partition the
  identifier space into m equivalence classes
• Use the congruence classes modulo f
• The cardinality of each class is f
• Each node replicates the equivalence class of
  every identifier it is responsible for

22
Symmetric replication

• Replication degree f4, Id0,,15
• 4 equivalence classes

0, 4, 8, 12
1, 5, 9, 13
Data 15, 0
2, 6, 10, 14
Data 14, 13, 12, 11
0
15
1
3, 7, 11, 15
2
14
Data 1, 2, 3
13
3
12
4
Data 4, 5
11
5
6
10
7
9
Data 6, 7, 8, 9, 10
8
23
Symmetric replication

• Replication degree f4, Id0,,15
• 4 equivalence classes

Data 3, 4
Data 7, 8
0, 4, 8, 12
Data 2, 1, 0, 15
Data 11, 12
Data 6, 5, 4, 3
1, 5, 9, 13
Data 15, 0
Data 10, 9, 8, 7
Data 5, 6, 7
2, 6, 10, 14
Data 14, 13, 12, 11
Data 9, 10, 11
0
15
1
Data 13, 14, 15
3, 7, 11, 15
2
14
Data 1, 2, 3
13
3
Data 8, 9
Data 12, 13
12
4
Data 0, 1
Data 10, 11, 12, 13, 14
Data 4, 5
11
5
Data 14, 15, 0, 1, 2
6
10
Data 2, 3, 4, 5, 6
7
9
Data 6, 7, 8, 9, 10
8
24
Symmetric replication joins!

• Replication degree f4, Id0,,15
• 4 equivalence classes

Data 3, 4
Data 7, 8
0, 4, 8, 12
Data 2, 1
Data 11, 12
Data 6, 5
1, 5, 9, 13
Data 15, 0
Data 10, 9
Data 5, 6, 7
2, 6, 10, 14
Data 14, 13,
Data 9, 10, 11
0
15
1
Data 13, 14, 15
3, 7, 11, 15
2
14
Data 0, 15
Data 1, 2, 3
Data 3, 4
13
3
Data 11, 12, 7, 8, 3, 4, 0, 15
Data 8, 9
Data 7, 8
Data 12, 13
Data 11, 12
12
4
Get items
Data 0, 1
Data 10, 11, 12, 13, 14
Data 4, 5
11
5
Data 14, 15, 0, 1, 2
6
10
Data 2, 3, 4, 5, 6
7
9
Data 6, 7, 8, 9, 10
8
25
Symmetric replication failures
Data 3, 4
Data 7, 8
0, 4, 8, 12
Data 2, 1
Data 11, 12
Data 6, 5
1, 5, 9, 13
Data 15, 0
Data 10, 9
Data 5, 6, 7
2, 6, 10, 14
Data 14, 13,
Data 9, 10, 11
0
15
1
Data 13, 14, 15
3, 7, 11, 15
2
14
Data 1, 2, 3
Successor has to fetch 11, 12 from closest
replica
13
3
Data 8, 9
Data 12, 13
12
4
Data 0, 1
Data 10, 11, 12, 13, 14
Data 4, 5
11
5
Data 14, 15, 0, 1, 2
6
10
Data 2, 3, 4, 5, 6
7
9
Data 6, 7, 8, 9, 10
8
26
Restoring Replication Degee failures

• Assume a node responsible for (10, 14) fails
• Classes 101, 10, 19 145, 14, 23
• We decide to fetch replicas from (1, 5)
• Use restricted broadcast to fetch (1, 5)
• ansary-iptps02, ghodsi-pdcs02
• Reliable, O(log R) time complexity to reach R
  nodes
• Several nodes might be responsible for (1, 5)
• Given uniform distribution of node identifiers
• Epeers in range (1,5) 1

27
Outline

• Background
• Proposed scheme symmetric replication
• Extensions
• Related Work Summary

28
Extensions to Symmetric Replication

• Augment the routing information to
• Add proximity awareness
• Example node 2, f3 22, 5, 8

29
Extensions to Symmetric Replication

• Augment the routing information to
• Add proximity awareness
• Example node 2, f3 22, 5, 8

30
Extensions Erasure Codes

• Symmetric Replication with Erasure Codes
• Encode data into n blocks, only k needed to
  reconstruct original data
• Example n10, k5
• Replication degreen ( encoded blocks)
• fetch any k blocks to reconstruct data

31
Outline

• Background
• Proposed scheme symmetric replication
• Extensions
• Related Work Summary

32
Related Work

• Several systems use multiple hash functions
• Content Addressable Networks (CAN)
• Tapestry
• Simple Load Balancing in Chord

33
Problem with Multiple Hash Functions

• Hash1(music.mp3)11
• Hash2(music.mp3)2

Data 15, 0
Data 14, 13,
0
15
1
2
14
Data 1, 2, 3
Node 12 crashed, We need the inverse of
Hash1-1(11)music.mp3
13
3
12
4
Data 4, 5
11
5
6
10
7
9
Data 6, 7, 8, 9, 10
8
34
Summary

• A general replica placement scheme
• Algorithms for join, leave, fail, insert in paper
• O(1) messages to restore replication degree f
  instead of ?(f)
• Can access any replica without a bottleneck
• Concurrent Requests
• Load balance queries
• Distributed Voting
• Current Use
• Implemented in DKS, http//dks.sics.se/

35

• Thank you for listening!