Update Propagation with Variable Connectivity

1
Update Propagation with Variable Connectivity
Prasun Dewan
Department of Computer Science University of
North Carolina dewan_at_unc.edu
2
Evolution of wireless networks

• Early days disconnected computing (Coda91)
• Laptops plugged in at home or office
• No wireless network
• Later weakly connected computing (Coda 95)
• Assume a wireless network available, but
• Performance may be poor
• Cost may be high
• Energy consumption may be high
• Intermittent disconnectivity causes involuntary
  breaks

3
Issues raised by Coda91

• Considers strong connection and disconnection
• weak connection? hoarding, emulation, or
  something else?

4
Design Principles for Weak Connectivity

• Dont make strongly-connected clients suffer
• E.g. wait for weakly connected client to give
  token
• Weak-connection should be better than
  disconnection
• Do communication tasks in background if possible
• user patience threshold
• In weakly connected mode, communicate
• the more important info
• what is important?
• When in doubt ask, user
• Should be able to work without user advice

5
Disadvantages of Disconnection

• Updates not visible to others
• Updates at risk due to theft, loss, damage
• Conflicts more likely
• Cache misses may impede progress
• Exhaustion of cache space

6
State Transition Diagram

• Serve cache miss
• Mark stale data
• Background adapted hoard walk
• Track changes trickle merging

• Serve cache miss
• Hoard walk
• Replace stale data
• Write through

Disconnection
Logical Reconnection
Physical Reconnection

• Detect conflicts
• Resolve conflicts

• Provide access to cached, possibly stale data
• Track changes to cached data

7
Immediate vs. Delayed Service of Cache Miss

• On cache miss
• if fetch time lt user-patience-threshold
• get remote object (in foreground)
• else
• add to items fetched during next hoard walk
  (in background)

(2 e0.01p )
8
Replacing vs. Marking Stale Data

• Replacing stale data would make strongly
  connected clients wait for weakly connected ones.
• Stale data replaced by invoking client callbacks
• Stale data marked by breaking a client callback
• Requires sending of message to client
• Only on first remote update to cached object
• Client probably sends message to break callback
  so that server does not have know who is weakly
  connected
• Could give feedback to user, disallow changes or
  viewing of stale data, but not mentioned in paper

9
Volume Callbacks

• Volume callbacks another efficiency mechanism
• Broken when any object in a volume changes
• Also make merging more efficient
• If entire volume is unchanged then individual
  objects not examined
• Can still invoke (any) file callbacks if volume
  callback broken
• Adaptation to fine-grained (Sync-like) merging
• Hierarchical callbacks

10
Hoard Walk

• Done every
• h time units (10 minutes)
• or user request
• Not the same as connected hoard walk
• Stale small files automatically included
• User has option to marks large files to be fetched

11
Trickle Merging

• Do not propagate local changes immediately
• reduces of messages sent
• reduces size of data sent
• buffering changes allows log compression
• important because whole file sent
• what if small (Sync-like) changes sent?

12
Trickle Merging vs. Write-back Caching

• Do not propagate local changes immediately
• reduce network traffic
• Weak consistency
• Conflicts
• Changes buffered in persistent storage
• Buffering time large
• Many Minutes, days

• Do not propagate local changes
• reduce latency
• Strong consistency
• No conflicts
• Changes buffered in volatile memory
• Buffering time small
• Seconds, few minutes

13
Trickle Merging Implementation
write f1
mkdir d1
mkdir d2
mkdir d3
TM
TM
14
Trickle Merging Implementation
write f1
mkdir d1
mkdir d2
mkdir d3
TM
Do high priority task (e.g. servicing cache miss)
15
Trickle Merging Implementation
mkdir d1
mkdir d2
mkdir d3
rm d1
TM
Committed items not candidate for compression
16
Trickle Merging Implementation
mkdir d3
rm d1
17
Trickle Merging Implementation

• Consider those log changes created gt A time units
  (600 s) ago
• Allows change to age
• 50 compression effectiveness on all traces
• Break these into chunks (C time units)
• Amount of data transmitted in time C based on
  available bandwidth
• Urgent messages sent between chunk transmission
• Cache miss services
• Trickle merging not done atomically
• New log changes
• not combined with changes chosen at merge start
  time
• not sent for merging
• Not specified when initiated
• Periodic, during inactivity, when earliest item
  ages

18
Merging

• Times chosen to get desired level of compression
• Desired level of consistency?

19
Trading Consistency for Performance
Performance
Consistency (Divergence)
20
Two methods to control divergence

• Lotus Notes, Suite, CodaKeep optimistic model
• Reduce transaction length
• based on real-time
• push model
• see other ways in TACT

• Rover Mix pessimistic and optimistic
• Worry about W-W conflicts but not R-W conflicts
• Some objects may be cached in read-only mode.
• Verify object not changed before allowing change
• Pull model client can check
• Push model server can inform (Coda)
• Worry about R-W conflicts.
• Expire object after timeout.

21
TACT Combining and extending aspects of the two
approaches

• Optimistic and pessimistic combined
• Can choose which conflicts are delayed
• Optimistic transaction length bounded
• Real-time used as in Rover and Coda
• Logical properties of operations also used
• Three tunable parameters per replicating host
• Can simulate pessimistic, optimistic and points
  in between.
• Control staleness of data read and tentativeness
  of local writes
• Model called continuous consistency

22
Time-based Staleness

• Staleness of data controlled based on real-time
• Pull or push?
• Push requires a hosts synchronization times to
  be known by all remote hosts
• How to guarantee staleness level given message
  delay
• TACT local host pulls data from remote host
• after it has not heard form the latter t seconds
• guaranteed to not see data more stale than t
  seconds
• blocks during pull, so message delays do not
  count
• performance probably worse
• Called staleness error
• Physical Staleness

23
Content-based Staleness

• Staleness of data controlled based on how much
  remote values diverge from local value.
• Measuring value divergence?
• Could be number of remote writes, but
• some writes more major than others
• allocate five seats vs. 1 seat
• some writes cancel others
• allocate vs. deallocate seat
• Sum of (/-) weights of remote writes
• alloc n seats given weight n
• decalloc n seats given weight -n
• Local host pull vs. remote hosts push
• push because remote hosts knows what changes made

24
Content-based Staleness (contd)

• For each host i, one value, e i, defined for all
  other hosts
• conceptually want to think of divergence of value
  v I at host i from final value with all
  distributed changes taken into account
• Value divided evenly among all hosts
• each remote host checks abs(divergence) i lt e i
  /n -1
• if not sends its local changes to host i
• thus each remote host processes independently
• decentralized process (local decision on each
  write)
• change of parameter requires broadcast and
  reallocation
• e i is called numeric error since it is a number
• in case of numeric objects, represents error of
  local version
• but also applicable to non-numeric results
• bulletin board
• represents a numeric value for staleness
  depending on content of object
• logical staleness

25
Tentativeness of Local Writes

• Local writes tentative until ordered wrt to
  remote writes
• Parameter determines how many local writes
  buffered until resolution process occurs
• a subset of remote peers may be sent the writes
• primary server
• quorum
• Called order error
• assumes merge process simply orders the writes
• other approaches possible
• taking avg, min
• discarding
• order not important for commuting operations
• In case list insertions, causes insertions to be
  out of order
• bulletin board example

26
TACT Example
Host A
Host B
X 1
X 3
Y 6
Y 3
X 1
X 1
Y 2
Y 3
X 2
Y 4
27
Simulating Optimistic (Weak Consistency)

• No special process used to reduce conflicts
• Content-based staleness infinity
• Time-based staleness infinity
• Write tentativeness infinity
• A replica uses regular mechanisms to merge
• export in Rover
• anti-entropy in Bayou
• strong connection in Coda91

28
Simulating Pessimistic

• Each operation checked for global conflict
• Content-based staleness 0 or
• Time-based staleness 0 or
• Tentativeness 0
• of all replicas
• A replica syncs with all other replicas on each
  operation

29
Staleness vs. Tentativeness

• Not orthogonal
• Reducing staleness reduces tentativeness
• implies more frequent synchronization
• Not equivalent
• Pull vs push
• Reducing tentativeness may not reduce staleness
• all hosts may not be contacted
• Tentativeness 0 vs Staleness 0
• In both cases all local allowed changes will be
  committed in the correct order
• But some remote changes may be reordered
• Chances of reordering smaller if staleness 0
• Serialiazability vs Linearizability (Strong
  consistency)

30
Possible Extensions

• Use type-based serializability
• for numeric values such as int, real
• automatically determine weights
• table
• automatically determine
• put(k, e) cancelled by remove(e)
• put(k1, e1) commutes with put (k2, e2) and
  remove(k2)
• do not have to worry about merging them or order
  errors among them
• sequence
• automatically determine
• insert (i, e) cancelled by del(e)
• simulate Coda and Rover weak merging

31
Simulating Trickle Merging

• Merge with some period, p
• Content-based staleness infinity
• Time-based staleness p
• Write tentativeness infinity
• Not the same, because pull in TACT instead of
  push in CODA95