Randomized Algorithms

1
ADVANCED ALGORITHM ANALYSISLECTURE
2RANDOMIZED ALGORITHMS

• Prof. Vuda Sreenivasarao
• Bahir Dar University-ETHIOPIA

2
Randomized Algorithms

• Objectives
• 1.Randomized Algorithms
• 1.1.Basic concept of Randomized algorithms with
  example.
• 1.2.Probabilistic inequalities in analysis with
  examples.
• 1.3.Amortized analysis with examples.
• 1.4.Competitive analysis with examples.

3
Deterministic Algorithms

• Input
  Output
• Goal to prove that the algorithm solves the
  problem correctly always and quickly typically
  the number of steps should be polynomial in the
  size of the input.

Algorithm
4
A short list of categories

• Algorithm types we will consider include
• Simple recursive algorithms
• Backtracking algorithms
• Divide and conquer algorithms
• Dynamic programming algorithms
• Greedy algorithms
• Branch and bound algorithms
• Brute force algorithms
• Randomized algorithms

5
Randomized algorithms

• Randomization. Allow fair coin flip in unit
  time.
• Why randomize? Can lead to simplest, fastest, or
  only known algorithm for a particular problem.
• Examples Symmetry breaking protocols, graph
  algorithms, quicksort, hashing, load balancing,
  Monte Carlo integration, cryptography.

6
Randomized algorithms

• A randomized algorithm is just one that depends
  on random numbers for its operation.
• These are randomized algorithms
• Using random numbers to help find a solution to a
  problem.
• Using random numbers to improve a solution to a
  problem.
• These are related topics
• Getting or generating random numbers.
• Generating random data for testing (or other)
  purposes.

7
1.Randomized Algorithms

• 
  
• INPUT
  OUTPUT
• RANDOM NUMBERS
• In addition to input algorithm takes a source of
  random numbers and makes random choices during
  execution.
• Random algorithms make decisions on rolls of the
  dice.
• Ex Quick sort, Quick Select and Hash tables.

ALGORITHM
8
Why use randomness?

• Avoid worst-case behavior randomness can
  (probabilistically) guarantee average case
  behavior
• Efficient approximate solutions to inflexible
  problems.

9
Making Decision
Flip a coin.
10
Making Decision
Flip a coin!
An algorithm which flip coins is called a
randomized algorithm.
11
Why Randomness?
Making decisions could be complicated.
A randomized algorithm is simpler.
Consider the minimum cut problem
Can be solved by max flow.
Randomized algorithm?
Pick a random edge and contract.
And repeat until two vertices left.
12
Why Randomness?
Making good decisions could be expensive.
A randomized algorithm is faster.
Consider a sorting procedure.
5 9 13 8 11 6 7 10
5 6 7
8
9 10 11 13
Picking an element in the middle makes the
procedure very efficient, but it is expensive
(i.e. linear time) to find such an element.
Picking a random element will do.
13
Why Randomness?
Making good decisions could be expensive.
A randomized algorithm is faster.

• Minimum spanning trees.
• A linear time randomized algorithm,
• known but no linear time deterministic
  algorithm.
• Primality testing
• A randomized polynomial time algorithm,
• but it takes thirty years to find a
  deterministic one.
• Volume estimation of a convex body
• A randomized polynomial time approximation
  algorithm,
• but no known deterministic polynomial time
  approximation algorithm.

14
Why Randomness?
In many practical problems, we need to deal with
HUGE input, and dont even have time to read it
once. But can we still do something useful?
Sub linear algorithm randomness is essential.

• Fingerprinting verifying equality of strings,
  pattern matching.
• The power of two choices load balancing,
  hashing.
• Random walk check connectivity in log-space.

15
Advantages of randomized algorithms

• Simplicity.
• Performance.
• For many problems, a randomized algorithm is the
  simplest, the fastest, or both.

16
Scope of Randomized Algorithms

• Number theoretic algorithms primality testing
  Monte Carlo.
• Data structures Sorting, order statistics,
  searching, computational geometry.
• Algebraic identities Polynomial and matrix
  identity verification. Interactive proof systems.
• Mathematical programming Faster algorithms for
  linear programming. Rounding linear program
  solutions to integer program solutions.

17
Scope of Randomized Algorithms

• Graph algorithms Minimum spanning trees shortest
  paths, minimum cuts.
• Counting and enumeration Matrix permanent.
  Counting combinatorial structures.
• Parallel and distributed computing Deadlock
  avoidance, distributed consensus.
• Probabilistic existence proofs Show that a
  combinatorial object arises with nonzero
  probability among objects drawn from a suitable
  probability space.

18
Randomized algorithms

• In a randomized algorithm (probabilistic
  algorithm), we make some random choices.
• 2 types of randomized algorithms
• For an optimization problem, a randomized
  algorithm gives an optimal solution. The average
  case time-complexity is more important than the
  worst case time-complexity.
• For a decision problem, a randomized algorithm
  may make mistakes. The probability of producing
  wrong solutions is very small.

19
Types of Random Algorithms

• Las Vegas
• Guaranteed to produce correct answer, but running
  time is probabilistic.
• Monte Carlo
• Running time bounded by input size, but answer
  may be wrong.

20
Las Vegas or Monte Carlo?
21
Las Vegas

• Always gives the true answer.
• Running time is random.
• Running time is bounded.
• Quick sort is a Las Vegas algorithm.
• A Las Vegas algorithm always produces the correct
  answer its running time is a random variable
  whose expectation is bounded (say by a
  polynomial).

22
Monte Carlo

• It may produce incorrect answer!
• We are able to bound its probability.
• By running it many times on independent random
  variables, we can make the failure probability
  arbitrarily small at the expense of running time.
• A Monte Carlo algorithm runs for a fixed number
  of steps and produces an answer that is correct
  with probability ?1/2.

23
RP Class ( randomized polynomial )

• Bounded polynomial time in the worst case.
• If the answer is Yes Pr return Yes gt ½.
• If the answer is No Pr return Yes 0.
• ½ is not actually important.

24
PP Class ( probabilistic polynomial )

• Bounded polynomial time in worst case.
• If the answer is Yes Pr return Yes gt ½.
• If the answer is No Pr return Yes lt ½.
• Unfortunately the definition is weak because the
  distance to ½ is important but is not considered.

25
Routing Problem

• There are n computers.
• Each computer has a packet.
• Each packet has a destination D(i).
• Packets can not follow the same edge
  simultaneously.
• An oblivious algorithm is required.
• For any deterministic oblivious algorithm on a
  network of N nodes each of out degree d, there is
  an instance of permutation routing requiring
  (N/d) ½.

26
Routing Problem

• Pick random intermediate destination.
• Packet i first travels to the intermediate
  destination and then to the final destination.
• With probability at least 1-(1/N), every packet
  reaches its destination in 14n of fewer steps in
  Qn.
• The expected number of steps is 15n.

27
1.Randomized Algorithms

• EXAMPLE Expectation-
• X()------ flips a coin.
• Heads One second to execute.
• Tails Three seconds.
• Let X be running time of one cell to X()
• with probability 0.5------ X is 1.
• With probability 0.5------X is 3.
• Here random variable is X.
• Expected value of XEX0.5x10.5x3 2 seconds
  expected time.
• Suppose we run X(),// take time X
• X(),// take
  time Y
• Total running time is TXY , here T is random
  variable.
• What is expected total time ET?
• Linearity of expectation EXYEXEY224
  seconds expected time.

28
Min_Cut Problem

• Definition
• Min_cut Problem is to find the minimum edge set
  C such that removing C disconnects the graph.
• Traditional Solution
• Max-flow The maximum amount of flow is equal to
  the capacity of a minimum cut

29
Example of Min_Cut
a
b
e.g. Min_Cut 2
30
Intuition

• Let a graph G has n nodes and size of min_cut
  k, that is C k then degree for each node gt
  k
• total number of edges in G gt nk/2.
• Randomized Min_Cut
• Input a graph G(V, E), V n
• Output min_cut C
• Repeat Pick any edge uniformly at random,
  collapse it and remove self-loops
• Until V down to 2.
• Running time is O(n-2)

31
Example of Randomized Min_Cut
min_cut 2
Or maybe
min_cut 4
32
Las Vegas VS Monte Carlo

• Las Vegas Algorithm It always produces the
  correct answer and the expected running time is
  finite (e.s.p. randomized quick sort).
• Monte Carlo Algorithm It may produce incorrect
  answer but with bounded error probability (e.s.p.
  randomized min_cut).

33
Analysis

• Probability of the first edge C
• Prob (kn/2 k ) / (kn/2)
• (n-2) / n
• Probability of the second edge C
• Prob (k(n-1)/2 k ) / (k(n-1)/2)
• (n-3) / (n-1)

min_cut
34
Analysis
Iteration Probability of avoiding C
1 (n 2) / n
2 (n 3) / (n 1)
3 (n 4) / (n 2)
4 (n 5) / (n 3)

n - 2 1 / 3
Prob. Of outputting C Pr gt

35
Analysis

• Probability of getting a min_cut is at least
  2/n(n-1)
• Might look like small, but gets bigger after
  repeating the algorithm e.s.p. If algorithm is
  running twice, probability of outputting C would
  be
• Pr 1 ( 1 ) 2
• Let r be the number of running times of
  algorithm.
• Total running time O(nr)

36
Internet Minimum Cut
June 1999 Internet graph, Bill Cheswick http//res
earch.lumeta.com/ches/map/gallery/index.html
37
1.Randomized Algorithms

• EXAMPLE Hash Tables-
• Random hash code maps each possible key to
  randomly chosen bucket, but a keys random hash
  code never changes.
• Good model for how a good hash code will
  perform-
• Assume hash table uses chaining , no duplicate
  keys.
• Perform find (k). K hashes to bucket b cost of
  search is one birr , plus birr for every entry in
  the bucket b whose key is not k.
• Suppose there are n keys in table besides k.
• V1,V2,Vn Random variables for each key Ki,
  Vi 1 if key Ki hashes to bucket b , Zero
  otherwise.

38
1.Randomized Algorithms

• Cost of find(k) is T 1V1V2.Vn.
• Expectation cost is ET 1EV1EV2-------E
  Vn
• N buckets- each key has 1/N probability of
  hashing to bucket b 0
• EVi1/N
• ET1n/N.
• If load factor C,ET O(1).
• Hash table operations take O(1) expected
  amortized time.

39
Contention Resolution in a Distributed System

• Contention resolution. Given n processes P1, ,
  Pn, each competing for access to a shared
  database. If two or more processes access the
  database simultaneously, all processes are locked
  out. Devise protocol to ensure all processes get
  through on a regular basis.
• Restriction. Processes can't communicate.
• Challenge Need symmetry-breaking
• paradigm.

P1
P2
...
Pn
40
Contention Resolution Randomized Protocol

• Protocol. Each process requests access to the
  database at time t with probability p 1/n.
• Claim. Let Si, t event that process i
  succeeds in accessing the database at time t.
  Then 1/(e ? n) ? PrS(i, t) ? 1/(2n).
• Pf. By independence, PrS(i, t) p
  (1-p)n-1.
• Setting p 1/n, we have PrS(i, t) 1/n (1 -
  1/n) n-1. ?
• Useful facts from calculus. As n increases from
  2, the function
• (1 - 1/n)n-1 converges monotonically from 1/4 up
  to 1/e
• (1 - 1/n)n-1 converges monotonically from 1/2
  down to 1/e.

none of remaining n-1 processes request access
process i requests access

between 1/e and 1/2
value that maximizes PrS(i, t)
41
Contention Resolution Randomized Protocol

• Claim. The probability that process i fails to
  access the database inen rounds is at most 1/e.
  After e?n(c ln n) rounds, the probability is at
  most n-c.
• Pf. Let Fi, t event that process i fails to
  access database in rounds 1 through t. By
  independence and previous claim, we havePrF(i,
  t) ? (1 - 1/(en)) t.
• Choose t ?e ? n?
• Choose t ?e ? n? ?c ln n?

42
1.Randomized Algorithms

• EXAMPLE Nuts and Bolts
• Suppose we are given n nuts and n bolts of
  different sizes.
• Each nut matches exactly one bolt and vice versa.
• The nuts and bolts are all almost exactly the
  same size, so we cant tell if one bolt is bigger
  than the other, or if one nut is bigger than the
  other. If we try to match a nut witch a bolt,
  however, the nut will be either too big, too
  small, or just right for the bolt.
• Our task is to match each nut to its
  corresponding bolt.

43
1.Randomized Algorithms

• Suppose we want to find the nut that matches a
  particular bolt.
• The obvious algorithm test every nut until we
  find a match requires exactly n-1 tests in the
  worst case.
• We might have to check every bolt except one if
  we get down the last bolt without finding a
  match, we know that the last nut is the one were
  looking for.
• Intuitively, in the average case, this
  algorithm will look at approximately n/2 nuts.
  But what exactly does average case mean?

44
Deterministic vs. Randomized Algorithms

• Normally, when we talk about the running time of
  an algorithm, we mean the worst-case running
  time. This is the maximum, over all problems of a
  certain size, of the running time of that
  algorithm on that input
• On extremely rare occasions, we will also be
  interested in the best-case running time
• The average-case running time is best defined by
  the expected value, over all inputs X of a
  certain size, of the algorithms running time for
  X

45
Randomized Algorithms

• Two kinds of algorithms deterministic and
  randomized.
• A deterministic algorithm is one that always
  behaves the same way given the same input the
  input completely determines the sequence of
  computations performed by the algorithm.
• Randomized algorithms, on the other hand, base
  their behavior not only on the input but also on
  several random choices.
• The same randomized algorithm, given the same
  input multiple times, may perform different
  computations in each invocation. This means,
  among other things, that the running time of a
  randomized algorithm on a given input is no
  longer fixed, but is itself a random variable.

46
EXAMPLE Nuts and Bolts

• Finding the nut that matches a given bolt.
• Uniformly is a technical term meaning that each
  nut has exactly the same probability of being
  chosen.
• So if there are k nuts left to test, each one
  will be chosen with probability 1/k.
• Now whats the expected number of comparisons we
  have to perform? Intuitively, it should be about
  n2, but lets formalize our intuition.

47
EXAMPLE Nuts and Bolts

• Let T(n) denote the number of comparisons our
  algorithm uses to find a match for a single bolt
  out of n nuts.
• We still have some simple base cases T(1) 0
  and T(2) 1, but when n gt 2, T(n) is a random
  variable.
• T(n) is always between 1 and n-1 its actual
  value depends on our algorithms random choices.
  We are interested in the expected value or
  expectation of T(n), which is defined as follows

48
EXAMPLE Nuts and Bolts

• If the target nut is the kth nut tested, our
  algorithm performs mink, n-1 comparisons.
• In particular, if the target nut is the last nut
  chosen, we dont actually test it. Because we
  choose the next nut to test uniformly at random,
  the target nut is equally likelywith probability
  exactly 1/nto be the first , second, third, or
  kth bolt tested, for any k. Thus

49
EXAMPLES

• Contention Resolution.
• Global Minimum Cut.
• Linearity of Expectation.
• MAX 3-SATISFIABILITY.
• Universal Hashing.
• Chernoff Bounds.
• Load Balancing.
• Randomized Divide-and-Conquer.
• Queuing problems.

50
Randomized Algorithms Examples

• Verifying Matrix Multiplication
• Problem Given three nxn matrices ABC is AB
  C?
• Deterministic algorithm
• (A) Multiply A and B and check if equal to C.
• (B) Running time? O(n3) by straight forward
  approach. O(n237) with fast matrix multi-
  plication (complicated and impractical).
• Randomized algorithm
• (A) Pick a random n x 1 vector r.
• (B) Return the answer of the equality ABr Cr.
• (C) Running time? O(n2)!

51
Quicksort Vs. Randomized Quicksort

• Quicksort
• (A) Pick a pivot element from array
• (B) Split array into 3 sub arrays those smaller
  than pivot, those larger than pivot, and the
  pivot itself.
• (C) Recursively sort the sub arrays, and
  concatenate them.
• Randomized Quicksort
• (A) Pick a pivot element uniformly at random from
  the array.
• (B) Split array into 3 sub arrays those smaller
  than pivot, those larger than pivot, and the
  pivot itself.
• (C) Recursively sort the sub arrays, and
  concatenate them.

52
Quicksort Vs. Randomized Quicksort

• Quicksort can take O(n2) time to sort array of
  size n.
• Randomized Quicksort sorts a given array of
  length n in O(n log n) expected time.
• Randomization can NOT eliminate the worst-case
  but it can make it less likely!

53
Examples
54
Examples

• Quicksort algorithm is very efficient in
  practice, its worst-case running time is rather
  slow. When sorting n elements, the number of
  comparisons may be O(n2).
• The worst case happens if the sizes of the sub
  problems are not balanced.
• we prove that if the pivot is selected uniformly
  at random, the expected number of comparisons of
  this randomized version of Quicksort is bounded
  by O(n log n).

55
Average-Case Analysis of Quicksort

• Let X total number of comparisons performed in
  all calls to PARTITION
• The total work done over the entire execution of
  Quicksort is
• O(ncX)O(nX)
• Need to estimate E(X)

56
Review of Probabilities
57
Review of Probabilities
(discrete case)
58
Random Variables

• Def. (Discrete) random variable X a function
  from a sample space S to the real numbers.
• It associates a real number with each possible
  outcome of an experiment.

X(j)
59
Random Variables
E.g. Toss a coin three times
define X numbers of heads
60
Computing Probabilities Using Random Variables

61
Expectation

• Expected value (expectation, mean) of a discrete
  random variable X is
• EX Sx x PrX x
• Average over all possible values of random
  variable X

62
Examples
Example X face of one fair dice EX 1?1/6
2?1/6 3?1/6 4?1/6 5?1/6 6?1/6 3.5
Example
63
Indicator Random Variables

• Given a sample space S and an event A, we define
  the indicator random variable IA associated
  with A
• IA 1 if A occurs
• 0 if A does not occur
• The expected value of an indicator random
  variable XAIA is
• EXA Pr A
• Proof
• EXA EIA

1 ? PrA 0 ? PrA
PrA
64
Examples
65
Examples
66
Quick Sort
67
Quick Sort
Partition set into two using randomly chosen
pivot
68
Quick Sort
69
Quick Sort
70
min-cut Examples

• An efficient algorithm cannot check all
  partitions.
• A sample execution of Algorithm on a graph with
  5 nodes. Throughout the execution, the edges of
  one min-cut of G are colored blue. At each step,
  the red, dashed edge is contracted.

71
Examples

• Algorithm Randomized Min-Cut(G (VE)).
• while the graph has more than two nodes do
• choose an edge e (u , v) uniformly at
  random
• contract e (the node that combines u and
  v inherits all the node labels of u and v)
• and remove self-loops note that parallel edges
  are not removed
• end while
• return the cut defined by the labels of one of
  the remaining nodes.

72
Probabilistic Analysis
73
Probabilistic analysis

• Probabilistic analysis is the use of probability
  in the analysis of problems.
• Most commonly, we use probabilistic analysis to
  analyze the running time of an algorithm.
• Then we analyze our algorithm, computing an
  expected running time.
• The expectation is taken over the distribution of
  the possible inputs. Thus we are, in effect,
  averaging the running time over all possible
  inputs.

74
Probability Measures

• Random Variables Binomial and Geometric.
• Useful Probabilistic Bounds and Inequalities.
• A probability measure (Prob) is a mapping from a
  set of events to the reals such that
• For any event A
• 0 ? Prob(A) ? 1
• Prob (all possible events) 1
• If A, B are mutually exclusive events, then
• Prob(A ? B) Prob (A) Prob (B)

75
Conditional Probability

• Define

76
Bayes Theorem

• If A1, , An are mutually exclusive and contain
  all events then

77
Random Variable A (Over Real Numbers)

• Density Function

78
Random Variable A (contd)

• Prob Distribution Function

79
Random Variable A (contd)

• If for Random Variables A,B
• Then A upper bounds B and B lower bounds A

80
Expectation of Random Variable A

• A is also called average of A and mean of A
  ?A

81
Variance of Random Variable A
82
Variance of Random Variable A (contd)
83
Discrete Random Variable A
84
Discrete Random Variable A (contd)
85
Discrete Random Variable A Over Nonnegative
Numbers

• Expectation

86
Pair-Wise Independent Random Variables

• A,B independent if
• Prob(A ? B) Prob(A) X Prob(B)
• Equivalent definition of independence

87
B is Binomial Variable with Parameters n , p
88
B is Binomial Variable with Parameters n , p
(contd)
89
Probabilistic Inequalities

• For Random Variable A

90
Markov and Chebychev Probabilistic Inequalities

• Markov Inequality (uses only mean)
• Chebychev Inequality (uses mean and variance)

91
Gaussian Density Function
92
Normal Distribution

• Bounds x gt 0

93
Sums of Independently Distributed Variables

• Let Sn be the sum of n independently distributed
  variables A1, , An
• Each with mean and variance
• So Sn has mean ? and variance ?2

94
Strong Law of Large Numbers Limiting to Normal
Distribution

• The probability density function of to normal
  distribution ?(x)
• Hence
• Prob

95
Strong Law of Large Numbers (contd)

• So
• Prob
• (since 1- ?(x) ? ?(x)/x)

96
Moments of Random Variable A

• nth Moments of Random Variable A
• Moment generating function

97
Amortized analysis
98
Amortized analysis

• After discussing algorithm design techniques
  (Dynamic programming and Greedy algorithms) we
  now return to data structures and discuss a new
  analysis method-Amortized analysis.
• Until now we have seen a number of data
  structures and analyzed the worst-case running
  time of each individual operation.
• Sometimes the cost of an operation vary widely,
  so that that worst-case running time is not
  really a good cost measure.

99
Amortized analysis

• Similarly, sometimes the cost of every single
  operation is not so important.
• the total cost of a series of operations are more
  important.(e.g when using priority queue to sort)
• We want to analyze running time of one single
  operation averaged over a sequence of operations.
• Again keep in mind Average" is over a sequence
  of operations for any sequence
• not average for some input distribution (as in
  quick-sort).
• not average over random choices made by algorithm
  (as in skip-lists).

100
Amortized Analysis

• Amortized analysis is a technique for analyzing
  an algorithm's running time.
• The average cost of a sequence of n operations on
  a given Data Structure.

101
Applications of amortized analysis

• Vectors/ tables
• Disjoint sets
• Priority queues
• Heaps, Binomial heaps, Fibonacci heaps
• splay trees
• union-find.
• Red black trees
• Maximum flow
• Dynamic arrays / hash tables

102
Difference between amortized and average cost

• To do averages we need to use probability
• For amortized analysis no such assumptions are
  needed
• We compute the average cost per operation for any
  mix of n operations

103
Amortized Analysis

• Not just consider one operation, but a sequence
  of operations on a given data structure.
• Average cost over a sequence of operations.
• Probabilistic analysis
• Average case running time average over all
  possible inputs for one algorithm (operation).
• If using probability, called expected running
  time.
• Amortized analysis
• No involvement of probability
• Average performance on a sequence of operations,
  even some operation is expensive.
• Guarantee average performance of each operation
  among the sequence in worst case.

104
Amortized Analysis

• In an amortized analysis the time required to
  perform a sequence of data structure operations
  is averaged over all the operations performed.
• Amortized analysis differs from average case
  analysis in that probability is not involved an
  amortized analysis guarantees the average
  performance of each operation in the worst case.

105
Indirect Solution
10 yards / minute 100 yards / minute
(Source Mark Allen Weiss. Data Structures and
Algorithm Analysis in Java.
106
Indirect Solution
10 yards / minute 100 yards / minute
Geometric Series?
The easy solution is indirect. It takes a kitten
5 minutes togo 50 yards, how far can the mother
go in 5 minutes?.... 500 yards!
(Source Mark Allen Weiss. Data Structures and
Algorithm Analysis in Java.
107
Amortized Analysis

• The worst-case running time is not always the
  same as the worst possible average running time.
• Example
• Worst case-time is O(n)
• Amortized worst-case is O(1)
• This could be from a series of table inserts and
  clears

(Source Arup Guha. CS2 Notes Summer 2007.
108
Binomial Queue

• Binomial Trees
• B0 B1 B2
• B3
• B4

Each tree doubles the previous.
109
Binomial Queue

• Binomial Queue
• A queue of binomial trees. A Forest.
• Each tree is essentially a heap constrained to
  the format of a binary tree.
• Example B0, B2, B3
• Insertion Create a B0 and merge
• Deletion remove minimum( the root) from tree Bk.
  This leaves a queue of trees B0, B1, , Bk-1.
  Merge this queue with the rest of the original
  queue.

(Source Mark Allen Weiss. Data Structures and
Algorithm Analysis in Java.
110
Binomial Queue Example

• The Most important step is Merge
• Merge Rules (for two binomial queues)
• 0 or 1 Bk trees ? just leave merged
• 2 Bk trees ? meld into 1 Bk1 tree.
• 3 Bk trees ? meld two into 1 Bk1, and leave the
  third.
• Insertion Runtime
• M1 steps, where M is smallest tree not present.
  Worst-case is k2, when Bk1 is the smallest tree
  not present. How does k relate to the total
  number of nodes in the tree?
• k lg n, thus (non amortized) worst-case time is
  O(lg n).

111
Binomial Queue Example

• Make Bin Q Problem Build a binomial queue of N
  elements. (Like make Heap). How long should this
  take?
• Insertion worst-case runtime
• Worst-case O(lg n) for 1 insert? O(n lg n) n
  inserts, but we want O(n) like make Heap
• Try amortized analysis directly
• Considering each linking step of the merge. The
  1st, 3rd, 5th, etc. odd steps require no linking
  step because there will be no B0. So ½ of all
  insertions require no linking, similarly ¼
  require 1 linking steps.
• We could continue down this path, but the itll
  be come especially difficult for deletion (we
  should learn an indirect analysis).

112
Binomial Queue Example

• Indirect Analysis (time M 1)
• If no B0 ? cost is 1 (M is 0)
• Results in 1 B0 tree added to the forest
• If B0 but no B1 ? cost is 2 (M is 1)
• Results in same of trees (new B1 but B0 is
  gone)
• When cost is 3 (M is 2)
• Results in 1 less tree (new B2 but remove B0 and
  B1)
• .etc.
• When cost is c (M is c 1)
• Results in increase of 2 c trees

113
Binomial Queue Example

• increase of 2 c trees
• How can we use this?
• Ti of trees after ith iteration
• T0 0 of trees initially
• Ci Cost of ith iteration
• Then, Ti Ti-1 (2 Ci) ?
• Ci (Ti Ti-1) 2
• This is only the ith iteration

114
Binomial Queue Example

• Ci (Ti Ti-1) 2
• To get all iterations
• C1 (T1 T0) 2
• C2 (T2 T1) 2
• Cn-1 (Tn-1 Tn-2) 2
• Cn (Tn Tn-1) 2
• n
• S Ci (Tn T0) 2n (T1 .. Tn-1 cancel out)
• i1

115
Binomial Queue Example

• n
• S Ci (Tn T0) 2n
• i1
• T0 0 and Tn is definitely not negative, so Tn
  T0 is not negative.
• n
• ?S Ci lt 2n
• i1

Thus, the total cost lt 2n ? make Bin Q
O(n) Since, make Bin Q consists of O(n) inserts,
then the amortized worst-case of each insert is
O(1).
116
Amortized Analysis

• There exists three common techniques used in
  amortized analysis
• Aggeregate method
• Accounting method
• Potential method

117
1.Aggeregate method

• 1.Aggeregate method
• We show that a sequence of n operations take
  worst-case time T(n) in total.
• In the worst case, the average. cost, or
  amortized cost, per operation is therefore
• T(n) / n.
• In the aggregate method, all operations have the
  same amortized cost.
• The other two methods, the accounting tricky and
  the potential function method, may assign
  different amortized costs to diferent types of
  operations.

118
An example push and pop

• A sequence of operations OP1, OP2, OPm
• OPi several pops (from the stack) and
• one push (into the stack)
• ti time spent by OPi
• the average time per operation

119

• Example a sequence of push and pop
• p pop , u push

tave (11311132)/8 13/8
1.625
120

• Another example a sequence of push and pop
• p pop , u push

tave (12111161)/8 14/8
1.75
121
Amortized Analysis

• Example Stack operations
• As we know, a normal stack is a data structure
  with operations
• Push Insert new element at top of stack
• Pop Delete top element from stack
• A stack can easily be implemented (using linked
  list) such that Push and Pop takes O(1) time.
• Consider the addition of another operation
• Multipop(k) Pop k elements off the stack.
• Analysis of a sequence of n operations
• One Multipop can take O(n) time ? O(n2) running
  time.
• Amortized running time of each operation is O(1)
  ? O(n) running time.
• Each element can be popped at most once each time
  it is pushed
• Number of Pop operations (including the one done
  by Multipop) is bounded by n
• Total cost of n operations is O(n)
• Amortized cost of one operation is O(n)/n O(1).
• Notice that no probability involved.

122
The Accounting Method

• Assign different charges to different operations.
• Some are charged more than actual cost, some
  are charged less.
• Amortized cost amount we charge.
• When amortized cost gt actual cost, store the
  difference on a credit object.
• Use credit later to pay for operations whose
  actual cost gt amortized cost.
• Need credit to never go negative.

123
The Accounting Method
124
Stack with Multi-Pop Operations
125
Potential Method

• Like the accounting , but credit stored with the
  entire structure.
• Accounting method stores credit with specific
  objects.
• Potential method stores potential in the data
  structure as a whole.
• Can release potential to pay for future
  operations.
• Most flexible of the amortized analysis
  methods.

126
Potential Method
127
Potential Method
128
Stack with Multi-Pop Operations
129
Skew heaps

• meld merge swapping

Two skew heaps
Step 1 Merge the right paths. 5 right heavy nodes
130
Step 2 Swap the children along the right path.
No right heavy node
131
Amortized analysis of skew heaps

• meld merge swapping
• operations on a skew heap
• find-min(h) find the min of a skew heap h.
• insert(x, h) insert x into a skew heap h.
• delete-min(h) delete the min from a skew heap h.
• meld(h1, h2) meld two skew heaps h1 and h2.
• The first three operations can be implemented
  by melding.

132
Potential function of skew heaps

• wt(x) of children of node x, including x.
• heavy node x wt(x) ? wt(p(x))/2, where
• p(x) is the parent node of x.
• light node not a heavy node
• potential function ?i of right heavy nodes of
  the skew heap.

133

• Any path in an n-node tree contains at most
  ?log2n? light nodes.

light nodes ? ?log2n?
heavyk3? ?log2n? possible heavy nodes
of nodes n

• The number of right heavy nodes attached to the
  left path is at most ?log2n ?.

134
Amortized time
light ? ?log2n1? heavy k1
light ? ?log2n2? heavy k2
135
(No Transcript)
136
AVL-trees
height balance of node v hb(v)height of right
sub tree - height of left sub tree
137

• Add a new node A.

Before insertion, hb(B)hb(C)hb(E)0 hb(I)?0
the first nonzero from leaves.
138
Amortized analysis of AVL-trees

• Consider a sequence of m insertions on an empty
  AVL-tree.
• T0 an empty AVL-tree.
• Ti the tree after the ith insertion.
• Li the length of the critical path involved in
  the ith insertion.
• X1 total of balance factor changing from 0 to
  1 or -1 during these m insertions (rebalancing
  cost)

139
Case 1 Absorption

• The tree height is not increased, we need not
  rebalance it.

Val(Ti)Val(Ti-1)(Li?1)
140
Case 2 Rebalancing the tree
141
Case 2.1 single rotation

• After a right rotation on the sub tree rooted at
  A

Val(Ti)Val(Ti-1)(Li-2)
142
Case 2.2 double rotation
143
Case 2.2 double rotation

• After a left rotation on the sub tree rooted at B
  and a right rotation on the sub tree rooted at A

Val(Ti)Val(Ti-1)(Li-2)
144
Case 3 Height increase

• Li is the height of the root.

Val(Ti)Val(Ti-1)Li
145
Amortized analysis of X1
146
Example Two-Stack System

• Suppose there are two stacks called A and B,
  manipulated by the following operations
• push(S, d) Push a datum d onto stack S.
• Real Cost 1.
• multi-pop(S, k) Removes min(k, S) elements
  from stack S.
• Real Cost min(k, S).
• transfer(k) Repeatedly pop elements from stack A
  and push them onto stack B. until either k
  elements have been moved, or A is empty.
• Real Cost of elements moved min(k, A).

147
Design

B
A
148
1.Aggregate Method

• For a sequence of n operations, there are n
  data pushed into either A or B. Therefore there
  are n data popped out from either A or B and
  n data transferred from A to B. Thus, the total
  cost of the n operations is 3n. Thus.

Operation Real Cost Amortized Cost
Push(A, d) 1 3
Push(B, d) 1 3
Multi pop(A, k) min(k, A) 3
Multi pop(B, k) min(k, B) 3
Transfer(k) min(k, A) 3
149
Illustration

B
A
150
2.Accounting Method

• push(A, d) 3 -- This pays for the push and a
  pop of the push a transfer and a pop.
• push(B, d) 2 -- This pays for the push and a
  pop.
• multi-pop(S, k) 0
• transfer(k) 0
• After any n operations you will have 2A B
  dollars in the bank.
• Thus the bank account never goes negative.
  Furthermore the amortized cost of the n
  operations is O(n) (more precisely 3n).

151
Comparison
Operation Real Cost Aggregate Accounting Potential
Push(A, d) 1 3 3
Push(B, d) 1 3 2
Multipop(A, k) min(k, A) 3 0
Multipop(B, k) min(k, B) 3 0
Transfer(k) min(k, A) 3 0

152
3.Potential Method

• Let F(A, B) 2A B, then
• c(push(A, d)) 1 DF 1 2 3
• c(push(B, d)) 1 DF 1 1 2
• c(multi-pop(A, k)) min(k, A) DF -min(k,
  A)
• c(multi-pop(B, k)) min(k, B) DF 0
• c(transfer(k)) min(k, A) DF 0
• Sc Sc DF
• Sc Sc - DF Sc 3n O(n).
• If we can pick F so that F(Di) ? F(D0) for all i,
  and that Sc is easy to compute, then Sc/n
  upper-bounds the average cost.

153
Summary
Operation Real Cost Aggregate Accounting Potential
Push(A, d) 1 3 3 3
Push(B, d) 1 3 2 2
Multipop(A, k) min(k, A) 3 0 -min(k, A)
Multipop(B, k) min(k, B) 3 0 0
Transfer(k) min(k, A) 3 0 0

154
Competitive analysis
155
On Bounds

• Worst Case.
• Average Case Running time over some distribution
  of input. (Quicksort)
• Amortized Analysis Worst case bound on sequence
  of operations.
• Competitive Analysis Compare the cost of an
  on-line algorithm with an optimal perceptive
  algorithm on any sequence of requests.

156
Applications

• Resource Allocation
• Scheduling
• Memory Management
• Routing
• Robot Motion Planning
• Exploring an unknown terrain
• Finding a destination
• Computational Finance

157
Self-organizing lists

• List L of n elements
• The operation ACCESS(x)costs rankL(x)distance of
  x from the head of L.
• L can be reordered by transposing adjacent
  elements at a cost of 1.
• Example
• Accessing the element with key 14 costs 4.
• Transposing 3 and 50 costs 1.

158
On-line and off-line problems

• Definition. A sequence S of operations is
  provided one at a time. For each operation, an
  on-line algorithm A must execute the operation
  immediately without any knowledge of future
  operations (e.g., Tetris).
• An off-line algorithm may see the whole sequence
  S in advance.
• Goal Minimize the total cost CA(S).

159
Worst-case analysis of self-organizing lists

• An adversary always accesses the tail (nth)
  element of L. Then, for any on-line algorithm A,
  we have
• CA(S) O(Sn)
• in the worst case.

160
Average-case analysis of self-organizing lists

• Suppose that element x is accessed with
  probability p(x). Then, we have
• which is minimized when L is sorted in decreasing
  order with respect to p.
• Heuristic Keep a count of the number of times
  each element is accessed, and maintain L in order
  of decreasing count.

161
The move-to-front heuristic

• Practice Implementers discovered that the
  move-to-front (MTF)heuristic empirically produces
  good results.
• IDEA After accessing x, move x to the head of L
  using transposes
• cost 2 rankL(x) .
• The MTF heuristic responds well to locality in
  the access sequence S.

162
Competitive analysis

• Definition . An on-line algorithm A is
  a-competitive if there exists a constant k such
  that for any sequence S of operations,
• CA(S) aCOPT(S) k,
• where OPT is the optimal off-line algorithm
  (Gods algorithm).

163
MTF is O(1)-competitive

• Theorem MTF is 4-competitive for self-organizing
  lists.

164
Potential function
Define the potential function FLi ?R by F(Li)
2 (x, y) x ?Li y and y ?Li x 2
inversions
F(Li) 2
F(Li) 2 (E,C),
165
Potential function
F(Li) 2 (E,C), (E,A),
F(Li) 2 (E,C), (E,A), (E,D),
F(Li) 2 (E,C), (E,A), (E,D), (E,B),
166
Potential function
F(Li) 2 (E,C), (E,A), (E,D), (E,B), (D,B)
F(Li) 2 (E,C), (E,A), (E,D), (E,B), (D,B)
10.
Note that F(Li) 0 for i 0, 1, , F(L0) 0
if MTF and OPT start with the same list. How much
does F change from 1 transpose? A transpose
creates/destroys 1 inversion. ?F 2 .
167
What happens on an access?
Suppose that operation i accesses element x, and
define
168
What happens on an access?
169
Amortized cost

• The amortized cost for the i th operation of MTF
  with respect to F is
• cici F(Li) F(Li1)
• 2r 2(A B ti)
• 2r 2(A (r1 A) ti)
• (since r A B 1)
• 2r 4A 2r 2 2ti
• 4A 2 2ti
• 4(r ti)
• (since r A C 1
  A 1)
• 4ci.

170
The grand finale
171
Appendix

• If we count transpositions that move x toward the
  front as free(models splicing x in and out of
  Lin constant time), then MTF is 2-competitive.
• What if L0?L0?
• Then, F(L0)might be T(n2)in the worst case.
• Thus, CMTF(S) 4 COPT(S) T(n2), which is
  still 4-competitive, since n2is constant as S
  ?8.

172
Assignments

• 1. Example on Traveling salesman problem.
• 2.Show that RANDOMIZED-QUICKSORTs expected
  running time is ?(n log n).
• 3.E XAMPLES on competitive Analysis.

173
(No Transcript)
174
(No Transcript)
175
Analysis

• Worst- case computing times for two sorting
  algorithms on random inputs.
• Average- case computing times for two sorting
  algorithms on random inputs.
• Comparison of Quick Sort and RQuick Sort on the
  input aii, 1,ltiltn times are in
  milliseconds.

N 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Merge Sort 105.7 206.4 335.2 422.1 589.9 691.3 794.8 889.5 1067 1167
Quick Sort 41.6 97.1 158.6 244.9 397.8 383.8 497.3 568.9 616.2 738.1
N 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Merge Sort 72.8 167.2 275.1 378.5 500.6 607.6 723.4 811.5 949.2 1073.6
Quick Sort 36.6 85.1 138.9 205.7 269.0 339.4 411.0 487.7 556.3 645.2
N 1000 2000 3000 4000 5000
Quick Sort 195.5 759.2 1728 3165 4829
R Quick Sort 9.4 21.0 30.5 41.6 52.8
176
Matrix multiplication
177
(No Transcript)
178
Pivot element

• our pivot element is near the center of the
  sorted array (i.e. , the pivot is close to the
  median element)

179

• Thank you

180

• Any Question?

181
Have a nice randomized life