Algorithm Design Techniques

1
Algorithm Design Techniques

• Greedy Algorithms
• Find some local minimum without regard for future
  consequences.
• Take what you can get now.
• Hope that the sum of the local optima is equal to
  the global optimum Not always the case.
• Generally, this produces a suboptimal result, but
  a good result, in much less time that the optimum
  result would take.

2
Job Scheduling Example

• Given a set of jobs j1, j2,, jn, with running
  times t1, t2,,tn, a single processor when the
  job gets the processor, it will keep it until the
  job ends. What is the best way to schedule the
  jobs to minimize the average wait time?
• The best answer is shortest job first.

3
Proof

• Look at the average cost for shortest job next
  (we can look at just the sum).
• Assume t1ltt2ltlttn.
• Wait time(t1)(t1t2)(t1t2t3) (t1t2tn).
• Rearranging
• W

4
More Proof

• Notice that the first part will give the same
  result no matter what the ordering of jobs is.
• To make W small, make the second summation large.
• Another view Swap two values t3 and t6
• instead of 3t36t6, we have 3t66t3
• let t6t3p
• 3t36(t3p) vs. 3(t3p)6t3

5
Multiprocessor Case

• We can apply the same algorithm to the same
  problem, but with multiple processors.
• The algorithm will give us the minimum average
  wait time.
• However, on a multiprocessor, we may be more
  concerned with minimum all completion time (if
  one user is waiting on all the jobs.
• This in an NP-complete problem.

6
Approximate Bin Packing

• Given N items of size s1, s2,sN where 0ltsi?1,
  pack these items into the fewest number of bins,
  given that each bin is of size 1.
• On-line algorithm place item into a bin before
  you get the next item. Cannot always get an
  optimal solution.
• Off-line algorithm wait until you have all the
  items and then do the bin packing.

7
Next Fit

• Check to see if the item fits in the current bin.
  If not go to the next bin (do not go to a
  previous bin).
• Let M be the optimal number of bins required to
  pack a list L of items. Then next fit never uses
  more than 2M bins. There exist sequences such
  that next fit uses 2M-2 bins.

8
Proof

• Consider any adjacent bins Bj and Bj1. The sum
  of the sizes of all items in Bj and Bj1 must be
  gt1 (otherwise they would be packed in the same
  bin). Therefore, at most half the space would be
  wasted.
• Assume si.5 for i odd, and si2/N if i is even.
  The optimal packing takes N/41 bins. Next fit
  uses N/2 bins.

9
First Fit

• Scan all the bins and place the new item in the
  first bin that it fits in.
• Naïve implementation is O(n2), but can be
  implemented in O(n log n).
• No more than one bin can be half empty.
• Let M be the optimal number of bins. First fit
  never uses more than ?17/10 M? bins.

10
Sample Case

• Look at 6T items of size 1/7? followed by 6T
  items of size 1/3?, followed by 6T items of size
  ½?. Use one of each in each bin (21146/42
  3? 41/423?). Uses 6T bins. First fit uses 10T
  bins. (T 3T 6T).

11
Best Fit

• Place the new item in the tightest spot
  available.
• Still bounded by 17/10 worse than optimal on
  worse case.
• However, average case is better.
• Takes O(n log n) time.

12
Off Line Algorithms

• Since we have all the items, and we know from
  experience that large items are more difficult to
  pack, we sort the items in descending order.
• Then use either best fit decreasing, or first fit
  decreasing.

13
First Fit Descending

• Let the N items have (decreasing) order
  s1,s2,,sn, and the optimal packing is M bins.
  Then all items that first fit decreasing places
  in extra bins have size at most 1/3.
• Assume the ith item is the first item placed in
  bin M1. Show this item has size lt1/3.

14
Proof

• Assume sigt1/3. Therefore s1,s2,si-1gt1/3.
• And B1,B2,,Bi-1 have at most 2 items in each
  bin.
• Look at bins just before inserting i.
• Let Bx have two items and By have one and xlty.
• Let x1 and x2 be the 2 item and y1 be the one.
• x1gty1. x2gtsi. Thus x1x2gty1si. Thus, there is
  no xlty with such a property when given our
  assumptions.

15
More Proof

• So the M bins have the first j bins with 1
  element and the other M-j bins with 2.
• No 2 s1,sj could be placed in one bin.
• Since first fit did not place sj1,,si into the
  first j bins, they must not fit with the other
  elements.
• Since all the elements are bigger than 1/3, we
  cannot rearrange things to put the ith element
  into a bin. So cannot do the optimal packing.
  So, silt1/3.

16
Rest of Proof

• The number of objects placed in extra bins is at
  most M-1.
• Assume at least M objects are placed in extra
  bins. We know that s1s2snltM since they fit
  into M bins.
• Let bin Bj have total weight Wj.

17
Conclusion

• Wjxj gt 1 otherwise they would be packed in the
  same bin.
• Thus the last term adds up to greater than M
  which contradicts the optimal assumption.
• So, first fit decreasing has at most
  ceil((M-1)/3) extra bins.

18
Best Fit Decreasing

• Best Fit decreasing never uses more than
  (11/9M4) bins.

19
Divide and Conquer

• Recursively divide the problem into smaller
  problems, until you get to a base case.
• Create the solution as the combination of
  solutions to smaller problems.

20
Divide Conquer Algorithms

• We have seen some
• Quicksort
• Divide into 2 groups
• Conquer one group must have all values less
  than all the values in the other group.
• Tree Search
• Divide into left and right subtree
• Conquer trivial

21
Running Time

• A typical formula for divide and conquer
  algorithms is
• T(N) 2T(N/2) O(N)
• Since the algorithms are recursive, this formula
  is recursive
• The O(N) is the time it takes to put the
  solutions together.
• More general form
• T(N) aT(N/b) O(Nk) a?1, b?1

22
Solutions

• If agtbk then O(n logba)
• If abk then O(nk log n)
• If altbk then O(nk)

23
Closest Points Problem

• Given many (x,y) pairs, find the 2 points that
  are the closest.
• Sort by x values (O(n log n))
• Divide x range in half.
• Either the closest pair are in first half, second
  half, or they cross the boundary.
• Can easily do the halves recursively. The
  problem is crossing the boundary.

24
Crossing the Boundary

• Look at the solutions for the 2 halves, then find
  the minimum, call it d.
• We only need to look in a range distance d from
  the dividing line (any further will not get a
  better result that crosses the boundary)
• Also within the 2d strip, the y values of any
  better point pairs cannot differ by more than d
• Sort the values in the strip by the y value

25
More Boundary Crossing

• This will greatly shorten the search.
• However, this sort will greatly increase the
  running time.
• At the beginning, sort all points by x and create
  another list with all points sorted by y.
• Now when splitting, you can send the appropriate
  part of y to each half with a simple O(N) pass
  through the y list.

26
Selection

• Find the kth smallest element in a collection of
  unsorted values.
• Of course no worse than O(n log n)
• Look at a variation of Quicksort called
  Quickselect.

27
Quickselect

• Perform partitioning.
• If left sidegtk then item is in left side
• If left side1k then done
• If left sideltk then item is in right side
• If the size of the proper side is ltCutoff then
  sort and done.
• Notice only one recursive call instead of 2.
• Average time is O(n). Worst is O(n2).

28
Avoiding the Worse Case

• We used median of 3 to pick the pivot.
• Could still have only 2 items in one partition.
• Better method
• Place the elements into N/5 groups of 5.
• Find the median of each group of 5.
• Find the median of the N/5 groups of 5.
• Guarantees at worst a 70/30 split.

29
Why?

• Assume N/5m is odd.
• Then there are (m-1)/2 medians smaller than v
  (the median of the N/5 medians).
• And there are (m-1)/2 medians larger than v.
• There are at least 3 items in the (m-1)/2 groups
  with values larger than v. And there are 2 items
  in vs group larger than v.
• We know there are at least 3(m-1)/22 3/2m- ½
  (3/2)(N/5)- ½ 3/10N ½ elements larger than v.
  Similarly for smaller.

30
Multiplying Integers

• Lets try a divide and conquer technique.
• Divide the 2 N digit values into 4 N/2 digit
  values.
• Now XYXLYL 10N(XLYRXRYL)10N/2XRYR
• Notice that there are 4 multiplications of half
  the size
• T(N)4T(N/2)O(N), still N2

31
The Grand Solution

• Look at
• XLYRXRYL(XLXR)(YRYL) XLYL XRYR
• But XLYL and XRYR have already been computed.
• Instead of taking 2 multiplies, this takes 1
• So now, T(N)3T(N/2) O(N).
• This is O(Nlog23) ? O(N1.59)
• Perform 1 or 2 digit multiplication with table
  lookup.

32
Matrix Multiplication

• Normally takes O(n3)
• Try the same thing.
• Take a matrix and divide it into 4 quadrants.
• Then C11A11B11A12B12
• C12A11 B12A12B22
• C21A21B11A22B21
• C22A21B12A22B22

33
Analysis

• This can be done recursively, until each of the
  A, B, Cs are of size one (I.e., the quadrants
  are of size 2x2).
• We then have
• T(N)8T(N/2)O(N2)
• Matrix additions take O(N2)
• Still is O(N3)

34
The Grand Finale

• Compute
• M1(A12-A22)(B21B22)
• M2(A11A22)(B11B22)
• M3(A11-A21)(B11B12)
• M4(A11A12)B22
• M5A11(B12-B22)
• M6A22(B21-B11)
• M7(A21A22)B11
• Then
• C11M1M2-M4M6
• C12M4M5
• C21M6M7
• C22M2-M3M5-M7

35
Analysis

• Takes 7 multiplications
• T(N)7T(N/2) O(N2)
• This is O(Nlog27) ? O(N2.81)