A Story of Greed and Power

1
A Story of Greed and Power

• The heavy stuff
• Greedy Algorithms. Dynamic Programming.
• CLR Ch. 16 17

2
Problems of Optimal Substructure

• A problem with optimal substructure has a
  solution that contains within it optimal
  solutions to subproblems
• For example, the max of n numbers is the max of
  the first number and the last (n-1) numbers
• Another example is codeword trees over a set of
  characters C
• Remove any two characters that have a common
  parent node z (call them x and y) and replace
  those characters with z, and let the frequency of
  z be the sum of the frequencies of x and y call
  the new character set C
• The optimal codeword tree for C with x and y
  removed is an optimal codeword tree for C (proof
  is by contradiction if the resulting tree is
  not optimal for C, there is another tree that is
  optimal for C such that C x, y is a more
  optimal tree for C)

3
Techniques for Problems of Optimal Substructure

• Divide-and-conquer works if the interaction
  between subproblems is relatively simple
• Usually the subproblems do not overlap
• A technique called dynamic programming tabulates
  solutions to subproblems and calculates them on
  the fly
• Greedy algorithms use a greedy choice (an
  optimal-looking choice) to solve subproblems

4
Greedy Choices Activity Selection

• Consider a set of activities that all require use
  of the same room
• No activities can share the room with another
  activity for any length of time
• From a list of starting and finishing times for
  each activity, schedule the largest number of
  activities possible for the room

5
A Surprisingly Simple Activity Selection Algorithm

• Sort the activities by increasing finishing time
• Until all activities are considered
• Select the one with the earliest finishing time
• If no conflict with others, schedule it,
  otherwise remove it
• Problem has optimal substructure
• A nonempty optimal scheduling of a set of jobs J
  must consist of an optimal scheduling of a job j
  from J and an optimal scheduling of J- j all
  jobs conflicting with j (or else the original
  schedule wasnt optimal to begin with)
• The greedy choice property says an optimal
  solution can be found by making a greedy choice
  and then solving any subproblems that result

6
Runtime for Greedy Algorithm

• Greedy solution O(n lg n) to sort jobs
• All n jobs considered
• Turns out you can check whether a job is
  schedulable in O(1) time
• It suffices to check that the start time of the
  job in question is not earlier than the job last
  scheduled, since the jobs are scheduled by
  descending finish
• O(n lg n) to schedule all n jobs, theta(n) if the
  jobs are already sorted

7
Other Greedy Algorithm Examples

• Minimum Spanning Tree
• Making change with U.S. Currency
• File layouts on tape

8
Dynamic Programming

• Consider the knapsack problem
• N items, item j has value v(j) and weight w(j)
• Take the most valuable load possible if you have
  a knapsack that only holds W pounds
• Cannot be solved by greedy algorithm
• Greedy choice Take the most valuable item you
  can fit in the knapsack, and then repeat with the
  remaining capacity
• Doesnt work for the following combination W
  50, v(1) 40, w(1) 40, v(2) 30, w(2) 25,
  v(3) 30, w(3) 25
• Dynamic programming considers all subproblems
  separately by using recursion with a series of
  base cases

9
Dynamic Programming for the Knapsack Problem

• Number all the items
• Let V(W, x) be the value of the most valuable
  load for a knapsack limit of W for all x items,
  work backwards
• V(W, x) max(V(W, x-1), V(W-w(x),x-1) v(x)) if
  item x fits
• V(W, x) V(W, x-1) if item x doesnt fit
• Base cases V(W, x) 0 if W lt 0 or x 0
• Example given in class
• EXERCISE Why does this problem have optimal
  substructure??

10
Runtime for Knapsack Problem Solved by Dynamic
Programming

• Greedy algorithms cannot solve knapsack problem
  as stated
• Dynamic programming T(n) 2T(n-1) O(1)
• Worst-case time solving by recursion tree,
  runtime at each node is a constant, so just count
  the number of nodes, which is an exponential
  bound, O(2n)
• DP is much less efficient than a greedy algorithm
• DP equivalent to trying many possibilities
• Greedy algorithms exploit a known problem
  structure

11
Practical Dynamic Programming

• Look up solutions to problems you have already
  computed in other parts of the recursion tree
  called memoization
• Results in a matrix recursively solve each
  problem, look up a solution if the solution has
  already been computed (store results of
  subproblems in a table)
• For knapsack, you only spend O(1) filling a
  particular box, so runtime is O(nW) for n items
  and load limit W