Greedy

1
Greedy

• We start off with some coverage of Greedy Methods
  (Graph Algorithms, Activity Selection, Knapsack
  Problem, Huffman Codes, etc.). Please look at
  the appropriate sections in the text to refresh
  your memories.

2
Greedy
What is a Greedy Algorithm?

• Solves an optimization problem
• the solution is best in some sense.
• Greedy Strategy
• At each decision point, do what looks best
  locally
• Choice does not depend on evaluating potential
  future choices or solving subproblems
• Top-down algorithmic structure
• With each step, reduce problem to a smaller
  problem
• Optimal Substructure
• optimal solution contains in it optimal solutions
  to subproblems
• Greedy Choice Property
• locally best globally best

3
Greedy

• Examples
• Minimum Spanning Tree
• Minimum Spanning Forest
• Dijkstra Shortest Path
• Huffman Codes
• Fractional Knapsack
• Activity Selection

4
Greedy

• Minimum Spanning Tree

Time O(ElgE) given fast FIND-SET, UNION
Produces minimum weight tree of edges that
includes every vertex.
Time O(ElgV) O(ElgE) slightly faster
with fast priority queue
for Undirected, Connected, Weighted Graph
G(V,E)
source 91.503 textbook Cormen et al.
5
Greedy

• Single Source Shortest Paths Dijkstras
  Algorithm

for (nonnegative) weighted, directed graph G(V,E)
source 91.503 textbook Cormen et al.
6
Greedy

• Huffman Codes

7
Greedy

• Fractional Knapsack

50
Value 60 100 120
30
20
10
knapsack
item1
item2
item3
8
Greedy Example

• Activity Selection Problem Instance
• Set S 1, 2,..., n of n activities
• Each activity i has
• start time si
• finish time fi
• Activities i, j are compatible iff
  non-overlapping
• Objective
• select a maximum-sized set of mutually compatible
  activities

9
Greedy

• Greedy Algorithm

source 91.503 textbook Cormen, et al.

• Algorithm
• S presort activities in S by nondecreasing
  finish time
• and renumber
• GREEDY-ACTIVITY-SELECTOR(S)
• n lengthS
• A 1
• j 1
• for i 2 to n
• do if
• then
• j i
• return A

10
Greedy

• Why does this all work?
• Why does it provide an optimal (maximal)
  solution? It is clear that it will provide a
  solution (a set of non-overlapping activities),
  but there is no obvious reason to believe that it
  will provide a maximal set of activities.
• We start with a restatement of the problem in
  such a way that a dynamic programming solution
  can be constructed. This solution will be shown
  to be maximal. It will then be modified into a
  greedy solution in such a way that maximality
  will be preserved.

11
Greedy

• Consider the set S of activities, sorted in
  monotonically increasing order of finishing time
• Where the activity ai corresponds to the time
  interval si, fi).
• The subset a3, a9, a11 consists of mutually
  compatible activities, but is not maximal.
• The subsets a1, a4, a8, a11, and a2, a4, a9,
  a11 are also compatible and are maximal.

i

• 2 3 4 5 6 7 8 9 10 11
• 1 3 0 5 3 5 6 8 8 2 12
• 4 5 6 7 8 9 10 11 12 13 14

si
fi
12
Greedy

• First Step find an optimal substructure an
  optimal solution to a subproblem that can be
  extended to a full solution.
• Define an appropriate space of subproblems
  Sij ak S fi sk lt fk sj,
• the set of all those activities compatible with
  ai and aj and with those that finish no later
  than ai and start no earlier than aj. Add two
  activities
• a0, 0) an1 , )
• which come before and after, respectively, all
  activities in S S0,n1.
• Assume the activities are sorted in
  non-decreasing order of finish f0 f1 fn
  lt fn1.
• Then Sij is empty whenever i j.

13
Greedy

• We define the suproblems find a maximum size
  subset of mutually compatible activities from
  Sij, 0 i lt j n1.
• What is the substructure?
• Substructure suppose a solution to Sij contains
  some activity ak, fi sk lt fk sj. ak generates
  two subproblems, Sik and Skj.
• The solution to Sij is the union of a solution
  to Sik, the singleton activity ak, and a
  solution to Skj.
• Any solution to a larger problem can be obtained
  by patching together solutions for smaller
  problems.
• Cardinality (number of activities) is also
  additive.

14
Greedy

• Optimal Substructure assume Aij is an optimal
  solution to Sij, containing an activity ak.
• Then Aij contains a solution to Sik (the
  activities that end before the beginning of ak)
  and a solution to Skj (the activities that begin
  after the end of ak) .
• If these solutions were not already maximal, then
  one could obtain a maximal solution for, say, Sik
  , and splice it with ak and the solution to Skj
  to obtain a solution for Sij of greater
  cardinality, thus violating the optimality
  condition assumed for Aij.

15
Greedy

• Use Optimal Substructure to construct an optimal
  solution
• Any solution to a nonempty problem Sij includes
  some activity ak, and any optimal solution to Sij
  must contain optimal solutions to Sik and Skj.
• For each ak in S0,n1, find (recursively)
  optimal solutions of S0k and Sk,n1, say A0k and
  Ak,n1. Splice them together, along with ak, to
  form solutions Ak. Take a maximal Ak as an
  optimal solution A0,n1.
• We need to look at the recursion in more detail
  (cost??)

16
Greedy

• Second Step the recursive algorithm.
• Let ci,j denote the maximum number of
  compatible activities in Sij. It is easy to see
  that ci,j 0, whenever i j - and Sij is
  empty.
• If, in a non-empty set of activities Sij, the
  activity ak occurs as part of a maximal
  compatible subset, this generates two
  subproblems, Sik and Skj, and the equation
• ci,j ci,k 1 ck,j,
• which tells us how the cardinalities are related.
  The problem is that k is not known a priori.
  Solution
• ci,j maxiltkltj (ci,k 1 ck,j), if Sij
  ? .
• And one can write an easy bottom up (dynamic
  programming) algorithm to compute a maximal
  solution.

17
Greedy

• A better mousetrap. Recall Sij ak S fi
  sk lt fk sj.
• Theorem 16.1. Consider any nonempty subproblem
  Sij, and let am be the activity in Sij with the
  earliest finish time fm minfk ak in Sij.
  Then
• am is used in some maximum size subset of
  mutually compatible activities of Sij (e.g., Aij
  Aim U am U Amj).
• The subproblem Sim is empty, so that choosing am
  leaves the subproblem Smj as the only one that
  may be nonempty.
• Proof 2) if Sim is nonempty then Sij must
  contain an activity (finishing) prior to am.
  Contradiction.
• 1) Suppose Aij is a maximal subset. Either am is
  in Aij, and we are done, or it is not. If not,
  let ak be the earliest finishing activity in Aij.
  It can be replaced by am (since it finishes no
  earlier than am) thus giving a maximal subset
  containing am.

18
Greedy

• Why is this a better mousetrap?
• The dynamic programming solution requires
  solving j - i 1 subproblems to solve
  Sij. Total Running Time???
• Theorem 16.1 gives us conditions under which
  solving Sij requires solving ONE subproblem only,
  since the other subproblems implied by the
  dynamic programming recurrence relation are
  empty, or irrelevant (they might give us other
  maximal solutions, but we need just one)
• Lots less computation
• Another benefit comes from the observation that
  the problem can be solved in a top-down
  fashion take the earliest finishing activity,
  am, and you are left with the problem Sm,n1. It
  is easy to see that each activity needs to be
  looked at only once linear time (after sorting).

19
Greedy

• The Algorithm
• R_A_S(s, f, i, j)
• m i 1
• while m lt j and sm lt fi // find first activity in
  Sij
• do m m 1
• if m lt j
• then return amR_A_S(s, f, m, j)
• else return .
• The time is - fairly obviously .

20
Greedy
The Recursive Activity Selector Algorithm
Cormen, Leiserson, Rivest, Stein Fig. 16.1
21
Greedy

• A Slightly Different Perspective.
• Rather than start from a dynamic programming
  approach, moving to a greedy strategy, start with
  identifying the characteristics of the greedy
  approach.
• Make a choice and leave a subproblem to solve.
• Prove that the greedy choice is always safe -
  there is an optimal solution starting from a
  greedy choice.
• Prove that, having made a greedy choice, the
  solution of the remaining subproblem can be added
  to the greedy choice providing a solution to the
  original problem.

22
Greedy

• Greedy Choice Property.
• Choice that looks best in the current problem
  will work no need to look ahead to its
  implications for the solution.
• This is important because it reduces the amount
  of computational complexity we have to deal with.
• Cant expect this to hold all the time
  maximization of a function with multiple relative
  maxima on an interval - the gradient method would
  lead to a greedy choice, but may well lead to a
  relative maximum that is far from the actual
  maximum.

23
Greedy

• Optimal Substructure.
• While the greedy choice property tells us that we
  should be able to solve the problem with little
  computation, we need to know that the solution
  can be properly reconstructed an optimal
  solution contains optimal sub-solutions to the
  subproblems.
• An induction can then be used to turn the
  construction around (bottom-up).
• In the case of a problem possessing only the
  Optimal Substructure property there is little
  chance that we will be able to find methods more
  efficient than dynamic programming (with
  memoization).

24
Greedy

• Some Theoretical Foundations unifying the
  individual methods.
• Matroids. A matroid is an ordered pair M (S,
  I) s.t.
• 1. S is a finite non-empty set
• 2. I is a non-empty family of subsets of S,
  called the independent subsets of S, such that B
  I and A B, then A I. We say that I is
  hereditary if it satisfies this property. Note
  that Ø is a member of I.
• 3. If A I and B I, and A lt B, then there
  is an element x B-A such that A x I. We
  say that M satisfies the exchange property.
• Ex. the set of rows of a matrix.

25
Greedy

• Graphic Matroids MG (SG, IG), where we start
  from an undirected graph G(V, E).
• The set SG is defined to be E, the set of edges
  of G
• If A is a subset of E, then A IG is and only
  if A is acyclic - i.e., a set of edges is
  independent if and only if the subgraph GA (V,
  A) forms a forest.
• We have to prove that the object so created (MG)
  is, in fact, a matroid. The relevant theorem is

26
Greedy

• Theorem. If G is undirected graph, then MG (SG,
  IG) is a matroid.
• Proof. SG E is finite
• IG is hereditary, since a subset of a forest is
  still a forest (removing edges cannot introduce
  cycles).
• We are left with showing MG satisfies the
  exchange property. Let GA (V, A) and GB (V,
  B) be forests of G, with B gt A (A and B are
  acyclic sets of edges with B containg more edges
  than A).
• Claim a forest with k edges has exactly V - k
  trees.
• Proof start with a forest with no edges and add
  one edge at a time.
• GA contains V - A trees, while GB contains
  V - B trees GA contains more trees than GB.

27
Greedy

• GB must contain some tree T whose vertices are in
  two trees of GA (everything is acyclic). Since T
  is connected, it must contain an edge (u, v) such
  that vertices u and v are in different trees of
  GA, and so can be added to GA without creating a
  cycle.
• But this is exactly the exchange property.
• All three properties are satisfied, and MG is a
  matroid.

28
Greedy

• Def. given a matroid M (S, I), we call an
  element x / A an extension of A I if A x
  I.
• Ex. if A is an independent set of edges in MG,
  the edge e is an extension of A if adding e to A
  does not introduce a cycle.
• Def. if A is an independent set in a matroid M,
  A is maximal if it has no extensions.
• Theorem 16.6. All maximal independent subsets ina
  matroid have the same size.
• Proof if not, and A lt B, both maximal, then
  A can be extended. Contradiction

29
Greedy

• Ex. Let MG be a graphic matroid for a connected,
  undirected graph G.
• Every maximal independent subset of MG must be a
  free tree with exactly V - 1 edges a spanning
  tree of G.
• Def. a matroid is weighted if there is a
  strictly positive weight function on the edges of
  the matroid. It can be extended to sets of edges
  by summation
• w(A) sum x A w(x)
• Ex. minimum spanning tree problem. We must find
  a subset of the edges that conencts all the
  vertices and has minimum total length. How is
  this a matroid problem??

30
Greedy

• Let MG be a weighted matroid with weight function
  
• w(e) w0 - w(e),
• where w(e) is the positive weight function on the
  edges and w0 is a positive number larger than the
  weight of any edge.
• Each maximal independent subset A corresponds to
  a spanning tree and
• w(A) (V - 1)w0 - w(A)
• for any such set, an independent set that
  maximizes w(A) is one that minimizes w(A).
• The algorithm is on the next slide. SM denotes
  the edges, IM denotes the independent sets.

31
Greedy

• GREEDY(M, w)
• Running Time let n S. Then sorting takes n
  lg(n). Line 4 is executed n times, once for each
  element of S. This requires a check that
  is independent, for time, say O(f(n)). Thus the
  total time is O(n lg(n) n f(n)).
• Furthermore, A is independent.

32
Greedy

• Lemma. (Matroids exhibit the greedy choice
  property)
• Suppose that M (S, I) is a weighted matroid
  with weight function w and that S is sorted into
  nondecreasing order by weight. Let x be the first
  element of S such that x is independent, if any
  such x exists. If x exists, then there exists an
  optimal subset A of S that contains x.
• Proof. If no such x exists, the only independent
  subset is the empty set, and we are done.
  Otherwise, let B be any nonempty optimal subset.
  If x B, let A B, and we are done. If x /
  B, no element of B has weight greater than x (x
  is a heaviest independent element and every
  element of B is independent by the hereditary
  property of I).

33
Greedy

• Start with A x. A is independent because x
  is. Using the exchange property, repeatedly find
  a new element of B that can be added to A, while
  preserving the independence of A, until A
  B. The construction gives that A (B - y)
  x for some y B, and so
• w(A) w(B) - w(y) w(x) w(B).
• Since B is optimal, A must also be optimal, and
  since
• x A,
• The result follows.

34
Greedy

• Lemma. Let M (S, I) be a matroid. If x is an
  element of S such that x is not an extension of
  Ø, then x is not an extension of any independent
  subset A of S.
• Proof. The contrapositive assume x is an
  extension of an independent subset A of S. Then
  A x is independent. By the hereditary
  property x is independent, which automatically
  implies that it is an extension of Ø.
• Another way of stating this result is that any
  item than cannot be used right now, cannot be
  used in the future

35
Greedy

• Lemma. (Matroids exhibit the optimal substructure
  property) Let x be the first element of S chosen
  by GREEDY for the weighted matroid M (S, I).
  The remaining problem of finding the
  maximum-weight independent subset containing x
  reduces to finding a maximum-weight independent
  subset of the weighted matroid M' (S', I'),
  where
• and the weight function for M' is the weight
  function for M, restricted to S'. (We call M' the
  contraction of M by the element x)

36
Greedy

• Theorem. (Correctness of the greedy algorithm on
  matroids) If M(S, I) is a weighted matroid with
  weight function w, then the call GREEDY(M, w)
  returns an optimal subset.
• Proof.