Optimization Problems

1
Optimization Problems

• Greg Stitt
• ECE Department
• University of Florida

2
Introduction

• Why are we studying optimization problems?
• Just about every RC-tool related issue is an
  optimization problem
• This lecture should provide abstract solutions to
  many RC tool problems

3
Time Complexity

• Time Complexity - Informal definition
• Defines how execution time increases as input
  size increases
• We are interested in worst case execution time
• Big O Notation
• O(n)
• Execution time increases linearly with input size
• O(n2)
• Quadratic growth
• O(2n)
• Exponential growth

4
Problem Classes

• P
• A problem is in P if it has a polynomial time
  solution
• O(n), O(n2), O(n3), etc.
• Undecidable
• Provably impossible to solve
• Example halting problem
• NP
• Does not mean Not-Polynomial!!!!
• A NP problem has a non-deterministic polynomial
  time solution
• NP-hard
• A problem is NP-hard if every problem in NP is
  reducible to it
• Basically means that NP-hard problems are at
  least as hard as the hardest problems in NP
• NP-complete
• NP NP-hard
• Most interesting problems are NP-complete!!!!
• Traveling salesman, Subset sum, 0-1 knapsack,
  graph coloring, vertex cover
• Place and route, logic minimization, minimum
  resource scheduling, hw/sw partitioning, etc.

5
The Big Question

• Does P NP?
• Been studied for a long time
• Never been proven either true or false
• Why do we care?
• Currently, best known solutions for NP-complete
  problems typically have complexity of O(2n),
  O(n!)
• Known as intractable takes more than your
  lifetime to solve
• If one NP-complete problem can be solved in
  polynomial time (is in P), then all NP-complete
  problems can be solved in polynomial time
• Remember, many interesting problems are
  NP-complete
• If you can find a polynomial time solution to an
  NP-complete RC problem
• Ill give you an A

And, Ill get a million dollars
(www.claymath.org/millennium/P_vs_NP/ )
6
Problem Classes
7
Optimization Problems

• Informal definition
• Problem of finding the best solution from all
  possible solutions
• Typically involves finding best solution for
  millions, billions, or even more possibilities
• Possible solution exhaustive search
• Check every possible solution, save best one
• Works, but
• Many optimization problems are NP-complete
• Not feasible to generate all possible solutions
• Example O(2n)
• n 5 gt 32, n 10 gt 1024, n 100 gt 1.3
  1030
• Problem If most optimization problems are
  intractable, how do we solve them?
• Could use solution to any NP-complete problem!
• Remember any NP-complete problem can be reduced
  to any other NP-complete problem
• But, this doesnt really help
• There is no known polynomial solution to any
  NP-complete problem
• So, what can we do?

8
Branch and Bound

• Another solution
• Branch and bound
• Idea Try to eliminate many possibilities without
  evaluating them
• How it works
• Imagine a tree representing all possible
  solutions
• Algorithm progressively builds tree
• Maintains best found solution so far
• When considering a branch in the tree, determines
  best possible solution represented by branch
• If branch cant possibly be better than current
  best, dont bother to explore possible solutions
• Prunes solution space
• Result
• Very often, can eliminate large percentage of
  possible solutions
• Still finds optimal solution
• Usually, much faster than exhaustive search
• - But, still has exponential complexity ?
• O(2n), O(n!)
• - Still not feasible for large input sizes

9
Heuristics

• Another solution
• Forget about trying to find the best solution
• Focus on finding a good solution quickly
• Known as heuristics
• Good candidates for heuristics
• Map problem to other NP-complete problem, use
  heuristic for that problem
• Source of way too many publications
• Use generalized optimization problem heuristic
• Due to large number of interesting optimization
  problems, research has introduced general
  heuristics that apply to all optimization
  problems
• Examples
• Hill Climbing
• Simulated Annealing
• Genetic Algorithms

10
Hill Climbing

• Background Graph of solution space
• x axis possible solutions
• y axis quality of solution
• Height of solution represents goodness
• Peak represents best solution
• Informal description
• 1) Choose a solution s (usually randomly)
• 2) Find neighboring solution n (i.e. change 1
  thing)
• 3) If n better than s (climbs the hill)
• s n, Repeat from 2)
• 4) If all neighboring solutions worse than s
• s is answer (s is a peak)
• Example Traveling Salesman
• 1) Find a solution
• 2) Swap 2 cities, if improves solution keep, if
  not reject
• 3) repeat 2) until no improvement can be found

11
Hill Climbing

• Advantages
• Very fast, works well for certain problems
• Disadvantages
• What if there are multiple peaks?
• Hill climbing gets stuck at all peaks, known as
  local maxima
• Optimal solution is highest peak global maximum
• May result in extremely suboptimal solution if
  many peaks exist, which is common

12
Simulated Annealing

• Problem Hill climbing suffered from local peaks
• Need way of escaping local peaks
• gt Have to accept some bad moves
• Solution Simulated annealing
• Annealing is process of heating and cooling
  metals in order to improve strength
• Idea Controlled heating and cooling of metal
• When hot, atoms move around
• When cooled, atoms find configuration with lower
  internal energy
• i.e. makes metal stronger
• Analogy
• Temperature probability of accepting worse
  neighboring solution
• When temperature is high, likely to accept worse
  neighboring solutions (but may lead to better
  overall solution)
• Analogous to atoms wandering around
• Cooling represents shrinking probability of
  accepting worse solutions

13
Simulated Annealing

• Informal description
• 1) Set initial temperature and probability p
• 2) Find initial solution s
• 3) Find neighboring solution n
• 4) If n better than s
• s n (always accept better solutions)
• 5) If n worse than s
• Accept n with probability p
• 6) Reduce temperature and p by some amount
• 7) If final temperature not reached, repeat from
  3)
• 8) If final temperature reached, report best
  solution found

14
Simulated Annealing

• Advantages
• Can find near optimal solutions escapes local
  maxima
• Disadvantages
• Takes a long time to find near optimal solution
• But, still fast compared to algorithms
• Very sensitive to input parameters must be
  configured well to get good results
• How long to run?
• Initial temperature/final temperature
• What should initial probability be?
• Cooling schedule
• How much to reduce temperature and probability at
  each step?

15
Genetic Algorithms

• Another possible solution
• Imitate evolution survival of the fittest
• Assumption evolution produces better
  humans/animals
• Implies genetic processes must work well
• Genetic Algorithms
• Generates a population of solutions
• Selection chooses members of population
  (solutions) to survive by probability based on
  fitness function (i.e. natural selection)
• Bad solutions less likely to survive
• Reproduction combines attributes of selected
  population
• Crossover/Inheritence Combines traits for each
  parent (solution)
• Mutation Random change to characteristic
• Repeat until solution has evolved to acceptable
  lavel

16
Genetic Algorithms

• Advantages
• Can find near optimal solutions
• Disadvantages
• Takes a long time to find near optimal solution
• But, still fast compared to algorithms
• Very sensitive to input parameters must be
  configured well to get good results
• How large should population be?
• How does reproduction occur?
• How many generations should be considered?
• How much of each generations should be killed
  off?
• Same advantages/disadvantages as simulated
  annealing

17
Other Heuristics

• Ant colony optimization
• Tabu search
• Stochastic optimization
• Many others

18
Summary

• Optimizations problems search huge solution space
  for best solution
• Many optimization problems are NP-complete
• No known efficient solution
• Branch and bound helps a little
• Heuristics must be used
• Find a good solution in reasonable time
• General optimization problem heuristics
• Hill climbing fast, but gets stuck at local
  maxima
• Simulated annealing works well but sensitive to
  parameters
• Genetic algorithms - work well but sensitive to
  parameters
• Why should you care about any of this?
• Most RC tool problems are NP-complete
  optimization problems
• You now know how to solve almost all tool
  problems
• You can use branch and bound, hill climbing,
  simulated annealing, genetic algorithms, ant
  colony optimization, tabu search, etc.