Problem Complexity and NP-Complete Problems

1
Problem Complexity and NP-Complete Problems

• Presented by Ming-Tsung Hsu

2
I cant find an efficient algorithm, I guess
Im just too dumb
3
I cant find an efficient algorithm, because
no such algorithm is possible
4
I cant find an efficient algorithm, but
neither can all these famous people
5
Problem Complexity
6
Overview

• Complexity theory
• Part of the theory of computation
• Dealing with the resources required to solve a
  given problem
• time (how many steps it takes to solve a problem)
  
• space (how much memory it takes)
• Computational difficulty of computable functions

7
Relations between Problems, Algorithms, and
Programs
Problem
Algorithm
. . . .
Algorithm
. . . .
Program
Program
Program
Program
. . . .

• A single "problem" is an entire set of related
  questions, where each question is a finite-length
  string
• A particular question is called an instance

8
Time Complexity

• Number of steps that it takes to solve an
  instance of the problem
• Function of the size of the input
• Using the most efficient algorithm
• Exact number of steps will depend on exactly what
  machine or language is being used
• To avoid that problem, generally use Big O
  notation

9
Time Complexity (contd)

• Worst-case
• an upper bound on the running time for any input
• Average-case
• we shall assume that all inputs of a given size
  are equally likely
• Best-case
• to get the lower bound

10
Time Complexity (contd)

• Sequential search in a list of size n
• worst-case n times
• best-case 1 times
• average-case

11
Asymptotic Notation - O-notation (Big O, Order
of)

• Asymptotic upper bound
• Definition For a given function g(n), denoted by
  O(g(n)) is the set of functions O(g(n))
  there exist positive constants c and n0 such
  that f(n) lt cg(n) for all n gt n0

12
Asymptotic Notation - O-notation

• Asymptotic lower bound
• Definition For a given function g(n), denoted by
  O(g(n)) is the set of functions?(g(n)) there
  exist positive constants c and n0 such that f(n)
  gt cg(n) for all n gt n0

13
Asymptotic Notation ?-notation

• Definition For a given function g(n), denoted by
  ?(g(n)) is the set of functions ?(g(n))
  there exist positive constants c1, c2, and n0
  such that c1g(n) lt f(n) lt c2g(n) for all n gt
  n0

14
Cost and Complexity

• Algorithm complexity can be expressed in Order
  notation, e.g. at what rate does work grow with
  N?
• O(1) Constant
• O(logN) Sub-linear
• O(N) Linear
• O(NlogN) Nearly linear
• O(N2) Quadratic
• O(XN) Exponential
• But, for a given problem, how do we know if a
  better algorithm is possible?

15
Practical Complexities

• 109 instructions/second computer

16
Impractical Complexities

• 109 instructions/second computer

17
Faster Computer Vs Better Algorithm

• Algorithmic improvement more useful
• than hardware improvement.
• E.g. 2n to n3

18
The Problem of Sorting

• For example, in discussing the problem of
  sorting
• Algorithms
• Bubble-sort O(N2)
• Merge-sort O(N Log N)
• Quick-sort O(N Log N)
• Can we do better than O(N Log N)?? What is the
  problem complexity?

19
Algorithm vs. Problem Complexity

• Algorithm complexity is defined by analysis of an
  algorithm
• Problem complexity is defined by
• An upper bound defined by an algorithm (worst
  case)
• A lower bound defined by a proof (A lot of
  problems are still unknown)

20
The Upper Bound

• Defined by an algorithm
• Defines that we know we can do at least this good
• Perhaps we can do better
• Lowered by a better algorithm
• For problem X, the best algorithm was O(N3), but
  my new algorithm is O(N2).

21
The Lower Bound

• Defined by a proof
• Defines that we know we can do no better than
  this
• It may be worse
• Raised by a better proof
• For problem X, the strongest proof showed that
  it required O(N), but my new, stronger proof
  shows that it requires at least O(N2).
• Might get harder and harder

22
Upper and Lower Bounds

• The Upper bound is the best algorithmic solution
  that has been found for a problem
• Whats the best that we know we can do?
• The Lower bound is the best solution that is
  theoretically possible.
• What cost can we prove is necessary?

23
Changing the Bounds
Lowered by betteralgorithm
Upper bound
Lower bound
Raised by betterproof
24
(No Transcript)
25
Closed Problems

• The upper and lower bounds are identical

Upper bound
The inherent complexity of problem
Lower bound
26
Closed Problems (contd)

• Better algorithms are still possible
• Better algorithms will not provide an improvement
  detectable by Big O
• Better algorithms can improve the constant costs
  hidden in Big O characterizations

27
Open Problems

• The upper and lower bounds differ.

Lowered by betteralgorithm
Upper bound
Unknown
Who Falls Short?
Lower bound
Raised by betterproof
28
Open Problems (contd)

• D. Harel. Algorithmics The Spirit of Computing.
  Addison-Wesley, 2nd edition,1992
• . . . if a problem gives rise to a. . . gap, the
  deficiency is not in the problem but in our
  knowledge about it. We have failed either in
  finding the best algorithm for it or in proving
  that a better one does not exist, or in both

29
Examples - Closed

• Problem Searching an unordered list of n items
• Upper bound O(n) comparisons (from linear
  search)
• Lower bound O(n) comparisons
• No gap so we know the problem complexity O(n)
• Problem Searching an ordered list of n items
• Upper bound O(log n) comparisons (from binary
  search)
• Lower bound O(log n) comparisons
• No gap so we know the problem complexity O(log
  n)

30
Examples - Closed (contd)

• Problem Sorting n arbitrary elements
• Upper bound O(n log n) comparisons (from,
  e.g., merge sort)
• Lower bound O(n log n) comparisons
• No gap so we know the problem complexity O(n
  log n)
• Problem Towers of Hanoi for n disks (n gt 1)
• Upper bound O(2n) moves
• Lower bound O(2n) moves
• No gap so we know the problem complexity O(2n)

31
Examples - Open

• Problem Multiplication of two integers, where n
  is the number of digits
• Upper bound O(nlognloglogn)
• Lower bound O(n)
• Theres a gap (but only a small one)
• Problem Finding the minimum spanning tree in a
  graph of n edges and m vertices.
• Upper bound O(nlogn) or O(nmlogm)
• Lower bound O(n)
• Theres a gap (but only a small one)
• Do not be fooled by the last two examples into
  thinking that all gaps are small!

32
Tractable vs. Intractable

• Problems are tractable if the upper bounds and
  lower bounds have only polynomial factors
• O (log N)
• O (N)
• O (NK) where K is a constant
• Problems are intractable if the upper bounds and
  lower bounds have an exponential factor(are
  solvable in theory, but can't be solved in
  practice)
• O (N!)
• O (NN)
• O (2N)

33
Terminology

• Polynomial algorithms are reasonable
• Polynomial problems are tractable
• Exponential algorithms are unreasonable
• Exponential problems are intractable

34
Terminology
35
NP-Complete Problems
36
Definitions of P, NP

• A decision problem is a problem where the answer
  is always YES/NO
• An optimization problem is a problem can be many
  possible solutions. Each solution has a value,
  and we wish to fond a solution with the optimal
  (maximum or minimum) value
• We can re-cast an optimization problem to a
  decision problem by using loop
• EX Find Max Clique in a graph ? Is there a
  clique which size is kn, n-1, ?

37
Definitions (contd)

• A deterministic algorithm is an algorithm which,
  in informal terms, behaves predictably. If it
  runs on a particular input, it will always
  produce the same correct output, and the
  underlying machine will always pass through the
  same sequence of states
• A non-deterministic algorithm has two stages
• 1. Guessing (in nondeterministic polynomial time
  with correct guessing) Stage
• 2. Verification (in deterministic polynomial
  time) Stage

38
Definitions (contd)

• EX A (3, 11, 2, 5, 8, 16, , 200 ), Is the
  x5 in A?
• deterministic algorithm
• for i1 to n if A(i) x then print (i)
  and return truenextreturn false
• non-deterministic algorithm
• j ? choice (1n)if A(j) x then print (j)
  success endifprint (0) failure

39
Certificates

• Returning true in order to show that the
  solution can be made, we only have to show one
  solution that works
• This is called a certificate.
• Returning false in order to show that the
  solution cannot be made, we must test all
  solution.

40
Oracles

• If we could make the right decision at all
  decision points, then we can determine whether a
  solution is possible very quickly!
• If the found solution is valid, then True
• If the found solution is invalid, then False
• If we could find the certificates quickly, NPC
  problems would become tractable O(N)
• This (magic) process that can always make the
  right guess is called an Oracle.

41
Determinism vs. Nondeterminism

• Nondeterministic algorithms produce an answer by
  a series of correct guesses
• Deterministic algorithms (like those that a
  computer executes) make decisions based on
  information.

42
Definitions (contd)

• The complexity class P is the set of decision
  problems that can be solved by a deterministic
  machine (algorithm) in polynomial time
• The complexity class NP is the set of decision
  problems that can be solved by a
  non-deterministic machine (algorithm) in
  polynomial time
• Since deterministic algorithms are just a special
  case of deterministic ones ? P?NP
• The big question Does P NP?

43
Definitions (contd)
44
Problems that Cross the Line

• What if a problem has
• An exponential upper bound
• A polynomial lower bound
• We have only found exponential algorithms, so it
  appears to be intractable.
• But... we cant prove that an exponential
  solution is needed, we cant prove that a
  polynomial algorithm cannot be developed, so we
  cant say the problem is intractable...

45
Reduction

• A problem P can be reduced to another problem Q
  if any instance of P can be rephrased to an
  instance of Q, the solution to which provides a
  solution to the instance of P
• This rephrasing is called a transformation
• If P is polynomial-time reducible to Q, we denote
  this P ? Q
• Intuitively If P reduces in polynomial time to
  Q, P is no harder to solve than Q
• EX Maximum ? Sorting but Sorting cannot be
  reduced to Maximum

46
The Boolean Satisfiability Problem (SAT)

• An instance of the problem is a Boolean
  expression written using only AND, OR, NOT,
  variables, and parentheses
• Question given the expression, is there some
  assignment of TRUE and FALSE values to the
  variables will make the entire expression true?
• Clause OR together a group of literals
• Conjunctive normal form (CNF) formulas that are
  a conjunction (AND) of clauses
• SAT ? NP O(2nm) (n of literals, m of
  clauses)

47
SAT (contd)

• EX CNF a1 V a2 V a3 (clause1) a1
  (clause2) ? (0, 0, 1)
  satisfied a2 (clause3)

48
Cooks Theorem

• NP P iff SAT ? P
• pf(?) ? SAT ? NP ? SAT?P(?) Difficult (every
  NP problem can be reduced to SAT problem)

49
NP-Complete and NP-Hard

• A problem P is NP-Complete if P ? NP and every NP
  problem can be reduced to P
• P ? NP and X ? P for all X ?NP
• If any NP-Complete problem can be solved in
  polynomial time, then NP P
• SAT problem is an NP-complete problem
• A problem P is NP-Hard if every NP problem can be
  reduced to P
• We say P is NP-Complete if P is NP-Hard and P ?
  NP

50
NP-Complete

• NP-Complete comes from
• Nondeterministic Algorithm in Polynomial time
• Complete - Solve one, Solve them all
• There are more NP-Complete problems than provably
  intractable problems.

51
NP-Hard

• NP-Hard comes from
• Nondeterministic Algorithm in Polynomial time
• Hard - as "hard" as any problem in NP

52
NP-Complete and NP-Hard (contd)
NP NPC
NPH
53
Example NP-Complete Problems

• Path-Finding (Traveling salesman)
• Map coloring
• Scheduling and Matching (bin packing)
• 2-D arrangement problems
• Planning problems (pert planning)
• Clique

54
Traveling Salesman
55
5-Clique
56
Map Coloring
57
The Knapsack Problem

• The 0-1 knapsack problem
• A thief must choose among n items, where the ith
  item worth vi dollars and weighs wi pounds
• Carrying at most W pounds, maximize value
• EX B 13,17,19,21,24,27,36,42, S 93, Find
  X(X1,X2,,X8), where Xi0 or 1, i1,2,3,,8 such
  that SXiBi S
• A variation, the fractional knapsack problem
• Thief can take fractions of items
• Think of items in 0-1 problem as gold ingots, in
  fractional problem as buckets of gold dust

58
Class Scheduling Problem

• With N teachers with certain hour restrictions M
  classes to be scheduled, can we
• Schedule all the classes
• Make sure that no two teachers teach the same
  class at the same time
• No teacher is scheduled to teach two classes at
  once

59
Proving NP-Completeness

• Show that the problem A is in NP. (i.e. Show that
  a certificate can be verified in polynomial time)
• Assume A is not NP complete
• Show how to convert an existing NPC problem into
  the problem A (in polynomial time)
• P is NP-Complete
• P ? A
• Proving that reduction is polynomial
• ? A is NP-Complete
• If we can do it weve done the proof!
• If we can turn an existing NP-complete problem
  into our problem in polynomial time... ???

60
3-SAT is NPC ?

• 3-SAT is NP
• SAT ? 3-SAT (by construct sets of unsatisfiable
  clauses)
• Clause contains only one literal Li
• Q1 V Q2 Li V Q1 V Q2
  O1 V Q2 Li V O1 V Q2 Q1 V
  O2 ? Li V Q1 V O2 O1 V O2
  Li V O1 V O2
• Clause contains two literals Li and Lj
• Q1 Li V Lj V
  Q1 O1 ? Li V Lj V
  O1

61
3-SAT is NPC? (contd)

• Clause contains three literals ? do nothing
• Clause contains more than three literals (L1, L2,
  , Ln )
• Reduction is O(mn) (m of clauses , n of
  literals)

62
3-SAT is NPC? (contd)

• Example

63
3-SAT is NPC? (contd)
L1 L2 L3 L4 L5 Y1 Y2 Y3 Y4 Y5
1 1 1 1 0 0 0 0 0 0
1 1 0 0 0 0 0 0 0 1
1 1 0 1 0 0 0 0 0 0
1 1 0 0 1 0 0 0 0 0
1 1 0 1 1 0 0 0 0 0
1 1 1 1 1 0 0 0 0 0
1 0 0 0 0 0 0 0 0 1
1 0 0 1 0 0 0 0 0 0
1 0 0 0 1 0 0 0 0 0
1 0 0 1 1 0 0 0 0 0
L1 L2 L3 L4 L5
1 1 1 1 0
1 1 0 0 0
1 1 0 1 0
1 1 0 0 1
1 1 0 1 1
1 1 1 1 1
1 0 0 0 0
1 0 0 1 0
1 0 0 0 1
1 0 0 1 1
64
Clique Decision Problem (CDP) is NPC

• Construct a graph G (V,E) from
  such that G will have a clique of size
  at least m iff F is satisfiable (SAT ? CDP)
• Construction Rule
• A literal in a clause is a vertex (node) in G
• A edge is a intersection of two clauses
• An assignment of F
• (a) One literal is true then the clause is true
  
• (b) A A does not exist
• ? Add edges between any two nodes exclusive of
• nodes are in the same clause and nodes are
  exclusive

65
CDP (contd)

• Example F (a1 V a2 V a3) (a1 V a2 V a3)
  (a2) and G constructed by F
• F is satisfied by (1,0,0) or (0,0,1)? G has a
  clique of size at least 3
• m 3 ? F is satisfied

a1, 1
a1, 2
a2, 1
a2, 2
a3, 1
a3, 2
a2, 3
66
CDP (contd)

• Proof support F has m clauses
• (? ) F is satisfiable ? exists an assignment
• the literals in this assignment forms a clique
  with size equal m (a complete subgraph of G with
  degree m-1)
• (?)If we can find a clique C with size equal m
• Every clause has exactly one literal in C
• Any two vertices in C are jointed and each edge
  is the intersection of two clauses ? Every
  clause is true by literals in C? F is satisfied

67
Become Famous!

• To get famous in a hurry, for any NP-Complete
  problem
• Raise the lower bound (via a stronger proof)
• Lower the upper bound (via a better algorithm)
• Theyll be naming buildings after you before you
  are dead!

68
Decidable vs. Undecidable Problems
69
Decidable Problems

• We now have three categories
• Tractable problems
• NP-Complete problems
• Intractable problems
• All of the above have algorithmic solutions, even
  if impractical.

70
Undecidable Problems

• No algorithmic solution exists
• Regardless of cost
• These problems arent computable
• No answer can be obtained in finite amount of time

71
The Unbounded Tiling Problem

• What if we took the tiling puzzle and removed the
  bounds
• For a given number (T) of different kinds of
  tiles
• Can we arrive at an arrangement that will fill
  any size area?
• By removing the bounds, this problem becomes
  undecidable.

72
The Halting Problem

• Given an algorithm A and an input I, will the
  algorithm reach a stopping place?
• loop
• exitif (x 1)
• if (even(x)) then
• x lt- x div 2
• else
• x lt- 3 x 1
• endloop
• In general, we cannot solve this problem in
  finite time.

73
Partially and Highly Undecidable

• Most undecidable problems have finite
  certificates. These are partially decidable.
• Some undecidable problems do not have finite
  certificates even with an oracle. These are
  called highly undecidable.

74
A Hierarchy of Problems

• For a given problem, is there or will there ever
  be an algorithmic solution?
• Problem In Theory In Application
• Highly Undecidable NO NO
• Partially Undecidable NO NO
• Intractable YES NO
• Tractable YES YES

75
Thanks