Trace of find_gcd from last class

1
Trace of find_gcd from last class

• gcd(9,6) find_gcd(9,6,6)
• find_gcd(9,6,6)6test_rem(9,6,6)
  find_gcd(9,6,5)test(test_rem(9,6,6))
  60 find_gcd(9,6,5)1 find_gcd(9,6,5)
• find_gcd(9,6,5)5test_rem(9,6,5)
  find_gcd(9,6,4)test(test_rem(9,6,5))
  50 find_gcd(9,6,4)1 find_gcd(9,6,4)
• find_gcd(9,6,4)4test_rem(9,6,4)
  find_gcd(9,6,3)test(test_rem(9,6,4))
  40 find_gcd(9,6,3)1 find_gcd(9,6,3)
• find_gcd(9,6,3)3test_rem(9,6,3)
  find_gcd(9,6,2)test(test_rem(9,6,3))
  31 find_gcd(9,6,2)0 3
• find_gcd(9,6,2)2test_rem(9,6,2)
  find_gcd(9,6,1)test(test_rem(9,6,2))
  20 find_gcd(9,6,1)1 find_gcd(9,6,1)
• find_gcd(9,6,1)1test_rem(9,6,1)
  find_gcd(9,6,0)test(test_rem(9,6,1))
  11 find_gcd(9,6,0)0 1
• find_gcd(9,6,0) 0

2
10a Undecidable Problems Intro. to Complexity
12.1 14.1

• Computable Functions Decidable Problems
• Undecidable Problems
• Unsolvability of T.M. Halting Problem
• Reducing One Undecidable Problem to Another
• Review the Concept of Order of Complexity

3
Study Sheet for Final Exam

• Fall 2004 Final Exam Friday, November 19, from
  1130am-130pm
• Sample Test cs.union.edu/csc140/tests/
• Answer Key on my Bulletin Board

4
Computable Functions

• Input tapes consisting of ?'s and 1's
• There exists a Turing machine to compute the
  answer for "reasonable" input values
• May not be a total function
• e.g. quotient a/b may never halt for b0
• (algorithm involved subtracting b from a until
  value is less than b i.e. the remainder)

5
Decidable Problems

• Turing machine ALWAYS halts
• Halt in state qy (or write Y) for "yes"
• Halt in state qn (or write N) for "no"
• a/bc?
• TM must halt even for the case b0

6
Undecidable Problems

• TM DOES NOT EXIST that always halts with "yes" or
  "no" answer
• Just because one TM doesn't halt for a particular
  input, doesn't mean that no TM exists to decide
  for all inputs
• i.e. a/bc?
• one potential solution looped forever if b0
• we could modify the solution to check for 0 and
  halt with "no"

7
The Halting Problemfor Turing Machines

• Does a given machine M halt after a finite period
  of time when initially presented with a given
  input word w?
• A solution to this problem (i.e. a TM which
  answers the question) must ALWAYS halt with
  yes/no no matter what M and w its looking at.

8
Potential Solvability of Halting Problem for
Turing Machines

• Encode the "program" of TM M on the input tape
  followed by the input to the simulated machine
• wMw where wM is the encoded TM M, and w is its
  input
• Recall how the Universal Turing Machine simulates
  an arbitrary TM
• If simulated machine never halts, neither does
  the simulation
• Potential solution to Halting problem would be a
  modification of Universal TM. However, we must
  have a yes/no decision in all cases, not just a
  yes or I don't know yet...

9
Unsolvability of Halting Problemfor Turing
Machines (1 of 4)

• Proof by Contradiction
• Assume that there exists a TM that solves the
  Halting problem. Call this machine H. (note
  this will be the only assumption)
• Note Assumption means that H always halts with
  yes or no.

10
Unsolvability of Halting Problemfor Turing
Machines (2 of 4)

• From H, construct H' which loops when H halts in
  qy and halts when H halts in qn.
• How would we perform this construction?

11
Unsolvability of Halting Problemfor Turing
Machines (3 of 4)

• From H', construct H" which first makes a copy of
  its input tape, then invokes H'.
• We will present H" with the encoded description
  of the TM M in question to determine whether it
  halts when looking at the encoded description of
  itself as input.
• Input tape presented to H' would be wMwM
• Note that H" halts if M does not halt when
  presented with itself as input, and H" does not
  halt if M halts when presented with itself as
  input.

12
Unsolvability of Halting Problemfor Turing
Machines (4 of 4)

• The Sting
• Encode the program of H" and present it to H" as
  input
• H" halts when presented with H" if H" does not
  halt when presented with itself as input, and H"
  does not halt if H" halts when presented with
  itself as input.
• Contradiction--Assumption False!

13
Reducing One Undecidable Problem to Another

• State-Entry Problem Example 12.1, page 304-5
• From an arbitrary M, construct M' which enters
  state q iff M halted.
• If we had a solution to the state-entry problem,
  we would have a solution to the halting problem.
  By modus tolens the state-entry problem must
  therefore be unsolvable.
• Blank-Tape Halting Problem Example 12.2, page
  305
• For any M and w, construct Mw which starts by
  writing w on tape then proceeds to program of M.
• Mw will halt on blank tape iff M halts on w.

14
Complexity Theory
Weve dealt with Computability Theory What is
and what is not possible with a computer?
For the problems that are computable, this
leavesthe next question If we can solve a
problem, how easy or hard is it to do so?
Complexity Theory tries to answer this.
15
Measuring Complexity
Just as with computability, the complexity of a
problem should take into account all possible
Turing machine programs.
Consider a the prime decision problem PRIME
x x?N, x is a prime
The (time) complexity of a specific instance
x?PRIME? is meaningless.(The number x can be
listed in a big table.)
16
Asymptotics
The (perceived) difficulty of PRIME is
illustrated by the fact that the x?PRIME?
question seems to become much more difficult as x
gets bigger.
We study the relation between the size of a
problem instance and the required time/space
complexity of the solution for such an instance
(worst case).
17
Measuring the Efficiency of Algorithms

• Rapid advances in computer technology make
  physical measures (run-time, memory requirements)
  irrelevant.
• A more standardized measure is the number of
  elementary computer operations it takes to solve
  the problem in the worst case.
• Average performance is not safe.
• There may be particular cases (technically called
  instances of a problem) that behave much worse
  than the average and nobody can afford to rely on
  luck.

18
Counting Resources
Complexity theory is the art of counting
resources.
Time complexity How many time-steps does
thecomputation of a problem cost?
Space complexity How many bits of memoryare
required for the computation?
Again we use the Turing machine model.
19
Number of operations depends on "size" of
problem data.

• Sorting 1 billion numbers is quite different from
  sorting 10, so we express the number of
  elementary operations (expected in the worst-case
  scenario) as a function of some characteristic
  number(s) that suffice to capture the volume of
  the work to be done.
• For a sorting algorithm, this number is simply
  the number n of numbers to be sorted.

20
Time Complexity
Let M be a Turing machine that halts on all
inputs. Definition The running time or time
complexityof M is the function fN?N, defined by
the maximization
We say f(n) is the running time of M, andM
is an f(n) time Turing machine.
21
f(n) O(g(n))
It is extremely convenient to use the
followingorder-notation to express our
complexities.
Let f and g be two functions N?R. We say and
write f(n) O(g(n)) if and only if thereare two
positive constant c and n0 such that f(n) ?
cg(n) for all n?n0.
We say that g(n) is an (asymptotic) upper
bound on f(n).
22
Some "big O" examples

• Let f(n) 15n2 7n and g(n) ½n3.
• We have f(n)O(g(n)) because for n016 and
  c2,we see indeed for all n ? n0
• f(n) 15n2 7n ? 16n2 ? n3 cg(n).

• 5n4 27n O(n4).
• Take n01 and c32.(But n03 and c6 works also.)

23
Hierarchy of Complexity

• Note that for sufficiently large n log n lt n lt
  n log n lt n2 lt n3 lt 2n
• log n is interpreted as base-2 logarithm
• it does not really matter, since log2n log10n
  / log102
• and constants are ignored
• This is sometimes stated as O(log n) lt O(n) lt
  O(n log n) lt O(n2) lt O(n3) lt O(2n)

24
Example Traveling Salesperson

• Notoriously difficult one to solve exactly.
• The formulation is very simple though "Given the
  coordinates of n cities, find the shortest closed
  tour which visits each city exactly once".
• This problem has served as a benchmark for almost
  every new algorithmic idea and was one of the
  first optimization problems conjectured and then
  shown to be hard.
• To see this famous problem in action TSP Java
  applet.

25
A Brute Force Approach

•   input n (number of cities), C(n x n) (distance
  matrix)   min sufficiently large number
  (say, n times the maximum element of C)
    for all possible tours i.e.
  cyclic permutations of 1,2,...,n      find
  length of tour      if length lt min then       
  minlength, store tour      endif   next tour
  end
• Since there are (n-1)! possible tours, the for
  ... next loop is executed (n-1)! times, each one
  requiring time O(n) (the time it takes to find
  the length of the tour, that is add n integers).
• Therefore, this algorithm is of order (n-1)! O(n)
  O(n!). This is certainly prohibitive.
• If, for example, a solution to a 10-city problem
  can be found in 1 second, increasing the number
  of cities to 20 would result to time 20!/10! or
  about 20,000 years!

26
Example Recall MergeSort from Data Structures
O(n log n)

• The following algorithm is based on the
  observation that a list can be easily sorted if
  it consists of two individually sorted lists. To
  sort the n elements of a vector A(1),...,A(n) one
  would call the following recursive procedure with
  parameters low1, highn
• Procedure mergesort(A,low,high)
• input A(low),A(low1),...,A(high) output same
  numbers in non-decreasing order
• begin   if low lthigh then     mid
  (lowhigh)/2     call mergesort(low,mid)    
  call mergesort(mid1,high)    
  merge(low,mid,high) !combine results, takes O(n)
    endif
• end

27
Polynomial and Exponential Time

• Algorithms with running times of orders O(log n),
  O(n), O(n log n), O(n2), O(n3) etc. are called
  polynomial-time algorithms.
• On the other hand, algorithms with complexities
  which cannot be bounded by polynomial functions
  are called exponential-time algorithms. These
  include "exploding-growth" orders which contain
  exponential factors, like n!.
• There are several arguments to support the thesis
  that polynomial is a synonym to practical.

28
Faster Computers Dont Help!

• The effect of improved technology is
  multiplicative in polynomial-time algorithms
• and only additive in exponential-time algorithms.
  
• The situation is much worse if complexities
  involve factorials.
• In the TSP, if an algorithm of order O(n!) solves
  a 300-city problem in the maximum time allowed,
  increasing the computation speed by 1000 will not
  even enable solution of problems with 302 cities
  in the same time.

29
Efficiency of Algorithms

• The technical term for a hard problem is
  "NP-complete" which essentially means "abandon
  all hope of finding an efficient algorithm for
  the exact solution of this problem".
• Knowing such limitations, experts do not waste
  time on impossible projects and instead turn to
  less ambitious approaches, for example
• to find approximate solutions
• to solve special cases
• to alter the problem a little so that it becomes
  tractable (even at the loss of some fit to the
  real-life situation).
• The goal of complexity theory is therefore to
  assist algorithm designers in directing their
  efforts towards promising areas and avoid
  impossible tasks.

30
Recognition Problems

• A special class of problems require only a "yes"
  or "no" answer. These are called recognition
  problems. The theory of Computational Complexity
  restricts attention to these problems for
  uniformity. However, the results can easily be
  generalized, since
• For every optimization problem, there is a
  recognition version
• Any complexity results for the recognition
  version also hold for the original problem, i.e.
  the original problem is not much harder than
  the recognition version
• Of course, there are problems which are already
  in a recognition form, like the Halting Problem.

31
Converting a General Problem into a Recognition
Problem

• Problem. Multiplication Input. A pair of numbers
  x and yOutput. The product x times y
• We can convert a multiplication problem into a
  yes/no answer by joining together the original
  question and the answer and asking if they form a
  correct pair. In the case of multiplication, we
  can convert a question like 4 x 9 ??
• into a yes/no statement such as
• "is it true that 4 x 9 36?" (yes), or "is it
  true that 5 x 7 48?" (no).
• In general we can apply this technique to most
  (if not all) problems, simplifying formal
  treatment of problems.

32
Traveling Salesperson Recognition

• The corresponding recognition problem is
• Given a number n of cities, n?3, a non-negative
  n x n distance matrix of integers Ccij, and a
  non negative integer L Is there a closed tour
  passing through every city exactly once, with
  total length L?
• A solution to the original TSP problem is a tour
  of specific length which we can immediately
  compare to L and so also solve TSP-rec. Conversely
  , suppose we have an algorithm, say A, for the
  TSP-rec problem. Then, we could produce a
  solution for the original problem by looping
  through values for L.

33
The Polynomial Time Class P
The class of languages that can be decided by a
single tape TM in polynomial time is denoted by P
(DTIME stands for deterministic time solvable)
Many, many problems are in P.The problems in P
are efficiently solvable.
34
Robustness of P
In general, every kind of TM (k-tape, r-heads,
etc.)can be solved by a standard, single tape TM
with only polynomial time/space overhead.
As a result, the poly-time class P does not
dependon the specific computational model that
you use.
If some TM can solve A in poly-time, then A?P. As
we have seen, we can extend the TM-model in many
ways
35
In-Class ExerciseExercise 12.1.5 page 308

• Work in groups of 2 or 3
• Answer on next slide (no peeking)

36
Knapsack Optimization

• We have unlimited amounts of n items of sizes Si,
  i1,...,n and values Vi, i1,...,n and also a
  container of capacity C. The goal is to load the
  container with X1 pieces of item 1, X2 pieces of
  item 2 etc., so that the total value is
  maximized. This is the integer Knapsack problem
  and the formulation is
• Maximize X1V1X2V2...XnVn
• subject to X1S1X2S2...XnSn ? C
• where X1,X2,...,Xn are the decision variables
  (how many of a particular item to load) and take
  non-negative integer values.

37
Knapsack Recognition

• The corresponding recognition problem (Integer
  Knapsack-Rec) is
• Given two positive integers K, C, and Si,Vi,
  i1,...,n, is there an assignment of non negative
  integers to the variables X1,X2,...,Xn such that
• X1V1X2V2...XnVn K and
• X1S1X2S2...XnSn ? C
• Again, no one has found a good solution to this.