Todays Lecture

1
Todays Lecture

• Basic Turing Machines
• definition some examples
• configuration, yields relation
• Turing Machines as Algorithms
• TM decides a language - recursive languages
• TM accepts a language - recursively enumerable
  languages
• TM computing a string function
• input/output representation

2
Todays Lecture II

• Turing Machines with Multiple Strings
• definition some examples
• definition of time complexity, time complexity
  classes
• O(n2) simulation of k-string TM by a 1-string TM
• O(n log n) simulation (hint) of k-string TM by a
  2-string TM
• Linear Speedup
• any TM can be sped up by a constant factor
• Space Bounds
• input-output TM
• palindromes in O(log n) space
• constant-compression

3
Turing Machines - Definition

• TM formal definition
• quadruple M (K, S, d, s)
• K finite set of states
• S finite set of symbols, disjoint with K
• always contains blank _ and the first symbol D
• d transition function, the program of the TM
• K x S K U h, yes, no x S x , ,
  
• s initial state (from K)
• M(x) yes, no, output string y, or

D
i
n
p
u
t
_
_
_
_
_
_
_
s
4
Turing Machine- Example
TM for incrementing a binary number

• Does it actually work?
• what about 11111?
• lets correct it
• Lets do a TM for palindromes

5
TM Computation

• configuration complete description of the
  current state of the computation
• TM state, status of the string, position of the
  cursor
• (q, w, u)
• Configuration (q,w,u) yields (q,w,u) in one
  step
• intuitively, one step from (q,w,u) results in
  (q,w,u)
• define formally
• yields transitive closure of yields
• execution sequence of configurations, starting
  with the initial configuration, and i-th
  configuration yields the next one in one step

6
TMs recognizing languages
Language a subset of (S-_) Let x be the
input for a TM M. TM M decides language L if for
any x from (S-_) M(x)
yes if x is in L and M(x) no if x is not it
L. If such M exists, L is called recursive
language. M accepts L if for any x in L M(x)
yes, and for x not in L M does not halt. If
such M exists, L is called recursively enumerable
language. Proposition If L is recursive, it is
recursively enumerable.
7
TMs computing functions

• f (S-_) S
• TM M computes f, for any string in (S-_), M(x)
  f(x).
• If such M exists, then f is called a recursive
  function.
• TM is a general machine, because any
  computational problem can be coded as a string
  function
• just choose a string representation of the
  input/output data structures
• all reasonable representations are polynomially
  related
• note unary representation of integers is NOT
  reasonable

8
TMs with multiple strings
Motivation single string TMs can be quite
unwieldy, perhaps more work strings might
help. k-string TM a TM with k strings, input is
on the first string, all other strings start
empty, output is on the last string. K, S and s
are the same as in standard TMs, d looks at the
symbols below all heads, and moves all
heads. Note that 1-string TM is exactly the
standard TM. Exercise/example how to do
palindromes using a 2-string TM?
9
Time Complexity

• Of a single computation of k-string TM M on input
  x the number of steps to reach yes, no or
  halt (infinity if M does not halt).
• Of the TM
• let f(n) 0,1,2, 0,1,2,
• M operates within time f(n) if for any input
  string x, the time required by M is at most
  f(x).
• Note that this is worst-case time complexity.
• TIME(f(n))
• the class of all languages decided by
  multistring TMs operating in time f(n)
• our first complexity class

10
k-strings vs 1-string

• 1-string TM palindrome solution O(n2)
• 2-string TM palindrome solution O(n)
• Perhaps we can find better 1-string solution?
• see exercises 2.8.4 and 2.8.5 for hints why n2
  is the lower bound for 1-string TMs
• So,2-strings are quadratically better the
  1-string.
• would more strings help even more?
• are there problems where 2(or k) strings speed
  the TM more then quadratically
• Lets simulate k-string TM on a 1-string TM
• the complexity of the simulation will give us
  answers to those questions

11
k-strings vs 1-string

• Theorem Given any k-string TM M operating within
  time f(n), there is a 1-string TM M operating
  within time O(f2(n)) such that for all inputs x
  M(x) M(x).
• Proof (ideas)
• store the content of all k-strings within the
  single string of M, as well as positions of the
  cursors
• decide on encoding
• simulate one step of M
• scan the whole string to collect (into the
  state) the symbols below the heads
• traverse the whole string again to update the
  head positions and written symbols

12
k-strings vs 1-string

• Technical details
• need new symbols to delimit ends of tapes and to
  indicate head/cursor positions
• might need to shift the end of the tape to the
  right in order to insert a new _ symbol (when
  moving past the end of one of the tapes)
• think carefully what new states are needed and
  how does d look like
• Complexity
• note that none of Ms strings becomes longer then
  f(n), i.e. the whole Ms string is no longer then
  kf(n).
• simulating one step takes O(k2f(n)) steps
• together O(k2f2(n)) O(f2(n))

13
k-strings vs 1-string

• Discussion
• adding more strings allowed much more convenient
  programming, but did not result in
  superpolynomial speedup
• What about k-strings vs l-strings, for lgt1?
• for sure they can be simulated quadratically,
  but can we do better?
• yes, k-string TM operating in time f(n) can be
  simulated by 2-string TM in time f(n) log f(n)
• we might come back to this at the end of the
  lecture

14
Linear speedup

• Theorem Let L be from TIME(f(n)). Then for any
  egt0, L is from TIME(f(n)), where f(n)
  ef(n)n2.
• Intuitively, we can speed up any TM by a constant
  factor
• similar to speeding-up a computer by using
  larger words
• Proof (ideas)
• we show M that in one step simulates m steps of
  M
• set S S U Sm (contains all m-tuples of
  symbols of M)
• transform the input into condensed form using
  S
• the linear term comes from here
• in one meta-step M reads also the neighbouring
  symbols (m-tuples), and then simulates (in at
  most 2 steps) m-steps of M

15
Linear speedup
Consequence we dont have to worry about
constant factors and are free to use O()
notation So, TIME(nk) will mean the class of
languages decided in time O(nk) Skgt0 TIME(nk)
P, the class of languages decided in polynomial
time
16
Space Bounds

• How to define the space complexity of a TM?
• Possible solutions
• sum of string sizes
• maximum of string sizes
• Problem
• includes the input and possibly output, which do
  not properly reflect how much memory was needed
• Solution
• TMs with input and output

17
TMs with input and output

• Idea
• separate the input and output from the working
  memory
• dont allow to write the input tape, dont allow
  to read the output tape
• Definition
• k-string TM with input and output is a k-string
  TM with the following restrictions on d
• Whenever (d(q,s1,s2, , sk) (p,r1, D1, r2, D2,
  , rk, Dk) then
• (a) r1 s1, (b)Dk is not left and (c) if s1 _
  then D1 is not left
• Note For any k-string TM M operating in time
  f(n) there is an equivalent (k2) string TM M
  with input and output which operates in time
  O(f(n)).

18
TMs with input and output - Example
Design a TM with input-output that solves
palindromes using as small work memory as
possible. Any ideas?
19
Space Complexity

• Define the space required by a TM M with an input
  and output as the sum of the sizes of the working
  tapes (excluding the input and output tapes).
• If f is a function from N to N, we say that M
  operates within space bound f(n) is for any input
  x M requires space at most f(x)
• SPACE(f(n)) the class of languages decided by
  turing machines operating within space bound
  f(n).
• Notes
• palindromes are in SPACE(log n)
• SPACE(log n) is usually referred to as L

20
Random Access Machines

• A model of computation much closer to real
  computers then TMs.
• finite program consisting of instructions
• infinite array of integer registers r0, r1,
• program counter k, automatically incremented,
  unless stated otherwise
• three modes of addressing x, x, x
• instructions READ, STORE, LOAD, ADD, SUB, HALF,
  JUMP, JPOS, JZERO, JNEG, HALT
• initially all registers are 0, and k 1
• input finite array of input registers i1, i2,
  , in
• upon halting, the accumulator contains the
  result
• the registers can contain arbitrarily large
  integers

21
Random Access Machines
22
Random Access Machines

• We can define configuration of RAM
• C (k, R), where R is the content of the
  registers
• Similarly, yields relation can be defined between
  configurations
• Time of computation on a given input
• total number of steps (i.e. each instruction
  takes 1 time step) until HALT instruction is
  executed
• Similarly as in TMs, we can define what it means
  for RAM to work in time O(f(n)). (Note the
  input size is not of input registers, but the
  total number of bits stored in input registers.)
• Question How powerful is RAM?
• can RAMs compute functions/languages that TMs
  cannot?
• can RAMs compute functions/languages
  significantly faster then TMs?

23
RAMs vs TMs, round 1

• Theorem Suppose that language L is in
  TIME(f(n)). Then there is a RAM program which
  decides L in time O(f(n)).
• We just have to simulate the TM on a RAM.
• registers 1,2,3 will be used for simulation
• registers 4, 5, contain the content of the
  TMs tape
• for each pair (state,symbol) there is a
  constant-size subprogram that implements the
  action of the TM
• update the register containing the overwritten
  symbol
• move the head (its position stored in r1)
• jump to the code for the resulting state (change
  state)
• code for yes store 1 in r0 and HALT,no
  similar, but 0

24
RAMs vs TMs, round 2

• Theorem If a RAM program P computes a fuction f
  in time f(n), then there is a 7-string TM M which
  computes f in time O(f(n)3) .
• This means RAMs can compute only the same
  functions as TMs, and cannot do that
  exponentially faster, despite the huge registers.
• How to prove this theorem?
• we have to simulate the RAM on a TM

25
RAMs vs TMs, round 2 cont.

• For any RAM P we have to find a TM M that
  computes the same function.
• M will use the following tapes
• tape 1 stores the input string, never
  overwritten
• tape 2 store the contents of the registers,
• as a list (b(i), b(ri)), terminated by endmarker
• if a register ri is updated, the pair (b(i),
  b(ri)) is overwritten by blanks and the new pair
  is appended at the end
• tape 3 stores the program counter
• tape 4 holds the register address currently
  sought
• tapes 5,6,7 working tapes, holding operands
  result

26
RAMs vs TMs, round 2, cont.

• One instruction of RAM is simulated as follows
• in one scan (or two scans if the operand uses
  indirect addressing) of tape 2 the operands of
  the instruction are retrieved
• compare the content of tape 4 with the pair
  (b(i), b(ri)) currently found on tape 4, if
  matches, retrieve b(ri) into the appropriate
  operand tape
• execute the operation using the working tapes
• Lemma After i-th step of P on input I, the
  contents of any register has length at most
  tl(I)l(B), where B is the largest integer
  referred to in the program.
• proof by induction in time, induction step by
  case analysis of the instruction
• maximal register content grows only in ADD, and
  by at most 1

27
RAMs vs TMs, round 2, cont.

• So, what is the time complexity of the
  simulation?
• How long does it take to simulate one
  instruction?
• the length of the tape 2 is O(f(n)2)
• O(f(n)) pairs, each of length O(f(n))
• so fetching operands can be done in O(f(n)2)
  time
• with the help of tape 4
• decoding instruction and involved constants
  O(1) time
• executing the simple arithmetic operations on
  operands of size O(f(n)) takes time O(f(n))
• There are f(n) instructions to simulate, so
  overall simulation time is O(f(n)3).

28
Non-deterministic TMs

• Quadruple M (K, S, d, s)
• K, S and s are the same as in deterministic TM
• d is not a function, but relation
• d K x S -gt 2(K x S x D)
• i.e. there are many possible successor
  configurations
• Configuration is defined the same as in
  deterministic TM
• Yields is different a given configuration can in
  one step yield many possible configurations
• Execution of deterministic TM can be captured as
  a path of configurations.
• Execution of non-deterministic TM can be captured
  as a tree of configurations.

29
Non-deterministic TMs
Crucial difference What does it mean to solve
the problem Definition Let L be a language and
N a nondeterministic TM. We say that N decides L
for any x from S it holds x is from L if and
only if (s, D, x) -gtN (yes, w, u) for some w
and u. Informally If there exists an accepting
execution path in the execution tree of
N. NTIME(f(n)) the depth of the execution
tree NP U NTIME(nk) Note that a deterministic
TM is a special case of non-deterministic TM. We
can easily define k-tape non-det TMs.
30
Non-deterministic TMs - Example

• TSP(B) Given a matrix representing distances
  between cities, is there a traversal visiting all
  cities with total length at most B?
• Note We do not know how to solve this
  deterministically in polynomial time.
• Can we solve TSP(B) in polynomial time using
  non-det TMs?
• Easily
• write an arbitrary string S, no longer then the
  input, to the working tape
• check whether S represents a traversal sequence
  visiting all cities
• compute the length of S- traversal and check
  whether it is less then B

31
Non-det TMs vs det TMs
Question Is there a problem that can be solved
by non-det TMs but cannot be solved using det
TMs? Note TSP can be solved using det TMs, just
a bit slowly. Theorem Suppose a language L is
decided by a non-det TM N in time f(n). Then
there is a det 3-string TM M that decides L in
time O(cf(n)) for some constant c depending on
N. Note We can rephrase this as NTIME(f(n)) is
a subset of Ucgt1 TIME(cf(n)) This means we can
simulate non-det TMs with exponential slowdown.
Can we simulate them with only polynomial
slowdown? Well, that is exactly P NP ? problem.
32
Non-det TMs vs det TMs

• Proof We need to simulate non-det TM N using det
  TM M. Any ideas?
• traverse the execution tree of N using BFS
• any given node corresponds to a sequence of
  non-deterministic choices
• M simulates N for given sequence, decides yes
  if N decides yes, otherwise considers the next
  sequence
• if the tree finishes (all branches halted, but
  no yes, then decide no
• The size of the tree is at most df(n), where d is
  the maximal number of non-det choices, and
  simulating one execution path is polynomial in
  its length.

33
Chapter 3 Undecidability
We promised that TMs are a good model of general
computers. But so far A TM per problem, i.e. a
TM corresponds to an algorithm, not a computer.
What is needed A TM that can solve all problems
that can be solved by TMs. In other words A
Universal TM, that can simulate all other TMs.
34
Universal TMs

• Universal TM U Take as an input a TM M and its
  input x and output the output of M when being run
  on input X.
• U(Mx) M(x)
• How to design U?
• TMs have different alphabet/state sizes, and for
  every k there is a TM with more symbols/states
• encode K and S in binary as integers
• S 1,2,S, K S1, SK
• few more integers for special symbols (lt-, -gt,
  -, h, yes, no)
• all encoded in binary using fixed-width words,
  with leading 0s where needed
• d is encoded as sequence of pairs ((q, r),(p,t,
  D)), where (,) and , are symbols of U and
  q,r,p,t,D are encoded as above

35
Chapter 3 Universal TMs

• How U on input Mx simulates M?
• note both M and x are encoded as above
• use 2nd string to store the simulated
  configuration of M, in format (w, q, u) (the
  state between the parts of the tape)
• to simulate one step of M
• scan the input tape until a tuple ((q, r),(p,t,
  D)) is found in the decription of d
• check on the 2nd tape whether r matches the
  symbol below the head, if not, look for another
  tuple matching the state q
• if yes, apply the changes
• if in terminating state, terminate accordingly,
  otherwise continue the simulation
• if the input is incorrect, just never halt

36
Undecidability

• Are there problems that cannot be solved by any
  algorithm?
• Are there languages that cannot be decided by a
  Turing machine?
• in our quest for defining computation/algorithms
  we hoped that there are no such
  problems/languages
• but there are more languages (all subsets of S)
  then Turing machines (just few strings encode
  proper TMs)
• so yes, there must be such languages
• hopefully they are obscure and meaningless
  random mess, with no structure/meaning
• Unfortunately, there is a language with quite
  good definition and applicability The Halting
  Problem

37
The Halting Problem
The problem Will a TM M halt on input
x? Language representation (language H) all Mx
in the encoding used by the universal TM U such
that M halts on x. Good news H is recursively
enumerable. Proof Universal TM U is the TM that
accepts H It simply simulates M on x, when M
halts (in either h, yes or no state), U halts
in yes state. Bad news There are no more good
news.
38
The Halting Problem

• Theorem H is not recursive.
• Proof By contradiction. Assume H is recursive,
  i.e. there is a TM MH deciding H. Modify MH to
  get machine D(M) as follows
• on input M, simulate MH on input MM until it is
  about to halt (it eventually will, as MH decides
  H), then cycle forever if MH accepts, or halt, if
  MH rejects.
• What does D do on input D?
• if it does not halt, then from the definition of
  D MH accepts DD, then from the definition of MH
  D halts on input D, contradiction
• if it halts, then from the definition of D MH
  rejects DD, i.e. D does not halt on input D,
  again contradiction

39
The Halting Problem

• Well, maybe the halting problem is a bad
  exception.
• Well, maybe not
• Theorem The following languages are not
  recursive
• M M halts on all inputs
• Mx there is a y such that M(x) y
• Mx the computation of M on input x uses all
  states of M
• Mxy M(x) y
• In fact, essentially any non-trivial property of
  TMs is undecidable!
• Lets show the first one

40
The Always -Halting Problem
Theorem The language M M halts on all inputs
is not recursive. Proof Trivially, by reducing
Halting to Always-Halting Given Mx, we
construct M as follows M(y) if (xy) then
M(x) else halt. Obviously, M always halts if and
only if M halts on x. In fact, H is complete for
the class of recursively enumerable problems if
we had an algorithm for H, we can have algorithm
for any recursive enumerable language L Let M
be a TM accepting L. Deciding whether x is from L
can be done by simply telling whether Mx is in H.
41
Other results

• Theorem If L is recursive, then also the
  complement of L is recursive.
• just replace yes with no and vice versa
• R (the class of recursive languages) is therefore
  closed under complement. However, RE is not
• Theorem L is recursive if and only if both L and
  complement of L are recursively enumerable
• gt From previous theorem
• lt If both are recursively enumerable by machines
  M and M, respectively, simulate M and M in an
  interleaved manner until one of them accepts.

42
Where did recursively enumerable came from?

• Define language E(M) The set of strings x that
  appear at some moment at the end of the tape of M
  running on empty input, separated by _(i.e. the
  tape looks Dy_x_)
• We say that E(M) is the language enumerated by M.
• Theorem L is recursively enumerable if and only
  if there is a TM M such that L E(M).
• lt Checking whether x is in L can be done
  easily just simulate M and accept if x appear at
  the end of the tape.
• gtLet L be accepted by TM M, we need to
  construct M such that x appears at some moment
  at the end of the tape iff it is accepted by M.
• for i1,2, simulate M on inputs 0,1,..i for up
  to i steps, if M accepts, write the current input
  at the end of the tape

43
Can we decide anything useful about TMs?
Rices Theorem NO! Formally Suppose C is a
proper, non-empty subset of the set of all
recursively enumerable languages, not containing
?. Then, the following problem is undecidable
Given TM M, is L(M) in C? (View C as a set of
languages satisfying some property.) Proof idea
By reduction to HALTING Let L be any non-empty
language in C. For an input x, construct
Mx Mx(y) if MH(x) yes then ML(y) else loop
forever On input y Mx simulates MH on x. If
MHyes, start simulating ML on y, otherwise
cycle forever.
44
Rices theorem
Claim L(Mx) is in C if and only if x is in
H. Therefore, deciding whether L(Mx) is in C
would solve H. Proof of the claim If x is in H
then Mx accepts language L, which is in C by
assumption. If x is not in H then Mx cycles
forever and accepts the empty language ?, which
is (by assumption) not in C.
45
Recursive Inseparability
Definition L1 and L2 are recursively inseparable
if there does not exist a recursive language R
such that L2 ? R and L1 ? R ?. Note H and
complement of H are recursively
inseparable. Theorem Let L1 MM(M) yes
and L2 MM(M) no. Then L1 and L2 are
recursively inseparable. Consider, by
contradiction, that there is such recursive
language R, and let MR be the TM deciding R. What
is MR(MR)?
46
Recursive Inseparability
Corollary Let L1 MM(e) yes and L2
MM(e) no. Then L1 and L2 are recursively
inseparable. Proof By reduction If we could
separate L1 and L2 by a recursive language R
then we could separate L1 and L2 of the previous
theorem. Let M be a TM that on any input x
generates Ms description(encoding) and then
simulates M on x.