91.304 Foundations of Theoretical Computer Science

1
91.304 Foundations of (Theoretical) Computer
Science

• David Martin
• dm_at_cs.uml.edu

This work is licensed under the Creative Commons
Attribution-ShareAlike License. To view a copy of
this license, visit http//creativecommons.org/lic
enses/by-sa/2.0/ or send a letter to Creative
Commons, 559 Nathan Abbott Way, Stanford,
California 94305, USA.
2
Exam 2

• mean 33.6
• median 35
• stdev 9.6
• max 47

3
Missing Content

• Variants of the TM that are equivalent to the
  "real" (Definition 3.1, ?1) model
• Multi-track machines
• Can enlarge the tape alphabet ? to keep track of
  multiple things simultaneously
• Multi-head machines
• Can read from multiple independent positions at
  once, write to them, and then move them
  independently
• Our technique for converting one of these into
  the "real" model was slow at run time but did the
  right thing
• Multi-tape machines
• A combination of multi-track and multi-head,
  proven equivalent in textbook's Theorem 3.8
• Nondeterministic TMs
• Proven equivalent in Theorem 3.10
• We didn't fully define these models..
• For instance, we didn't explicitly state how the
  relation works

4
Another model

• Definition An enumerator E is a 2-tape TM with a
  special state named qp ("print")
• The language generated by E isL(E) x2?
  (q0 t, q0 t ) ( u qp v, x qp z )
  for some u, v, z 2 ?
• Here the instantaneous description is split into
  two parts (tape1, tape2)
• So this says that "x appears to the left of the
  tape 2 head when E enters the qp state"
• Note that E always starts with a blank tape and
  potentially runs forever
• Basically, E generates the language consisting of
  all the strings it decides to print
• And it doesn't matter what's on tape 1 when E
  prints

5
Reminder

• The decidable languages ?0
• The recognizable languages ?1

6
Theorem 3.13

• L 2 ?1 , LL(E) for some enumerator E (in other
  words, enumerators are equivalent to TMs)
• Proof First we show that LL(E) ) L2?1. So assume
  that LL(E) we need to produce a TM M such that
  LL(M). We define M as a 3-tape TM that works
  like this

• input w (on tape 1)
• run E on M's tapes 2 and 3
• whenever E prints out a string x, compare x to w
  if they are equal, then acceptelse goto 2 and
  continue running E

7
Theorem 3.13 continued

• Now we show that L2?1 ) LL(E) for some
  enumerator E. So assume that LL(M) for some TM
  M we need to produce an enumerator E such that
  LL(E). First let s1, s2, ? be the
  lexicographical enumeration of ?. E behaves as
  follows

• for i1 to 1
• run M on input si
• if M accepts si then print string si(else
  continue with next i)

DOES NOT WORK!!
8
Theorem 3.13 continued

• Now we show that L2?1 ) LL(E) for some
  enumerator E. So assume that LL(M) for some TM
  M we need to produce an enumerator E such that
  LL(E). First let s1, s2, ? be the
  lexicographical enumeration of ?. E behaves as
  follows

exactly t steps of the relation

• for t1 to 1 / t time to allow /
• for j1 to t
• compute the instantaneous description uqv in M
  such that q0 sj t uqv, if M is still running
  then
• if q qacc then print string sj(else continue)

9
Theorem 3.13 continued

• First, E never prints out a string sj that is not
  accepted by M
• Suppose that q0 s5 27 u qacc v (in other
  words, M accepts s5 after exactly 27 steps)
• Then E prints out s5 in iteration t27, j5
• Since every string sj that is accepted by M is
  accepted in some number of steps tj, E will print
  out sj in iteration ttj and in no other
  iteration
• This is a slightly different construction than
  the textbook, which prints out each accepted
  string sj infinitely many times

10
Closure properties of ?0 and ?1

• ?1 is closed under , Å, , , reversal
• Proofs for and Å are similar to the NFA
  constructions we used, if you use a 2-tape TM
• Proof for reversal is also easy with a 2-tape TM
• and are harder
• ?0 is closed under all of these operations and
  complement as well

11
?0 closed under complement

• Proof Suppose L2?0. Then LL(M1) for some
  decider M1. We want to show there's a decider M2
  such that L(M2) Lc, whence (!) Lc 2 ?0.
• This is easy we just set M2 to be M1 with qacc
  and qrej swapped. This works because M1 never
  loops (forever) on any input it always reaches
  either qacc or qrej, and so M2 does the opposite
  thing from M1 in terms of accepting or rejecting
  the string.

12
Proof does not work for ?1

• Without the guarantee that M1 never loops on any
  input, this conversion does not produce an M2
  such that L(M2) L(M1) c.
• Because if x L(M1), we want x2 L(M2), but...

13
Preview ?1 is not closed under complement

• The previous discussion just showed that the
  attempted proof of closure under complement
  failed
• It didn't show that ?1 is not closed under
  complement
• However it is in fact true that ?1 is not closed
  under complement will see an actual proof later

14
Preview a non-recognizable L

• This all means that some L exists that is not
  recognized by any TM
• What does it look like?
• Is it important? YES, because of
  Church-Turing Thesis

15
Historical background

• Great concern in the early 20th century about the
  notion of effective computability.
• When someone defines what they want a function to
  mean, is it a reasonable definition? Only if
  its "effectively computable"
• Articulated by Hilbert in 1900 at the
  International Congress of Mathematics in Paris
• Note that it's easy to use a TM to define a
  function f? ! ? you just take the output to
  be whatever's on the tape when the machine halts
• Alonzo Church (with Stephen Kleene) 1932-34,
  pursued effective computability via
  ?-definability ("lambda")
• Was a proposal for a new foundation for writing
  logic and mathematics. Ultimately didnt resonate
  well enough to be widely adopted for that
  purpose, but certainly plays an important role
  even today
• Example lambda notation function ? x. xy
• Specifies "?-conversion" (renaming of bound
  variables parameter-argument matching) and
  "?-conversion" (evaluation by substitution
  running the function)
• Given a ?-function f and input n, there is a
  deterministic procedure for working towards a
  normal form namely, the output of the function

16
Church's Thesis

• In 1934 Church claimed that the ?-definable
  functions are all the effectively computable
  functions
• Kleene (his student) called it Churchs Thesis
• Gödel and Herbrand 34, developed a notion of
  effective computability based on provability in
  formal arithmetic system
• Previously, in 31, Gödel had proved that no
  sufficiently sophisticated/interesting formal
  system could be both consistent and complete.
  Still celebrated as a major breakthrough in
  mathematics.
• Gödel and Herbrand started with a formal system
  of arithmetic (axioms rules for proof).
• And said that a function is effectively
  computable if for each f(m)n we can prove that
  f(m)n within the system.
• Gödel did not at first accept Churchs thesis.

17
Kleene's partial recursive functions

• Kleene defined another important formalism in
  36
• A function is partial recursive if it can be
  generated from 0, projection, successor,
  substitutions, primitive recursion, and
  minimization.
• Kleene in 36 proved that these partial recursive
  functions are equivalent to the ?-definable
  functions in terms of what functions can be
  expressed
• In other words, a function in one model could be
  converted into a function in the other model,
  just like weve done on a smaller scale with
  finite automata variants and TM variants
• However, these forms of "effective computability"
  are all very abstract mathematical notions

18
Turing's 1936 paper

• Alan Turing was basically unaware of all of this
  work when he wrote his paper "On Computable
  Numbers with an application to the
  Entscheidungsproblem."
• Entscheidungsproblem One of 23 famous problems
  posed by Hilbert in 1900 at the International
  Congress of Mathematics
• Namely find a mechanical procedure that
  determines the truth or falsehood of an
  arbitrary, carefully-specified, mathematical
  assertion.
• Turing proved that there was no procedure to
  solve the Entscheidungsproblem
• Well see this proof, more or less
• Along the way he defined his notion of
  computation and described Universal Turing
  Machines (which well also see)

19
Turing's 1936 paper excerpt

• Alan. M.Turing, "On Computable Numbers, With an
  Application to the Entscheidungsproblem,'' Proc.
  London Math. Soc., 2(42) (1936), 230-265 "A
  correction'' ibid, 43, 544-546. Google search
  9. The extent of the computable numbers.
• No attempt has yet been made to show that the
  computable numbers include all numbers which
  would naturally be regarded as computable. All
  arguments which can be given are bound to be,
  fundamentally, appeals to intuition, and for this
  reason rather unsatisfactory mathematically. The
  real question at issue is What are the possible
  processes which can be carried out in computing a
  number?

20
Turing's 1936 paper excerpt

• The arguments which I shall use are of three
  kinds.
• A direct appeal to intuition.
• A proof of the equivalence of two definitions (in
  case the new definition has a greater intuitive
  appeal).
• Giving examples of large classes of numbers which
  are computable.
• ?
• Computing is normally done by writing certain
  symbols on paper. We may suppose this paper is
  divided into squares like a child's arithmetic
  book. In elementary arithmetic the
  two-dimensional character of the paper is
  sometimes used. But such a use is always
  avoidable, and I think that it will be agreed
  that the two-dimensional character of paper is no
  essential of computation.
• I assume then that the computation is carried
  out on one-dimensional paper, i.e. on a tape
  divided into squares. I shall also suppose that
  the number of symbols which may be printed is
  finite. If we were to allow an infinity of
  symbols, then there would be symbols differing to
  an arbitrarily small extent.  The effect of this
  restriction of the number of symbols is not very
  serious. It is always possible to use sequences
  of symbols in the place of single symbols. Thus
  an Arabic numeral such as 17 or 999999999999999
  is normally treated as a single symbol.

21
Turing's 1936 paper excerpt

• Similarly in any European language words are
  treated as single symbols (Chinese, however,
  attempts to have an enumerable infinity of
  symbols). The differences from our point of view
  between the single and compound symbols is that
  the compound symbols, if they are too lengthy,
  cannot be observed at one glance.

22

• Manners are not taught in lessons, said Alice.
  Lessons teach you to do sums, and things of that
  sort.
• And you do Addition? the White Queen asked.
  What's one and one and one and one and one and
  one and one and one and one and one?
• I don't know, said Alice. I lost count.
• She can't do Addition, the Red Queen
  interrupted.

Excerpt Through the Looking Glass, Lewis Carroll
23
Turing's 1936 paper excerpt

• This is in accordance with experience. We cannot
  tell at a glance whether 9999999999999999 and
  999999999999999 are the same.
• The behaviour of the computer at any moment is
  determined by the symbols which he is observing.
  and his state of mind at that moment. We may
  suppose that there is a bound B to the number of
  symbols or squares which the computer can observe
  at one moment. If he wishes to observe more, he
  must use successive observations. We will also
  suppose that the number of states of mind which
  need be taken into account is finite. The reasons
  for this are of the same character as those which
  restrict the number of symbols. If we admitted an
  infinity of states of mind, some of them will be
  arbitrarily close and will be confused. Again,
  the restriction is not one which seriously
  affects computation, since the use of more
  complicated states of mind can be avoided by
  writing more symbols on the tape.

24
Turing's 1936 paper excerpt

• Let us imagine the operations performed by the
  computer to be split up into simple operations
  which are so elementary that it is not easy to
  imagine them further divided. Every such
  operation consists of some change of the physical
  system consisting of the computer and his tape.
  We know the state of the system if we know the
  sequence of symbols on the tape, which of these
  are observed by the computer (possibly with a
  special order), and the state of mind of the
  computer.
• We may suppose that in a simple operation not
  more than one symbol is altered. Any other
  changes can be set up into simple changes of this
  kind. The situation in regard to the squares
  whose symbols may be altered in this way is the
  same as in regard to the observed squares. We
  may, therefore, without loss of generality,
  assume that the squares whose symbols are changed
  are always observed squares.
• Besides these changes of symbols, the simple
  operations must include changes of distribution
  of observed squares. The new observed squares
  must be immediately recognisable by the computer.
  I think it is reasonable to suppose that they can
  only be squares whose distance from the closest
  of the immediately previously observed squares
  does not exceed a certain fixed amount. Let us
  say that each of the new observed squares is
  within L squares of an immediately previously
  observed square.

25
Turing's 1936 paper excerpt

• The simple operations must therefore include
• (a) Changes of the symbol on one of the observed
  squares.
• (b) Changes of one of the squares observed to
  another square within L squares of one of the
  previously observed squares.
• It may be that some of these changes necessarily
  involve a change of state of mind. The most
  general single operation must therefore be taken
  to be one of the following
• A. A possible change (a) of symbol together with
  a possible change of state of mind.
• B. A possible change (b) of observed squares,
  together with a possible change of state of mind.

26
Turing's 1936 paper excerpt

• We may now construct a machine to do the work of
  this computer.
• To each state of mind of the computer corresponds
  an m-configuration of the machine. The machine
  scans B squares corresponding to the B squares
  observed by the computer. In any move the machine
  can change a symbol on a scanned square or can
  change anyone of the scanned squares to another
  square distant not more than L squares from one
  of the other scanned squares. The move which is
  done, and the succeeding configuration, are
  determined by the scanned symbol and the
  m-configuration.

27
Turing's 1936 paper excerpt

• We suppose, as in I, that the computation is
  carried out on a tape but we avoid introducing
  the state of mind by considering a more
  physical and definite counterpart of it.
• It is always possible for the computer to break
  off from his work, to go away and forget all
  about it, and later to come back and go on with
  it. If he does this he must leave a note of
  instructions (written in some standard form)
  explaining how the work is to be continued. This
  note is the counterpart of the state of mind.
• We will suppose that the computer works by such a
  desultory manner that he never does more than one
  step at a sitting. The note of instructions must
  enable him to carry out one step and write the
  next note. Thus the state of progress of the
  computation at any stage is completely determined
  by the note of instructions and the symbols on
  the tape.

28
Turing's 1936 paper excerpt

• That is, the state of the system may be described
  by a single expression (sequence of symbols),
  consisting of the symbols on the tape followed
  by A (which we suppose not to appear elsewhere)
  and then by the note of instructions. This
  expression may be called the state formula. We
  know that the state formula at any given stage is
  determined by the state formula before the last
  step was made, and we assume that the relation of
  these two formulae is expressible in the
  functional calculus.
• In other words we assume that there is an axiom U
  which expresses the rules governing the behaviour
  of the computer, in terms of the relation of the
  state formula at any stage to the state formula
  at the proceeding stage. If this is so, we can
  construct a machine to write down the successive
  state formulae, and hence to compute the required
  number.

29
Turing's 1936 paper

• So Turings idea of effective computation was a
  description of what humans could do.
• He submitted his paper in May, and in August
  submitted an appendix proving that Churchs
  effective computability (?-definable) was
  equivalent to his own (TM-computable).
• After seeing Turing's proof, Gödel became
  convinced that they were all talking about the
  same thing.
• Equivalence between these models then became
  known as the Church-Turing thesis.

30
The Church-Turing Thesis

• Any algorithmic-functional procedure that can be
  done at all can be done by a Turing machine
• This isn't provable, because algorithmic-function
  al procedure is vague. But this thesis (law)
  has not been in serious doubt for many decades
  now
• TMs are probably the most commonly used low-level
  formalism for algorithms and computation
• Commonly used high-level formalisms include
  pseudocode and all actual programming languages.
  By Church-Turing thesis, these are all equivalent
  in terms of what they can (eventually) do.
• Of course they have different ease-of-programming
  and time/memory efficiency characteristics.

31
Alan Turing biographical notes

• Turing was 24 at the time he published his 1936
  paper
• 1939, Turing was working at Bletchley Park on
  decrypting Enigma, the encryption device used by
  Germany in WWII.
• The existence of the Collosus decryptor was kept
  secret through most of WWII
• This work basically made him a war hero
• 1939, John Vincent Atanasoff and Clifford Berry
  designed and built the early ABC computer at Iowa
  State.
• The first electronic digital computer
• Personnel disruptions due to the war caused the
  patent application to be lost before it was
  submitted.
• 1941, Mauchly and Eckert visited ISU and went on
  to built ENIAC in 46
• The first general-purpose electronic digital
  computer
• Patent fight 1967 resolved in 1973, invalidating
  ENIAC patent
• Followed next day by Saturday Night Massacre

32
Alan Turing biographical notes

• 1945 (National Physics Lab), 1948- (Computing
  Machine Laboratory) Turing continued to worked on
  computing, artificial intelligence, and later,
  biology.
• "Turing test" for successful artificial
  intelligence
• ACM Turing award most prestigious award in
  computer science
• 1952 Turing was arrested after reporting burglary
• Offered a choice between prison and
  "organo-therapy" (hormone treatments) to "cure"
  his "deviance". Shrugged and went along with
  that, an interesting science experiment...
• Turing seems to me a prototype of the modern
  western gay person (at a time before the word
  "gay" meant that) fully integrated into
  mainstream society and enthusiastically oppressed
  by it. By all accounts, he had a positive
  outlook on life
• 1954 Turing was found dead at age of 42 with an
  apparently poisoned apple partially eaten at his
  bedside. (He kept cyanide in the house for
  various experiments.) Believed to be suicide but
  not fully investigated