CS 208: Computing Theory

1
CS 208 Computing Theory

• Assoc. Prof. Dr. Brahim Hnich
• Faculty of Computer Sciences
• Izmir University of Economics

2
Contact Info

• Room 415
• Email brahim.hnich_at_ieu.edu.tr
• Office hours
• Tuesdays 1416

3
Textbooks

• The course lecture are based on the following
  books
• Harry R. Lewis and C.H. Papadimitriou. "Elements
  of the Theory of Computation", ISBN
  0-13-272741-2, Prentice Hall. (Available in
  library)
• Michael Sipser. "Introduction to the Theory of
  Computation", ISBN 053494728X, PWS Publishing
  Company.

4
Textbooks

• The course lecture are based on the following
  books
• Michael R. Garey and David S. Johnson. Computer
  and Intractability", ISBN 0-7167-1045-5, W.H.
  Freeman and Company.
• C.H. Papadimitriou. Computational Complexity",
  ISBN 0-201-53082-1, Addison-Wesley Publishing
  Company, Inc.

5
Grading and exams

• 6 Homeworks 30
• Midterm Exam 30
• Final Exam 40

6
Programme

• Part A Preliminaries
• Introduction
• Mathematical Notations for Computations

7
Programme

• Part B Automata Theory
• Finite State Automata
• Context Free Grammars

8
Programme

• Part C Computability Theory
• Turing Machines
• Decidability and Undecidability

9
Programme

• Part D Complexity Theory
• Time Complexity
• Intractability

10
Programme

• Part E Summary and Concluding Remarks

11
Good Luck!
12
Outline

• Introduction to the course
• Goal
• Organization
• Outline
• Mathematical Notations for Computations

13
What is computation?

• Definition Processing of information based on a
  finite set of operations or rules

14
What is computation?

• Paper and pencil Arithmetic
• 99 - 8 91
• Calculators
• Bar code scanner
• Digital camera
• Computer programs in C and Java

15
What do we want in a theory

• Generality
• Technology-independent
• Abstraction ignores inessential details
• Precision
• Mathematical, formal
• Can prove theorems about computation
• Positive what can be computed
• Negative what cannot be computed

16
So you want to be a good computer scientist/
software engineer?

• Learn the rules
• Algorithms, data structures,
• Learn basic principles
• Data abstraction,
• Understand which problems you can solve with
  computers and which you simply cannot
• and why!

17
So you want to be a good computer scientist/
software engineer?

• Learn the rules
• Algorithms, data structures,
• Learn basic principles
• Data abstraction,
• Understand which problems you can solve
  efficiently with computers and which you cannot!

18
Course Overall Objective

• Understand the fundamental capabilities and
  limitations of computers
• Make a theory out of the idea of computation

19
Why study theory?

• Pragmatic Reasons
• Apply efficient algorithms to tractable problems
• Avoid intractable or impossible problems
• Learn to tell the difference

20
Why study theory?

• Mathematical education
• Automata Theory
• Computability Theory
• Complexity Theory

21
Why study theory?

• Philosophy
• What is computation?
• What is provable?

22
Course of two parts
23
The Good News

• You only have to take this course once
• Except those of you who will not pay attention to
  the lectures
• At the end, you can say that you understand what
  computers are really about
• Good for postgraduate and professional life

24
2nd half of the course
25
The Bad News

• Theory course
• Formal
• Requires mathematics
• Understanding Proofs
• Analytical skills

26
The Bad News

• Theory course
• Formal
• Requires mathematics
• Understanding Proofs
• Analytical skills

But, we will try to make it fun because it is
really interesting!
27
End of Introduction

• Next Topics to be covered

28
Three major topics

• What is a computer?
• Automata Theory
• What can (cannot) be computed?
• Computability Theory
• What can (cannot) be computed efficiently?
• Complexity Theory

29
Three major topics

• What is a computer?
• Automata Theory
• What can (cannot) be computed?
• Computability Theory
• What can (cannot) be computed efficiently?
• Complexity Theory

30

• Complexity Theory

31
Complexity Theory

• Key notion tractable vs. intractable problems

32
Complexity Theory

• Key notion tractable vs. intractable problems

EASY
HARD
33
Complexity Theory

• A problem is a general question
• Description of parameters
• Description of solution
• E.g.
• Input Given a list of integers
• Can you sort the list in ascending order?

34
Complexity Theory

• An algorithm is a step-by-step procedure
• A recipe
• A computer program
• A mathematical object

35
Complexity Theory

• We want the most efficient algorithm as a
  function of problem size
• Fastest (mostly)
• Most economical with memory (sometimes)

36
Example Traveling Salesman Problem

• Given
• A set of m cities
• A set of inter-city distances
• Find a permutation of cities which makes a tour
  of minimum length

37
Example TSP instance
a
9
10
5
c
6
3
b
d
9
(not drawn to scale)
38
Example TSP instance
a
9
10
5
c
6
3
b
d
9
A solution The tour a,b,d,c,a has length 27
39
Problem Size

• What is an appropriate measure of problem size?
• m the number of nodes?
• m(m1)/2 distances?

40
Problem Encoding

• Use an encoding of the problem
• Alphabet of symbols
• Strings abcd//10/5/9//6/9//3

41
Measures

• Problem size length of the encoding
• Time complexity how long an algorithm takes, as
  a function of problem size

42
Time Complexity

• What is tractable?
• A function f(n) is O(g(n)) whenever
• f(n) c. g(n)
• for n gt some constant

43
Time Complexity

• A polynomial-time algorithm is one whose time
  complexity is O(p(n)) for some polynomial p(n)
• e.g. O(n2)

44
Time Complexity

• An exponential-time algorithm is one whose time
  complexity cannot be bounded by a polynomial
• e.g. O(nlog n)

45
Tractability

• Basic distinction
• Polynomial time means tractable
• Exponential time means intractable

46
Tractability
Execution time
10 30 60
n .00001 sec .00003 sec .00006 sec
n2 .00001 sec .00009 sec .00036 sec
n3 .00001 sec .027 sec .216 sec
2n .001 sec 17.9 min 336 centuries
3n 0.59 sec 6.5 years 1.3 1013 centuries
47
Effect of Speed-Ups
Let us wait for faster hardware!
Present problem size 100 times faster 1000 times faster
n N1 100N1 1000N1
n2 N2 10N2 31.6N2
n3 N3 4.64N3 10N3
2n N4 N46.64 N49.97
3n N5 N54.19 N56.29
48
Back to TSP

• Your boss says
• Get me an efficient traveling-salesman
  algorithm, or else!
• What are you going to do?

49
Response

• Yes Maam, expect it this afternoon!

50
Response

• Yes Maam, expect it this afternoon!
• Problem is
• All known algorithms (essentially) check all
  possible paths
• Exhaustive checking is exponential!
• so best of luck!

51
Response

• hah!, I will prove that no polynomial algorithm
  is possible

52
Response

• hah!, I will prove that no polynomial algorithm
  is possible
• Problem is
• Proving intractability is very hard
• Many important problems have
• No known tractable algorithms
• No known proof for intractability

53
Response

• I cant find an efficient algorithm, I guess Im
  just a pathetic loser!

54
Response

• I cant find an efficient algorithm, I guess Im
  just a pathetic loser!
• Bad for job security

55
Perfect Response

• The problem is NP-Complete. I cant find an
  efficient algorithm, but neither can any of these
  famous people !
• Advantage is
• The problem is just as hard as other problems
  smart people cant solve efficiently.
• So it would no good to fire you and fire someone
  else to do the job?

56
Perfect Response

• Would you settle for a pretty good algorithm,
  but that does not give you the optimal solution?

57
Perfect Response

• Would you settle for a pretty good algorithm,
  but that does not give you the optimal solution?
• Intractability isnt the end of the story
• Find an approximate solution
• Use randomization
• Parallelism can help

58

• Computability Theory

59
Computability Theory

• Complexity theory divides problems into two
  classes
• Tractable
• Intractable
• Computability theory divides problems into two
  classes
• Decidable
• Un-decidable

60
Computability Theory important discovery

• In the first half of the 20th century,
  mathematicians like Kurt Goedel, Alan Turing, and
  Alonzo Church discovered that some problems do
  not have a computable solution!

61
Example of undecidable problem

• Is a mathematical statement true or false?
• N.B. what could be more amenable to automation?
• But, no computer algorithm can perform this task!

62
Example of undecidable problem

• Is a mathematical statement true or false?
• N.B. what could be more amenable to automation?
• But, no computer algorithm can perform this task!

These results require theoretical models for
computers? Automata Theory ? construction of
real computers
63
Another example

• Hilberts tenth problem (1900) Find a finite
  algorithm that decides whether a polynomial has
  an integer root.

64
Another example

• A polynomial is a sum of terms, each term is a
  product of variables and constants (coefficients)
• A root of a polynomial is an assignment of values
  to the variables such the value of the polynomial
  is 0
• x2 and y2 is a root for 5x 15y25
• There is no such algorithm (1970)!

65
More example of undecidable problems

• Does a program run forever?
• Is a program correct?
• Are two programs equivalent?
• Is a program optimal?

66

• Automata Theory

67
Automata Theory

• Deals with the definitions and properties of
  mathematical models of computation
• Several models
• Finite automata is used in text processing,
  compilers, and hardware design
• Context-free grammar is used in programming
  language and artificial intelligence (language
  processing)

68
Automata Theory

• Complexity and Computability theory require a
  precise definition of a computer
• Thus Automata theory is the best starting point
  to begin the study of the theory of computation!

69

• Mathematical Notations and Terminology

70
Outline

• Sets
• Functions and relations
• Graphs
• Strings and languages
• Boolean logic
• Proofs

71
Sets

• Sets are defined by its members
• E.g. 1,2,4 or blue, green, red
• Order is not important

72
Set Cardinality

• Sets can be
• finite such as 1, 2
• infinite such as x x is even i.e. the set of
  even naturals
• If A is finite then its cardinality (or size)
  denoted by A is the number of its elements
• The empty Ø set has cardinality 0

73
Set operations

• Membership
• x ? A means that x is a member of the set A
• Subset
• A ? B iff for every x, x ? A implies x ? A
• Equality
• AB means that for x, x ? A iff x ? B
• Union
• a,b U c,d a,b,c,d
• Intersection
• a,b n a,c a
• Difference
• a,b - b,c a

74
Set Operations

• A and B are disjoint iff A n B Ø
• The power set of A, P(A) B B ? A
• E.g., A 1, 2 and P(A)Ø, 1, 2, 1,2
• Cartesian Product
• A X B a,b a ? A and b ? B
• E.g. 1,2 x 3,4 (1,3), (1,4), (2,3), (2,4)

75
Relations and functions

• A k-ary relation R on A1,,Ak is a subset of A1 x
  A2 X X Ak
• A binary relation on A is a subset of A x A

76
Example of a relation
R (a, b), (a, c), (b, d), (d, b), (d, d)
a
b
c
d
77
Example of a relation
R (a, b), (a, c), (b, d), (d, b), (d, d)
a
a
b
b
d
d
c
c
78
Properties of binary relations

• Reflexive
• (a,a) ? R for each a ? A

a
79
Properties of binary relations

• Symmetric
• If (a,b) ? R then (b,a) ? R

b
a
a
b
then
If
80
Properties of binary relations

• Transitive
• If (a,b) ? R and (b,c) ? R then (a,c) ? R

a
b
If
Then
a
c
And
b
c
81
Equivalence relation

• An equivalence relation is a relation that is
• Reflexive
• Symmetric
• Transitive

82
Examples
Transitive Not symmetric Not reflexive
83
Examples
Not Transitive Symmetric Not reflexive
84
Examples
Not Transitive Symmetric Reflexive
85
Examples
Transitive Symmetric Reflexive
86
Partial Functions

• A partial function f S ? T satisfies
• f ? S X T
• For all s ? S there is at most one t ? T with (s,
  t) ? f
• Note that (s, t) ? f we write f(s)t

87
Partial function example
fa, b, c, d ? a, b, c, d f (a, b), (b,
d), (d, d)
a
a
b
b
d
d
c
c
88
Total Functions

• A partial function f S ? T satisfies
• f ? S X T
• For all s ? S there is exactly one t ? T with (s,
  t) ? f
• Note that (s, t) ? f we write f(s)t and we say
  that t is the image of s under f
• S is called the domain of f
• T is called the range of f

89
Total function example
fa, b, c, d ? a, b, c, d f (a, b), (b,
d), (c,d), (d, d)
a
a
b
b
d
d
c
c
90
Injections

• A function fS?T is injective if distinct
  elements in its domain S have distinct images
• For all a,b ? S. a ? b ? f(a) ? f(b)

91
Injections

• A function fS?T is injective if distinct
  elements in its domain S have distinct images
• For all a,b ? S. a ? b ? f(a) ? f(b)

1 2 3
1 2 3
1 2 3
1 2 3
92
Surjections

• A function fS?T is surjective if every b in T
  is the image of some a in S
• For all b ? T. there exists a ? S f(a) b

93
Surjections

• A function fS?T is surjective if every b in T
  is the image of some a in S
• For all b ? T. there exists a ? S f(a) b

1 2 3
1 2 3
1 2 3
1 2 3
94
Bijections

• A function fS?T is bijective iff it is
  injective and surjective. Also called a 1-1
  correspondence

95
Bijections

• A function fS?T is bijective iff it is
  injective and surjective. Also called a 1-1
  correspondence

1 2
1 2 3
1 2 3
1 2 3
96
Relations
97
Relations
Partial functions
98
Relations
Partial functions
Total functions
99
Relations
Partial functions
Total functions
Injections
100
Relations
Partial functions
Total functions
Injections
Surjections
101
Relations
Partial functions
Total functions
Injections
Bijections
Surjections
102
Graphs

• Graph GltV,Egt, where
• V is the set of nodes or vertices
• E is the set of edges
• Degree of a vertex is the number of edges
  including that vertex

103
Graphs

• Graph GltV,Egt, where
• V is the set of nodes or vertices
• E is the set of edges

1
2
b
a
5
3
4
c
d
104
Graph
Subgraph
105
Path
Graph
path
A path in a graph is a sequence of nodes
connected by edge A simple path is a path that
doesnt repeat any node
106
A graph is connected if every two nodes have a
path between them
Not fully connected graph
Fully connected graph
1
2
b
a
5
3
4
c
d
107
Cycle
cycle
A cycle is a path that starts and ends at the
same node A simple cycle is a cycle that doesnt
repeat any node except the first and the last
108
root
tree
leaves
A graph is a tree if it is connected and has no
simple cycles The nodes of degree 1 are called
leaves There is a specially designated node
called a root
109
Directed Graphs

• Graph GltV,Egt is directed if it has arrows
  instead of edges
• V is the set of nodes or vertices
• E is the set of directed edges

b
a
c
d
110
Directed Graphs

• The number of arrows pointing from a particular
  node is the outdegree
• The number of arrows pointing to a particular
  node is the indegree

b
a
c
d
111
Strings and languages

• An alphabet is a finite set of symbols
• A string over an alphabet is a finite sequence of
  symbols from the alphabet
• The length of a string is the number of symbols
  in the string
• The empty string is denoted by e and has length 0

112
Strings and languages

• Reverse abcd vs. dcba
• Substring bc is a substring of abcd
• Concatenation of ab and cd is abcd
• xk is x...x, k times
• A language is a set of strings

113
Boolean logic

• Boolean logic is a mathematical system built
  around the two values TRUE (1) and FLASE (0)
• Foundation of digital electronics and computer
  design
• TRUE and FALSE are called the Boolean values

114
Boolean operations

• Negation not(1)0 and NOT(0)1
• Conjunction
• 1 AND 0 0
• 1 AND 1 1
• 0 AND 0 0
• 0 AND 1 0

115
Boolean expressions

• Use Boolean operations for combining simple
  statements
• Let P stand for the truth value of The sun is
  shining
• Let Q stand for the truth value of Today is
  Tuesday
• P AND Q stands for The sun is shining and
  Today is Tuesday
• P OR Q stands for The sun is shining or Today
  is Tuesday
• P and Q are called operands of the operation

116
More logic operations

• Equality P ? Q is 1 iff both of its operands
  have the same truth value
• Implication P ? Q is equivalent NOT(P) OR Q

117
Distributive law

• P AND (Q OR R) equals
• (P AND Q) OR (P AND R)
• P OR (Q AND R) equals
• (P OR Q) AND (P OR R)

118
Definition

• Definitions describe the objects and the notions
  that we use.
• Simple or complex
• Precise and concise
• When defining some object we must make clear what
  constitutes that object and what does not!

119
Proofs

• Proofs determine the truth or falsity of a
  mathematical statement
• Finding proofs isnt always easy
• There is no recipe
• but there are some guidelines
• Actually the more proofs you do the better you
  get at it!

120
Proofs Guidelines

• Use examples and pictures
• Be patient
• Come back to it
• Be neat
• Be concise
• Use a strategy
• Ill attempt a proof by induction strategy
• It seems that I can find a counter-example easily
• etc

121
Proof types

• Proof by construction
• Proof by contradiction
• Proof by induction
• Or a combination

122
Proof by construction

• Many theorems state that a particular type of
  objects exist.
• Proof by construction demonstrates how to build
  such an object
• To show that there exists an O(nlog n) sorting
  algorithm, we can construct the quick-sort
  algorithm

123
Proof by contradiction

• Proof by contradiction we assume that the
  theorem is false and then show that this
  assumption leads to a contradiction

124
Proof by contradiction

• Prove by contradiction that the square root of 2
  is an irrational number
• N.B. A number is rational if it is a fraction m/n
  where m and n are integers.
• A number is irrational iff it is not rational

125
Proof by contradiction

• Let us assume that sqrt(2) is rational
• This means there exists relatively prime integers
  m and n such that
• sqrt(2) m/n
• Note that m and n cannot both be even
• Now rewrite sqrt(2) m/n as
• n sqrt(2) m

126
Proof by contradiction

• Let us take the square
• (n sqrt(2))2 m2
• n2 2 m2
• Thus m2 is even and so is m ( 2 k)
• So we get n2 2 (2k)2 4 k2

127
Proof by contradiction

• So we get n2 2 (2k)2 4 k2
• Which simplifies to n22k2
• Thus n2 is even and so is n
• This is a contradiction! QED

128
Proof by induction

• To prove P(n)
• Base case prove P(0)
• Induction hypothesis assume P(k) holds for some
  kn
• Induction step Given the induction hypothesis
  show that P(n1) holds

129
Proof by induction

• Prove by induction
• 12..n n(n1)/2
• Any volunteers?

130
Conclusions

• Introduction
• Course overview
• Complexity Theory
• Computability Theory
• Automata Theory
• Formal background