Thinking about Algorithms Abstractly

1
Introduction
Thinking about Algorithms Abstractly
Credits Steve Rudich, Jeff Edmond, Ping Xuan
2

• So you want to be a bioinformatician
  /mathematician /computer scientist?
• What is an Algorithm ?
• Grade school revisited How to multiply two
  numbers

3
So you want to be a computer scientist?
4
Is your goal to be a mundane programmer?
5
Or a great leader and thinker?
6
Original Thinking
7
Boss assigns task

• Given todays prices of pork, grain, sawdust,
• Given constraints on what constitutes a hotdog.
• Make the cheapest hotdog.

Everyday industry asks these questions.
8
Your answer

• Um? Tell me what to code.

With more suffocated software engineering
systems,the demand for mundane programmers will
diminish.
9
Your answer

• I learned this great algorithm that will work.

Soon all known algorithms will be available in
libraries.
10
Your answer

• I can develop a new algorithm for you.

Great thinkers will always be needed.
11
The future belongs to the computer scientist who
has

• Content An up to date grasp of fundamental
  problems and solutions
• Method Principles and techniques to solve the
  vast array of unfamiliar problems that arise in a
  rapidly changing field

12
Course Content

• A list of algoirthms.
• Learn their code.
• Trace them until you are convenced that they
  work.
• Impliment them.

class InsertionSortAlgorithm extends
SortAlgorithm void sort(int a) throws
Exception for (int i 1 i lt a.length i)
int j i int B ai while
((j gt 0) (aj-1 gt B))
aj aj-1
j-- aj B

13
Course Content

• A survey of algorithmic design techniques.
• Abstract thinking.
• How to develop new algorithms for any problem
  that may arise.

14
Study

• Many experienced programmers were asked to code
  up binary search.

15
Study

• Many experienced programmers were asked to code
  up binary search.

80 got it wrong Good thing is was not for a
nuclear power plant.
16
What did they lack?
17
What did they lack?

• Formal proof methods?

18
What did they lack?

• Formal proof methods?

Yes, likely Industry is starting to realize that
formal methods are important. But even without
formal methods . ?
19
What did they lack?

• Fundamental understanding of the algorithmic
  design techniques.
• Abstract thinking.

20
Course Content

• Notations, analogies, and abstractions
• for developing,
• thinking about,
• and describing algorithms

21
You will see some Math
22
What is an Algorithm?

• A step-by-step description of procedures
  performing certain task.
• Example Sorting.
• Given a list of numbers, put them in increasing
• (non-decreasing) order.
• Properties Generality, Termination, Correctness,
  Feasibility.
• Feasibility ? Analysis of Algorithm
• Complexity theory

23
Analysis of Algorithm

• Running time analysis
• Input size N
• Running time is a function of N T(N)
• Worst case, average case, , etc
• Time measured by number of computer operations
• Each basic operation (add, load, write, etc)
  count as 1
• Actual clock time differs by a constant factor
• Usually very complex
• The use of asymptotic bounds
• To study the rate of growth of T(N) compared to
  simpler functions f(N), such as N, N2, log(N),
  etc
• A constant factor can be ignored

24
Some Definitions Big O Notation

• T(N) O( f(N) )
• Exists constant c and n0 such that when Ngtn0,
  T(N) lt c f(N)
• Asymptotic Upper bound

c f(N)
T(N)
n0
N
25
Some Definitions Big Omega

• T(N) ?(g(N))
• Exists constant c and n0 such that when Ngtn0,
  T(N) gt c g(N)
• Asymptotic Lower bound

T(N)
c g(N)
n0
N
26
Some Definitions Big Theta

• T(N) ?( h(N) )
• if and only if T(N)O(h(N)) and T(N) ?(h(N))
• tight bound

c1 h(N)
T(N)
c2 h(N)
N
27
Some Definitions Little o

• T(N) o(p(N))
• if lim N ? 8 o. E.g. log(N)
  o(N).

28
Example Insertion Sort Algorithm

• class InsertionSortAlgorithm extends
• SortAlgorithm
• void sort(int a ) throws Exception
• for (int i 1 i lt a.length i)
• int j i
• int B ai
• while ((j gt 0) (aj-1 gt B))
• aj aj-1
• j--
• aj B

29
Iterative Algorithms
ltpreCondgt codeA loop exit
when ltexit Condgt codeB codeC ltpostCondgt
One step at a time
Relay Race
Code
30
Problem Specification

• Precondition The input is a list of n values
  with the same value possibly repeated.
• Post condition The output is a list consisting
  of the same n values in non-decreasing order.

31
Define Step

• Select arbitrary element from side.
• Insert it where it belongs.

14
30
62
25
98
79
23,31,52,62,88
32
Making progress while Maintaining the loop
invariant
14
30
62
25
98
79
23,31,52,62,88
Sorted sub-list
33
Beginning Ending
34
Running Time

• Inserting an element into a list of size i
• takes O(i) time in the worst case.
• Total 123n n(n1)/2 ?(n2)

14
30
62
25
98
79
23,31,52,62,88
35
Explaining Insertion Sort

• We maintain a subset of elements sorted within
  a list. The remaining elements are off to the
  side somewhere. Initially, think of the first
  element in the array as a sorted list of length
  one. One at a time, we take one of the elements
  that is off to the side and we insert it into the
  sorted list where it belongs. This gives a sorted
  list that is one element longer than it was
  before. When the last element has been inserted,
  the array is completely sorted.

36
Insertion Sort
The input consists of an array of integers
We read in the i1st object. We will pretend that
this larger prefix is the entire input. We
extend the solution we have by one for this
larger prefix.
37
Insertion Sort Algorithm Pseudo-Code

• Input an array (or list) a of numbers
• // data structure a0 an-1
• Operations
• Leave the first element (location 0) untouched.
• for each element from location 1 on
• insert it to the appropriate place in the
  
• (sorted) sub-array to its left.

a
0
1
2
n-1
38
Operations in an Algorithm

• Decision Making
• branching operation
• if then else
• Repetition
• loops for each element do
• conditional loops while ( . ) do

39
Insertion Sort Algorithm Code

• class InsertionSortAlgorithm extends
• SortAlgorithm
• void sort(int a ) throws Exception
• for (int i 1 i lt a.length i)
• int j i
• int b ai
• while ( (j gt 0) (aj-1 gt b) )
• aj aj-1
• j--
• aj B

40
Grade School RevisitedHow To Multiply Two
Numbers
Another Algorithm
41
Complex Numbers

• Remember how to multiply 2 complex numbers?
• (abi)(cdi) ac bd ad bc i
• Input a,b,c,d Output ac-bd, adbc
• If a real multiplication costs 1 and an addition
  cost a penny. What is the cheapest way to obtain
  the output from the input?
• Can you do better than 4.02?

42
Gauss 3.05 MethodInput a,b,c,d Output
ac-bd, adbc

• m1 ac
• m2 bd
• A1 m1 m2 ac-bd
• m3 (ab)(cd) ac ad bc bd
• A2 m3 m1 m2 adbc

43
Question

• The Gauss hack saves one multiplication out of
  four. It requires 25 less work.
• Could there be a context where performing 3
  multiplications for every 4 provides a more
  dramatic savings?

44
Odette
Bonzo
45
How to add 2 n-bit numbers.












46
How to add 2 n-bit numbers.












47
How to add 2 n-bit numbers.












48
How to add 2 n-bit numbers.












49
How to add 2 n-bit numbers.












50
How to add 2 n-bit numbers.













51
Time complexity of grade school addition
On any reasonable computer adding 3 bits can be
done in constant time.
T(n) The amount of time grade school addition
uses to add two n-bit numbers ?(n) linear
time.
52
f ?(n) means that f can be sandwiched between
two lines
time
of bits in numbers
53
Please feel free to ask questions!
54
Is there a faster way to add?

• QUESTION Is there an algorithm to add two n-bit
  numbers whose time grows sub-linearly in n?

55
Any algorithm for addition must read all of the
input bits

• Suppose there is a mystery algorithm that does
  not examine each bit
• Give the algorithm a pair of numbers. There must
  be some unexamined bit position i in one of the
  numbers
• If the algorithm is not correct on the numbers,
  we found a bug
• If the algorithm is correct, flip the bit at
  position i and give the algorithm the new pair of
  numbers. It give the same answer as before so it
  must be wrong since the sum has changed

56
So any algorithm for addition must use time at
least linear in the size of the numbers. Grade
school addition is essentially as good as it can
be.
57
How to multiply 2 n-bit numbers.

X










58
I get it! The total time is bounded by cn2.
59
Grade School Addition Linear timeGrade School
Multiplication Quadratic time
time
of bits in numbers

• No matter how dramatic the difference in the
  constants the quadratic curve will eventually
  dominate the linear curve

60
Neat! We have demonstrated that multiplication is
a harder problem than addition.Mathematical
confirmation of our common sense.
61
Dont jump to conclusions!We have argued that
grade school multiplication uses more time than
grade school addition. This is a comparison of
the complexity of two algorithms. To argue that
multiplication is an inherently harder problem
than addition we would have to show that no
possible multiplication algorithm runs in linear
time.
62
Grade School Addition ?(n) timeGrade School
Multiplication ?(n2) time
Is there a clever algorithm to multiply two
numbers in linear time?
63
Despite years of research, no one knows! If you
resolve this question, Tunghai will give you a
PhD!
64
Is there a faster way to multiply two numbers
than the way you learned in grade school?
65
Divide and Conquer(an approach to faster
algorithms)

• DIVIDE my instance to the problem into smaller
  instances to the same problem.
• Have a friend (recursively) solve them.Do not
  worry about it yourself.
• GLUE the answers together so as to obtain the
  answer to your larger instance.

66
Multiplication of 2 n-bit numbers

• X
• Y
• X a 2n/2 b Y c 2n/2 d
• XY ac 2n (adbc) 2n/2 bd

a
b
c
d
67
Multiplication of 2 n-bit numbers

• X
• Y
• XY ac 2n (adbc) 2n/2 bd

a
b
c
d
MULT( X, Y ) If X Y 1 then RETURN XY
Break X into ab and Y into cd RETURN
MULT(a,c) 2n (MULT(a,d) MULT(b,c)) 2n/2
MULT(b,d)
68
Time required by MULT

• T( n ) time taken by MULT on two n-bit
  numbers
• What is T(n)? What is its growth rate? Is it
  ?(n2)?

69
Recurrence Relation

• T(1) k for some constant k
• T(n) 4 T(n/2) k n l for some
  constants k and l

MULT(X,Y) If X Y 1 then RETURN XY
Break X into ab and Y into cd RETURN
MULT(a,c) 2n (MULT(a,d) MULT(b,c)) 2n/2
MULT(b,d)
70
Lets be concrete

• T(1) 1
• T(n) 4 T(n/2) n
• How do we unravel T(n) so that we can determine
  its growth rate?

71
Technique 1Guess and Verify

• Recurrence Relation
• T(1) 1 T(n) 4T(n/2) n
• Guess G(n) 2n2 n
• Verify

72
Technique 2 Decorate The Tree

• T(1) 1

T(1)

1

• T(n) n 4 T(n/2)

• T(n) n 4 T(n/2)

T(n)
T(n)
n
n


T(n/2)
T(n/2)
T(n/2)
T(n/2)
T(n/2)
T(n/2)
T(n/2)
T(n/2)
73
T(n)
n

T(n/2)
T(n/2)
T(n/2)
T(n/2)
74
T(n)
n

T(n/2)
T(n/2)
T(n/2)
75
T(n)
n

76
T(n)
n

n/2
n/2
n/2
n/2
n/4
n/4
n/4
n/4
n/4
n/4
n/4
n/4
n/4
n/4
n/4
n/4
n/4
n/4
n/4
n/4
11111111111111111111111111111111 . . . . . .
111111111111111111111111111111111
77
0
1
2
i
78
1n
4n/2
16n/4
4i n/2i
4lognn/2logn nlog41
Total ?(nlog4) ?(n2)
79
Divide and Conquer MULT ?(n2) time Grade School
Multiplication ?(n2) time
All that work for nothing!
80
MULT revisited
MULT(X,Y) If X Y 1 then RETURN XY
Break X into ab and Y into cd RETURN
MULT(a,c) 2n (MULT(a,d) MULT(b,c)) 2n/2
MULT(b,d)

• MULT calls itself 4 times. Can you see a way to
  reduce the number of calls?

81
Gauss HackInput a,b,c,d Output ac, adbc,
bd

• A1 ac
• A3 bd
• m3 (ab)(cd) ac ad bc bd
• A2 m3 A1- A3 ad bc

82
Gaussified MULT(Karatsuba 1962)
MULT(X,Y) If X Y 1 then RETURN XY
Break X into ab and Y into cd e MULT(a,c)
and f MULT(b,d) RETURN e2n (MULT(ab, cd)
e - f) 2n/2 f

• T(n) 3 T(n/2) n
• Actually T(n) 2 T(n/2) T(n/2 1) kn

83
T(n)
n

84
T(n)
n

T(n/2)
T(n/2)
T(n/2)
85
T(n)
n

T(n/2)
T(n/2)
86
T(n)
n

87
0
1
2
n/4 n/4 n/4 n/4 n/4 n/4 n/4 n/4
n/4
i
88
1n
3n/2
9n/4
3i n/2i
3lognn/2logn nlog31
Total ?(nlog3) ?(n1.58..)
89
Dramatic improvement for large n
Not just a 25 savings! ?(n2) vs
?(n1.58..)
90
Multiplication Algorithms
343333
91
Youre cool! Are you free sometime this weekend?
Not interested, Bonzo. I took the initiative and
asked out a guy in my 4277 class.