CSE 830: Design and Theory of Algorithms

1
CSE 830Design and Theory of Algorithms

• Dr. Eric Torng

2
Overview

• Administrative stuff
• Lecture schedule overview
• What is an algorithm? What is a problem?
• How we study algorithms
• Some examples

3
What is an Algorithm?
According to the Academic American
Encyclopedia An algorithm is a procedure for
solving a usually complicated problem by carrying
out a precisely determined sequence of simpler,
unambiguous steps. Such procedures were
originally used in mathematical calculations (the
name is a variant of algorism, which originally
meant the Arabic numerals and then "arithmetic")
but are now widely used in computer programs and
in programmed learning.
4
What is an Algorithm?
Algorithms are the ideas behind computer
programs. An algorithm is the thing that
stays the same whether the program is in C
running on a Cray in New York or is in BASIC
running on a Macintosh in Katmandu! To be
interesting, an algorithm has to solve a general,
specified problem.
5
What is a problem?

• Definition
• A mapping/relation between a set of input
  instances (domain) and an output set (range)
• Problem Specification
• Specify what a typical input instance is
• Specify what the output should be in terms of the
  input instance
• Example Sorting
• Input A sequence of N numbers a1an
• Output the permutation (reordering) of the input
  sequence such that a1 ? a2 ? ? an .

6
Types of Problems
Search find X in the input satisfying property
Y Structuring Transform input X to satisfy
property Y Construction Build X satisfying
Y Optimization Find the best X satisfying
property Y Decision Does X satisfy Y? Adaptive
Maintain property Y over time.
7
Two desired properties of algorithms

• Correctness
• Always provides correct output when presented
  with legal input
• Efficiency
• What does efficiency mean?

8
Example Odd or Even?
What is the best way to determine if a number is
odd or even? Some of the algorithms we can try
are

• Count up to that number from one and alternate
  naming each number as odd or even.
• Factor the number and see if there are any twos
  in the factorization.
• Keep a lookup table of all numbers from 0 to the
  maximum integer.
• Look at the last bit (or digit) of the number.

9
Example TSP

• Input A sequence of N cities with the distances
  dij between each pair of cities
• Output a permutation (ordering) of the cities
  ltc1, , cngt that minimizes the expression
• Sj 1 to n-1 dj,j1 dn,1

10
Example Circuit
11
Not Correct!
12
A better algorithm?
What if we always connect the closest pair of
points that do not result in a cycle or a
three-way branch? We finish when we have a
single chain.
13
Now it works
14
Still Not Correct!
15
A Correct Algorithm
We could try all possible orderings of the
points, then select the ordering which minimizes
the total length d ? For each of the n!
permutations, Pi of the n points, if cost(Pi) lt d
then d cost(Pi) Pmin Pi return Pmin
16
Areas of study of algorithms

1. How to devise correct and efficient algorithms
   for solving a given problem
2. How to express algorithms
3. How to validate/verify algorithms
4. How to analyze algorithms
5. How to prove (or at least indicate) no correct,
   efficient algorithm exists for solving a given
   problem

17
How to devise algorithms

• Something of an art form
• Cannot be fully automated
• We will describe some general techniques and try
  to illustrate when each is appropriate
• Major focus of course

18
Expressing Algorithms

• Implementations
• Pseudo-code
• English
• NOT our point of emphasis in this course

19
Verifying algorithm correctness

• Proving an algorithm generates correct output for
  all inputs
• One technique covered in textbook
• Loop invariants
• We will do some of this in the course, but it is
  not emphasized as much as algorithm design or
  algorithm analysis

20
Analyzing algorithms

• The process of determining how much resources
  (time, space) are used by a given algorithm
• We want to be able to make quantitative
  assessments about the value (goodness) of one
  algorithm compared to another
• We want to do this WITHOUT implementing and
  running an executable version of an algorithm
• Major point of emphasis of course
• Question How can we study the time complexity of
  an algorithm if we dont run it or even choose a
  specific machine to measure it on?

21
The RAM Model

• Each "simple" operation (, -, , if, call) takes
  exactly 1 step.
• Loops and subroutine calls are not simple
  operations, but depend upon the size of the data
  and the contents of a subroutine. We do not want
  sort to be a single step operation.
• Each memory access takes exactly 1 step.

22
Measuring Complexity

• The worst case complexity of the algorithm is the
  function defined by the maximum number of steps
  taken on any instance of size n.
• The best case complexity of the algorithm is the
  function defined by the minimum number of steps
  taken on any instance of size n.
• The average-case complexity of the algorithm is
  the function defined by an average number of
  steps taken on any instance of size n.

23
Best, Worst, and Average Case
24
Case Study Insertion Sort
One way to sort an array of n elements is to
start with an empty list, then successively
insert new elements into their proper position,
scanning from the right end back to the left
end Loop invariant used to prove
correctness How efficient is insertion sort?
25
Correctness Analysis
What is the loop invariant? That is, what is
true before each execution of the loop? for
i 2 to n key Ai j i - 1 while
j gt 0 AND Aj gt key Aj1 Aj j j
-1 Aj1 key
26
Exact Analysis
Count the number of times each line will be
executed Num Exec. for i 2 to n (n-1)
1 key Ai n-1 j i - 1 n-1 while
j gt 0 AND Aj gt key ? Aj1 Aj
? j j -1 ? Aj1 key n-1
27
Best Case
28
Worst Case
29
Average Case
30
Average Case (Zoom Out)
31
1 for i 2 to n 2 key Ai 3 j
i - 1 4 while j gt 0 AND Aj gt key 5
Aj1 Aj 6 j j -1 7
Aj1 key
3 7 4 9 4 1 i 3 2 key 4 3 j
2 4 (while true!) 5 A3 A2 3 7
7 9 4 6 j 1 4 (while false!) 7 A2
4 3 4 7 9 4 1 i 4 2 key
9 3 j 3 4 (while false!) 7 A4 9
3 4 7 9 4 1 i 5 2 key 4 3 j
4 4 (while true!) 5 A5 A4 3 4
7 9 9 6 j 3 4 (while true!) 5 A4
A3 3 4 7 7 9 6 j 2 4 (while
false!) 7 A3 4 3 4 4 7 9
3 7 4 9 4 1 i 2 2 key 7 3 j
1 4 ( while false! ) 7 A2 7
32
Measuring Complexity
What is the best way to measure the time
complexity of an algorithm? - Best case run
time? - Worst case run time? - Average case run
time? Which should we try to optimize?
33
Best Case Analysis
How can we modify almost any algorithm to have a
good best-case running time? Solve one
instance of it efficiently.
34
Average case analysis

• Based on a probability distribution of input
  instances
• Distribution may not be accurate
• Often more complicated to compute than worst case
  analysis
• Often worst case analysis is a pretty good
  indicator for average case
• Though not always true quicksort, simplex method

35
Best, Worst, and Average Case
36
Worst case analysis

• Need to find and understand input that causes
  worst case performance
• Typically much simpler to compute as we do not
  need to average out performance
• Provides guarantee that is independent of any
  assumptions about the input
• The standard analysis performed