HEURISTIC ALGORITHMS

1
ADVANCED ALGORITHM ANALYSISLECTURE
1INTRODUCTION-Heuristic Algorithms

• Prof. Vuda Sreenivasarao
• Bahir Dar University-ETHIOPIA

2
Course Information

• Course Name ADVANCED ALGORITHM ANALYSIS.
• Course Code6162

3
ADVANCED ALGORITHM ANALYSIS

• 1.Introduction
• 2.Basic concept of Heuristic algorithms.
• 2.1.Example of Heuristic algorithms.
• 2.2The Traveling Salesman Problem.
• 2.3Traveling Salesman Problem algorithm with
  example.

4
Textbooks

• There is no textbook required for the
  course. Lecture notes are available for the
  current course. Reference textbooks for each
  topic are listed in the readings section.

5
References Books

• 1. Fundamentals of Computer Algorithms, Ellis
  Horowitz , Satraj Sahni and Rajasekharam,Galgotia
  publications pvt. Ltd.
• 2.Introduction to Algorithms, second
  edition,T.H.Cormen,C.E.Leiserson, L.Rivest, and
  C.Stein,PHI Pvt. Ltd./ Pearson Education
• 3.Introduction to the design and analysis of
  Algorithms, second edition by Anany Levitin.
• 4.Introduction to Design Analysis Computer
  Algorithm 3rd, Sara Baase, Allen Van Gelder,
  Adison-Wesley, 2000.
• 5.Algorithms, Richard Johnsonbaugh, Marcus
  Schaefer, Prentice Hall, 2004.
• 6. Steven S. Skiena ,The Algorithm Design Manual
  ,Second Edition

6
References Books

• 7.Allan Borodin and Ran El-Yaniv Online
  Computation and Competitive Analysis, Cambridge
  University Press, 2005.
• 8.Vijay Vazirani Approximation Algorithms,
  Springer, 2001.
• 9.Dorit S. Hochbaum (ed.) Approximation
  Algorithms for NP-hard Problems, PWS Publishing,
  1997.
• 10.Joseph JáJá Introduction to Parallel
  Algorithms 1992.
• 11.Rajeev Motwani, Prabhakar Raghavan Randomized
  Algorithms, Cambridge University Press, 1995.
• 12.Jiri Matousek and Bernd Gärtner Understanding
  and Using Linear Programming , 2006.

7
Assignments and Examination

• Grading Criteria
• Assignment-20.
• Project/Term Paper -30.
• Examination-50.

8
Course Objectives

• This course introduces students to the analysis
  and design of advanced computer algorithms. Upon
  completion of this course, students will be able
  to do the following
• Analyze the asymptotic performance of algorithms.
• Demonstrate a familiarity with major algorithms .
• Apply important algorithmic design paradigms and
  methods of analysis.
• Synthesize efficient algorithms in common
  engineering design situations.
• Design of fast algorithms.

9
Cont.

• Depth knowledge in ADVANCED ALGORITHM ANALYSIS
  and develop programs efficiently by analyzing
  first the feasibility of an algorithm used before
  program coding.
• To develop an understanding about basic
  algorithms and different problem solving
  strategies.
• To improve creativeness and the confidence to
  solve non-conventional problems and expertise for
  analyzing existing solutions.
• An end of the Course Solve any type of
  problems.
• Algorithmic is more than a branch of computer
  science.

10
Cont-----

• It is the core of computer science, and, in all
  fairness, can be said to be relevant to most of
  science, business, and technology.
• Another reason for studying algorithms is their
  usefulness in developing analytical skills.
• A person well-trained in computer science knows
  how to deal with algorithms how to construct
  them, manipulate them, understand them, analyze
  them.
• This knowledge is preparation for much more than
  writing good computer programs it is a
  general-purpose mental tool that will be a
  definite aid to the understanding of other
  subjects, whether they be chemistry, linguistics,
  or music, etc.

11
Prerequisites

• Programming
• Data Structures.
• Discrete Mathematics.
• Design Analysis and Algorithms

12
Approach

• Analytical
• Build a mathematical model of a computer.
• Study properties of algorithms on this modal.
• REASON about algorithms PROVE facts about time
  taken.

13
Applications

• Study problems these techniques can be applied to
• sorting
• data retrieval
• network routing
• Games
• etc

14
Where We're Going

• Learn general approaches to algorithm design
• Divide and conquer
• Greedy method
• Dynamic Programming
• Basic Search and Traversal Technique
• Graph Theory
• Linear Programming
• Approximation Algorithm
• NP Problem

15
Where We're Going

• Examine methods of analyzing algorithm
  correctness and efficiency
• Recursion equations
• Lower bound techniques
• O,Omega and Theta notations for
  best/worst/average case analysis
• Decide whether some problems have no solution in
  reasonable time
• List all permutations of n objects (takes n!
  steps)
• Travelling salesman problem
• Investigate memory usage as a different measure
  of efficiency

16
Algorithm. (webster.com)

• A procedure for solving a mathematical problem
  (as of finding the greatest common divisor) in a
  finite number of steps that frequently involves
  repetition of an operation.
• Broadly a step-by-step procedure for solving a
  problem or accomplishing some end especially by a
  computer.
• "Great algorithms are the poetry of computation."
  
• Etymology
• "algos" Greek word for pain.
• "algor" Latin word for to be cold.
• Abu Jafar al-Khwarizmis 9th century Arab
  scholar.
• his book "Al-Jabr wa-al-Muqabilah" evolved into
  todays high school algebra text

17
Example

• How to make a coffee with a coffee machine1.
  Open the coffee machine2. Put water in it.3.
  Close it4. Open the coffee container (hope it's
  called like that)5. Fill it with finely ground
  coffee.6. Close it7. Turn the machine on8.
  Wait 3-4 minutes9. Open it 10. Pour the
  coffee.11. etc.

18
Imagine A World With No Algorithms

• Fast arithmetic.
• Cryptography.
• Quick sort.
• Databases.
• Signal processing.
• Huffman codes.
• Data compression.
• Network flow.
• Routing Internet packets.
• Linear programming.
• Planning, decision-making.

19
What is an Algorithm?- Anany Levitin

• An algorithm is a sequence of unambiguous
  instructions for solving a problem, i.e., for
  obtaining a required output for any legitimate
  input in a finite amount of time.

20
What is a Computer Algorithm?

• A computer algorithm is
• a detailed step-by-step method for
• solving a problem
• by using a computer.

21
Which algorithm is better?

• The algorithms are correct, but which is the
  best?
• Measure the running time (number of operations
  needed).
• Measure the amount of memory used.
• Note that the running time of the algorithms
  increase as the size of the input increases.

22
Importance of Analyze Algorithm

• Need to recognize limitations of various
  algorithms for solving a problem.
• Need to understand relationship between problem
  size and running time.
• When is a running program not good enough?
• Need to learn how to analyze an algorithm's
  running time without coding it.
• Need to learn techniques for writing more
  efficient code.
• Need to recognize bottlenecks in code as well as
  which parts of code are easiest to optimize.

23
Why do we analyze about them?

• understand their behavior, and (Job -- Selection,
  performance, modify) improve them. (Research)
• Correctness
• Does the input/output relation
• match algorithm requirement?
• Amount of work done ( complexity)
• Basic operations to do task
• Amount of space used
• Memory used
• Simplicity, clarity
• Verification and implementation.
• Optimality
• Is it impossible to do better?

24
Algorithm Classification

• There are various ways to classify algorithms
• 1. Classification by implementation
• Recursion or iteration
• Logical
• Serial or parallel or distributed
• Deterministic or non-deterministic
• Exact or approximate
• 2.Classification by Design Paradigm
• Divide and conquer.
• Dynamic programming.
• The greedy method.
• Linear programming.
• Reduction.
• Search and enumeration.
• The probabilistic and heuristic paradigm.

25
Solution Methods

• Try every possibility (n-1)! possibilities
• grows
  faster than exponentially
• If it took 1 microsecond to calculate each
  possibility takes 10140 centuries to calculate
  all possibilities when n 100

1. Optimising Methods obtain guaranteed optimal
   solution, but can take a very, very,
   long time

III. Heuristic Methods obtain
good solutions quickly
by intuitive methods.
No guarantee of optimality.
26
Heuristic Algorithms

• The term heuristic is used for algorithms which
  find solutions among all possible ones ,but they
  do not guarantee that the best will be found,
  therefore they may be considered as approximately
  and not accurate algorithms.
• These algorithms, usually find a solution close
  to the best one and they find it fast and easily.
• Sometimes these algorithms can be accurate, that
  is they actually find the best solution, but the
  algorithm is still called heuristic until this
  best solution is proven to be the best .
• EXAMPLE Heuristic algorithm for the Traveling
  Salesman Problem (T.S.P) .

27
Traveling Salesman Problem
A Salesman wishes to travel around a given set of
cities, and return to the beginning, covering the
smallest total distance.
Easy to State
Difficult to Solve
28
Traveling Salesman Problem (T.S.P) 

• William Rowan
  Hamilton
• This is one of the most known problems, and is
  often called as a difficult problem.
• A salesman must visit n cities, passing through
  each city only once, beginning from one of them
  which is considered as his base, and returning to
  it.
• The cost of the transportation among the cities
  (whichever combination possible) is given.
• The program of the journey is requested, that is
  the order of visiting the cities in such a way
  that the cost is the minimum.

29
Traveling Salesman Problem

• Let's number the cities from 1 to n , and
• let city 1 be the city-base of the salesman.
• Also let's assume that c(i, j) is the visiting
  cost from i to j.
• There can be c(i, j)ltgtc(j, i).
• Apparently all the possible solutions are (n-1)!.
• Someone could probably determine them
  systematically, find the cost for each and every
  one of these solutions and finally keep the one
  with the minimum cost.
• These require at least (n-1)! steps.

30
Traveling Salesman Problem

• If for example there were 21 cities the steps
  required are (n-1)!(21-1)!20! steps.
• If every step required a msec we would need
  about 770 centuries of calculations.
• Apparently, the exhausting examination of all
  possible solutions is out of the question. Since
  we are not aware of any other quick algorithm
  that finds a best solution we will use a
  heuristic algorithm.

31
Traveling Salesman Problem

• According to this algorithm whenever the salesman
  is in town i he chooses as his next city, the
  city j for which the c(i,j) cost, is the minimum
  among all c(i,k) costs, where k are the pointers
  of the city the salesman has not visited yet.
• There is also a simple rule just in case more
  than one cities give the minimum cost, for
  example in such a case the city with the
  smaller k will be chosen.
• This is a greedy algorithm which selects in every
  step the cheapest visit and does not care whether
  this will lead to a wrong result or not.

32
Traveling Salesman Problem Algorithm

• T.S.P. algorithm
• Input  Number of cities n and array of costs
  c(i,j) i, j1,..n
• (We begin from city number 1)Output Vector of
  cities and total cost.
• ( starting values )
• C0
• cost0
• visits0
• e1 (epointer of the visited city)
• ( determination of round and cost)
• for r1 to n-1 do
• choose of pointer j with
• minimumc(e,j)minc(e,k)visits(k)0 and
  k1,..,n
• costcost minimum
• ej
• C(r)j
• end r-loop
• C(n)1
• costcost c(e,1)

33
Traveling Salesman Problem Algorithm

• We can find situations in which the TSP algorithm
  don't give the best solution.
• We can also succeed on improving the algorithm.
• For example we can apply the algorithm t times
  for t different cities and keep the best round
  every time.
• We can also unbend the greeding in such a way to
  reduce the algorithm problem ,that is there is no
  room for choosing cheep sides at the end of
  algorithm because the cheapest sides have been
  exhausted.

34
If there is no condition to return to the
beginning. It can still be regarded as a
TSP. Suppose we wish to go from A to B visiting
all cities.
A
B
35
If there is no condition to return to the
beginning. It can still be regarded as a TSP.
Connect A and B to a dummy city at zero
distance (If no stipulation of start and finish
cities connect all to dummy at zero distance)
A
B
36
If there is no condition to return to the
beginning. It can still be regarded as a TSP.
Create a TSP Tour around all cities
A
B
37
A route returning to the beginning is known as a
Hamiltonian Circuit
A route not returning to the beginning is known
as a Hamiltonian Path
Essentially the same class of problem
38
Hamilton Circuits

• Euler circuit/path gt Visit each edge once and
  only once
• Hamilton circuit gt Visit each vertex once and
  only once (except at the end, where it returns to
  the starting vertex)
• Hamilton path gt Visit each vertex once and only
  once
• Difference Edge (Euler) ? Vertex (Hamilton)

39
Examples of Hamilton circuits

• Has many Hamilton circuits
• A, B, C, D, E, A
• A, D, C, E, B, A
• Has many Hamilton paths
• A, B, C, D, E
• A, D, C, E, B
• Has no Euler circuit, no Euler path gt 4 vertices
  of odd degree

Graph 1
Hamilton circuits can be shortened into a
Hamilton path by removal of the last edge
40
Examples of Hamilton circuits
A
B

• Has no Hamilton circuits
• What ever the starting point, we are going to
  have to pass through vertex E more than once to
  close the circuit.
• Has many Hamilton paths
• A, B, E, C, D
• C, D, E, A, B
• Has Euler circuit gt each vertex has even degree

E
D
C
Graph 2
41
Examples of Hamilton circuits
F
A
B

• Has many Hamilton circuits
• A, F, B, E, C, G, D, A
• A, F, B, C, G, D, E, A
• Has many Hamilton paths
• A, F, B, E, C, G, D
• A, F, B, C, G, D, E
• Has Euler circuit gt Every vertex has even degree

E
D
C
G
Graph 3
42
Examples of Hamilton circuits
G
F
Has no Hamilton circuits Has no Hamilton
paths Has no Euler circuit Has no Euler path
gt more than 2 vertices of odd degree
A
B
E
C
D
I
H
Graph 4
43
Complete graph

• A graph with N vertices in which every pair of
  vertices is joined by exactly one edge is called
  the complete graph.
• Total no. of edges N(N-1)/2

In K4, each vertex has degree 3 and the number
of edges 4 (3)/2 6
44
The six Hamilton circuits of K4
Rows gt 6 Hamilton circuits Colsgt same Hamilton
circuit with different reference points
Graph
Reference point A
Reference point B
Reference point C
Reference point D
A,B,C,D,A B,C,D,A,B C,D,A,B,C D,A,B,C,D
A,B,D,C,A B,D,C,A,B C,A,B,D,C D,C,A,B, D
A,C,B,D,A B,D,A,C,B C,B,D,A,C D,A,C,B,D
A,C,D,B,A B,A,C,D,B C,D,B,A, C D,B,A,C,D
A,D,B,C,A B,C,A,D,B C,A,D,B,C D,B,C,A,D
A,D,C,B,A B,A,D,C,B C,B,A,D,C D,C,B,A,D
45
Complete graph

• The number of Hamilton circuits in a complete
  graph can be computed by using factorials.
• The complete graph with N vertices has
• N! (factorial of N) 1x 2x3x4x x(N-1)x N
• (N-1)! Hamilton circuits.
• Example The complete graph with 5 vertices has
  4! 1x2x3x4 24 Hamilton circuits

46
Factorial
Which of the following is true? n! n! x
(n-1)! n! n! (n-1)! n! n x (n-1)! n! n
(n-1)!
47
Traveling Salesman Problem

• Traveling Salesman Problem (TSP) Given a
  complete graph with nonnegative edge costs, Find
  a minimum cost cycle visiting every vertex
  exactly once.
• Application (Example) Given a number of cities
  and the costs of traveling from any city to any
  other city, what is the cheapest round-trip route
  that visits each city exactly once and then
  returns to the starting city.

48
Applications of the TSP
Routing around Cities Computer Wiring
- connecting together computer
components using minimum
wire length Archaeological
Seriation - ordering sites in
time Genome Sequencing -
arranging DNA fragments in
sequence
Job Sequencing
- sequencing jobs in order to
minimise total
set-up time
between jobs Wallpapering to Minimise
Waste NB First three applications generally
symmetric Last three asymmetric
49
Major Practical Extension of the TSP
Vehicle Routing - Meet customers demands
within given time windows using lorries of
limited capacity
Depot
Much more difficult than TSP
50
History of TSP
1800s Irish Mathematician, Sir
William Rowan Hamilton
1930s Studied by Mathematicians Menger,
Whitney, Flood etc.
1954 Dantzig, Fulkerson, Johnson, 49 cities
(capitals of USA states) problem solved

1. 64 Cities

1975 100 Cities
1977 120 Cities
1980 318 Cities
1987 666 Cities
1987 2392 Cities (Electronic Wiring
Example)
1994 7397 Cities

• 13509 Cities (all towns in the USA with
  population gt 500)
  
• 
  

• 15112 Cities (towns in Germany)
• 24978 Cities (places in Sweden)
• But many smaller instances not yet
  solved (to proven optimality)

But there are still many smaller instances which
have not been solved.
51
Finding an Approximate Solution

• Cheapest Link Algorithm
• Edge with smallest weight is drawn first
• Edge with second smallest weight is drawn in
• Continue unless it closes a smaller circuit or
  three edges come out of one vertex
• Finished once a complete Hamilton Circuit is drawn

52
Cheapest Link Algorithm
53
Cheapest Link Algorithm
54
Cheapest Link Algorithm
55
Cheapest Link Algorithm
56
Cheapest Link Algorithm
57
Finding an Approximate Solution

• Nearest Neighbor Algorithm
• Start at any given vertex
• Travel to edge that yields smallest weight and
  has not been traveled through yet
• Continue until we have a complete Hamilton circuit

58
Nearest Neighbor Algorithm
59
Nearest Neighbor Algorithm
60
Nearest Neighbor Algorithm
61
Nearest Neighbor Algorithm
62
Nearest Neighbor Algorithm
63
Comparing Approximating Methods
Nearest Neighbor
Cheapest Link
Total weight for Cheapest Link 31 Nearest
Neighbor 33
64
Example (Doh!) We all know that Homer is lazy.
However, he has to run errands at the
Kwik-E-Mart, the Retirement Home and Moes.
Assuming that he wants to begin and end his day
at home find the route that will allow him to get
back to napping as soon as possible.
65

• We might represent this dilemma with the
  following graph

D
8
A
17
12
15
20
B
C
20
66

• We might represent this dilemma with the
  following graph

D
8
A
These numbers will represent the number of blocks
between each destination. When we place values
like this on the edges of a graph we refer to
them as the weights of the edges.
17
12
15
27
B
C
20
67

• One solution might begin as follows

D
8
A
17
12
15
20
B
C
20
68

• . . . And continue like so. . .

D
8
A
17
12
15
20
B
C
20
69

• . . . And so on . . .

D
8
A
17
12
15
20
B
C
20
70

• . . . Continuing until he arrives back at home.

D
8
A
17
12
15
20
B
C
20
71

• Following this circuit, we find that Homer has
  to travel 12 8 20 17 57 blocks to
  finish his errands and get back to napping.
• But is this the most efficient route he can
  take?
• How might we find this best route?

72

• Method 1 Make a list of all possible
  Hamiltonian circuits. Calculate the cost for
  each one. Select the circuit with the least cost.

73

• Method 1 Make a list of all possible
  Hamiltonian circuits. Calculate the cost for
  each one. Select the circuit with the least cost.

Circuit Mirror Image Weight
A, B, C, D, A A, D, C, B, A 12 8 20 17 57
A, B, D, C, A A, C, D, B, A 12 20 17 15 74
A, C, B, D, A A, D, B, C, A 15 20 20 8 63
74

• Method 1 Make a list of all possible
  Hamiltonian circuits. Calculate the cost for
  each one. Select the circuit with the least cost.

We can see that the first route was indeed the
most efficient--Homer can get back to his nap
after traveling only 57 blocks. Woo Hoo!
Circuit Mirror Image Weight
A, B, C, D, A A, D, C, B, A 12 8 20 17 57
A, B, D, C, A A, C, D, B, A 12 20 17 15 74
A, C, B, D, A A, D, B, C, A 15 20 20 8 63
75
Example The Galactica needs to survey a set of
planets (A, B, C, D, E, F, G) in order to find
water for the Fleet. the Commander has asked the
helm to chart the course that will use the lowest
amount of tylium fuel.
76
Again, we can model this situation using a graph
like the one on the screen to the right.

• this time, however, it is incredibly difficult
  to list all the possible paths, so the homer
  method (I.e. Method I( is not a good choice here.
• in fact, even listing the weight of each edge
  on the graph is hard given the number of
  vertices.
• so, the first thing we will do is list the
  weights in a table. . .

B
75
30
60
65
A
50
C
28
28
48
22
35
40
35
50
29
G
D
30
15
32
13
20
80
40
F
E
77
THE FOLLOWING TABLE TELLS US HOW MANY TONS OF
TYLIUM IT TAKES TO TRAVEL FROM ONE PLANET TO
ANOTHER
A B C D E F G
A 75 50 28 35 15 22
B 75 30 60 80 65 50
C 50 30 40 48 35 28
D 28 60 40 20 30 29
E 35 80 48 20 40 32
F 15 65 35 30 40 13
G 22 50 28 29 32 13
78
THE FOLLOWING TABLE TELLS US HOW MANY TONS OF
TYLIUM IT TAKES TO TRAVEL FROM ONE PLANET TO
ANOTHER
A B C D E F G
A 75 50 28 35 15 22
B 75 30 60 80 65 50
C 50 30 40 48 35 28
D 28 60 40 20 30 29
E 35 80 48 20 40 32
F 15 65 35 30 40 13
G 22 50 28 29 32 13
METHOD I IS IMPRACTICAL HERE (WE HAVE
(7-1)!6!720 POSSIBLE HAMILTONIAN CIRCUITS!) SO
WHAT CAN WE DO IN THIS CASE?
79
Milestones

• 1954, one major city per contiguous state and
  Washington, D.C.
• 1980, 318 vertices
• Took 7 years
• Drilling application

80
Proctor and Gamble Game
81
Milestones

• Most recently 15,112 cities in Germany
• April 21, 2001
• Approx. 40,500 miles
• Used 110 different computers from Rice and
  Princeton University
• Total runtime?

82
Work in Progress World Tour

• 1,904,711 cities worldwide
• Keld Helsgaun
• March 2004
• .075 to .076 from estimated lower-bound

83
USA Towns of 500 or more population 13509
cities 1998.
84
Sweden 24978 Cities 2004
85
Modern Applications

• Traveling Salesmen?
• Drilling applications
• Delivery efficiency
• Semi trucks
• School bus routes

86
For any query Feel Free to ask