Experimental Evaluation of Heuristic Optimization Algorithms

1
Experimental Evaluation of Heuristic
Optimization Algorithms

• Radin Uzsoy (2001)

2
Heuristic Optimization Algorithms

• Good Feasible Solutions
• Needed because of
• Complexity of the Problem
• Time Limitations

3
Main Question

• Is my heuristic algorithm work well ?
• How good the performance of a given heuristic ?
• Empirical Experiments

4
Two Criteria

• How Close to Being Optimum
• Expected Deviation
• Worst Case Performance
• How Fast Obtaining Solutions
• Polynomial Time

5
Focus of the Paper

• How such an Experiment should be done ?
• What are the Important Issues ?
• Experimental Design
• Source of test Instances
• Measures of Performance
• Analysis of Result

6
Purposes of Evaluation

• Research
• New methodologies for a problem
• What works, What does not
• Development
• Evolving the Most Efficient Heuristic
• Not Most of the Publication is Research Type

7
Time Availability Solution Quality

• Design Problems
• Abundant Time Availability
• Need of Good Heuristic Solution
• Telecommunication Network Design
• Control Problems
• Frequent Decision Over a Short Time Horizon
• Quality is Secondary Criteria
• Shop-floor Scheduling

8
Time Availability Solution Quality

• Planning Problems
• Intermediate level time availability
• Quality is more important
• Aggregate production planning

9
Design of Computational Experiment

• Problem A generic type of problem
• Single Machine Scheduling for min Flowtime
• Instance A particular numerical case of the
  problem

10
Instances vs Algorithms
11
(No Transcript)
12
Calculating Necessary Runs

• Problem Parameter Factor
• 8 Factor with 2 levels
• Instances for Each Parameter Factor Combination
• 3 instances
• Algorithm Characteristic
• 3 characteristic with 2 levels
• Run with Different Random Seeds
• 5 runs
• 28 x 3 x 23 x 5 30 720 runs

13
Refining Experimental Design

• Pilot Runs
• Gives idea about experiment dimensions
• Eliminating factors with minimal effect
• Determining which factor levels to test
• Expecting variability in outputs
• Danger
• Setting heuristic parameters to perform for good
  specific instances

14
Refining Experimental Design

• Full Factorial Design
• Running all algorithms on all instances for all
  combination of factorials
• Better evaluation, more computational effort

15
Refining Experimental Design

• Fractional Factorial Design
• For a carefully chosen subset of factors
• Approximately same evaluation, less effort
• Not appropriate when multiple replication is
  needed
• See Montgomery (1991) for statistical details
• Montgomery, D.C. (1991). Design and Analysis of
  Experiments, 3rd edn. New York John Wiley.

16
Refining Experimental Design

• Randomization vs Blocking
• Randomization to estimate the mean deviation from
  optimum
• Blocking to compare the effect of different
  factors

17
Balancing Time Quality

• Simple search vs Complicated Search
• Complicated one use more computational resource
• Fairer test by letting simple search to run
  multiple runs
• Same amount of computational resource usage in
  each search

18
Sources of Test Instances

• Main requirement for evaluation
• Instances should represent interested problem
  domain (problem characteristics)
• Otherwise performance on a different problem is
  measured

19
Real Word Data Sets

• Best if available
• Rarely possible
• Time consuming to collect
• Not in desired format
• Random variants of real data sets
• Macro factors taken from real word applications
• Details are randomly generated

20
Published and Online Libraries

• Provide good comparison with existing heuristics
• Disadvantages
• Not realistic anymore
• Different purposes of evaluation
• Biased instances (good for the published
  heuristic)

21
Randomly Generated Instances

• If none of the other sources available
• Random instances can be controlled with
  parameters

22
Advantages

• Desired problem characteristics can be
  implemented
• Instances can be regenerated for future use
• Unlimited instance availability

23
Pitfalls of Random Generation

• If the performance on the randomly generated data
  is good, will the performance be good for
  different setting
• How the parameters should be selected
• Some times correlation is needed

24
Performance Measurement In Solution Quality

• Most Problems are NP-Hard
• Optimum solution is available only for tiny cases
• Bounds on optimum is available
• Most of the bounds are not sharp

25
Exact Solution of Small Instances

• Widely used
• Least favorable by Radin Uzsoy(2001)
• Cost high computational effort
• Not statistically realistic
• Cases where larger the instance, the better
  heuristic works
• ex. Quadratic Assignment Problem
• Small instances are good for some search
  mechanism

26
Bounds on Optimal Value

• If bound is loose, expected deviation is not
  representative
• If bound is tight, too much computational
  resource is needed
• Combining bounds and optimum solution for small
  instances

27
Built in Optimal Solution

• Generating optimum solution, while generating the
  random combinatorial problem
• ex. Traveling Salesman, Graph Partitioning, etc
• (references at pg. 275)
• Instances must have a specific structure
• Does not represent whole problem domain

28
Statistical Estimation of Optimal Values

• For problems with enormous feasible solutions
• Use sample solutions to estimate distribution of
  the optimum value

29
Statistical Estimation of Optimal Values

• Take n independent sample solution, and let zi
  be minimum of sample
• a, b, c can be estimated from this samples

30
(No Transcript)
31
Best Known Solution

• From other researchers
• Having very long runs
• Using multiple heuristic solution methods

32
Example

• Single machine scheduling to minimize Lmax
• Without release dates but with sequence
  dependent release dates

33
Analysis of Results

• The averages and standard deviations of resulting
  performance measures for each algorithm should be
  calculated
• Averages help comparing the performance of
  different algorithms
• Statistical significance tests can be made to see
  whether a sampling error occurs

34
(No Transcript)
35
Analysis of Results

• Two-way tables of sample means
• How various factors affect the performance

36
(No Transcript)
37
Analysis of Results

• Analysis of Variance (ANOVA)
• Effect of multiple factor on performance
• Whether a factor is significant or not
• Which factor has negligible effect
• Which factors have interactions

38
(No Transcript)
39
Comparision of Means
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)