Robert J. Marks II

1
Does Evolution Require External Information? Some
Lessons From Computational Intelligence??????????
--?????????

• Robert J. Marks II
• Distinguished Professor Of Electrical and
  Computer Engineering

2
Abstract??

• Engineers use models of science to improve
  quality of life.  Computational intelligence is
  such a useful engineering tool. It can create
  unexpected, insightful and clever results.
  Consequently, an image is often painted of
  computational intelligence as a free source of
  information. Although fast computers performing
  search do add information to a design, the needed
  information to solve even moderately sized
  problems is beyond the computational ability of
  the closed universe.  Assumptions concerning the
  solution must be included.  For targeted search,
  the requirement for added information is well
  known.   The need has been popularized in the
  last decade by the No Free Lunch theorems.  Using
  classic information theory, we show the added
  information for searches can, indeed, be
  measured.  The total information available prior
  to search is determined by application of
  Bernoulli's principle of insufficient reason. The
  added information measures the information
  provided by the evolutionary program towards
  achieving the available information.  Some
  recently proposed evolutionary models are shown,
  surprisingly, to offer negative added information
  to the design process and therefore perform worse
  than random sampling.

3
What is Evolutionary Computation?????????

• Simulation of Darwinian Evolution on a Computer

How good is each solution?
A set of possible solutions
Computer Model
Survival of the fittest
Next generation
Mutation
Keep a set of the best solutions
Duplicate, Mutate Crossover
4
Search in Engineering Design???????

• Can we do better? Engineers
• Create a parameterized model
• Establish a measure designs fitness
• Search the N-D parameter space

5
Random Search You are told Yes No (Success
and no success)????????????
Target
6
Directed Search Information is given to
you????????????

• e.g.
• Warmer!
• Steepest Descent
• Conjugate Gradient Descent
• Interval Halving

Target
7
Blind Searches... ????

• Monkeys at a typewriter
• ??????

27 keys Apply Bernoulli's principle of
insufficient reason in the absence of any prior
knowledge, we must assume that the events have
equal probability Jakob Bernoulli, Ars
Conjectandi'' (The Art of Conjecturing), (1713).
Information Theoretic Equivalent Maximum
Entropy (A Good Optimization Assumption)
8
Random Searches... ????...

• IT WAS A DARK AND STORMY NIGHT
• ??????????

2730 8.73 x 1042 1 (143 bits) The same odds
as choosing a single atom from over twenty
trillion short tons of iron.
Using Avogadro's number, we compute 2730 atoms
times 1 mole per 6.022 1023 atoms times 55.845
grams per mole times 1 short ton per 907,185
grams 1.22 x 1012 short tons.
9
Converting Mass to Computing Power????????

• Minimum energy for an irreversible bit (Von
  Neumann-Landaurer limit
• ln(2) k T 1.15 x 10 -23 joules
• Mass of Universe 1053 kg. Convert all the mass
  in the universe to energy (Emc2) , we could
  generate1 7.83 x 1092 Bits
• Assume age of universe is 13.7 billion years.
• Making the universe conversion to energy every
  nanosecond since the big bang gives only
  3.4 x 10119 Bits

1. Assuming background radiation of 2.76 degrees
Kelvin
10
Random Searches... ????...

• A Definition of Impossible
• ????????

Convert all the mass in the universe to energy a
billion times per second since the big bang
10120 bits
120
10
? Impossible
11
How Long a Phrase? ????????
Target

• IN THE BEGINNING ... EARTH
• JFD SDKA ASS SA ... KSLLS
• KASFSDA SASSF A ... JDASF
• J ASDFASD ASDFD ... ASFDG
• JASKLF SADFAS D ... ASSDF
• .
• .
• .
• IN THE BEGINNING ... EARTH

Expected number NL
12
How Long a Phrase from the Universe?
?????????????
Number of bits expected for a random search

• pN-L
• 10120 bits NL log2 NL

For N 27, pN-L L 82 characters
13
Prescriptive Information in Targeted
Search???????????
14
Fitness ???
Each point in the parameter space has a fitness.
The problem of the search is finding a good
enough fitness.
15
Search Algorithms????
Steepest Ascent Exhaustive Newton-Rapheson
Levenberg-Marquardt Tabu Search Simulated
Annealing Particle Swarm Search Evolutionary
Approaches
Problem In order to work better than average,
each algorithm implicitly assumes something about
the search space and/or location of the target.
16
No Free Lunch Theorem???????

• With no knowledge of where the target is at and
  no knowledge about the fitness surface, one
  search performs, on average, as good as any
  another.

17
No Free Lunch Theorem Made EZ????????????
Find the value of x that maximimizes the fitness,
y.
y
x
Nothing is known about the fitness, y.
18
Quotes on the need for added information for
targeted search ???????????????

1. unless you can make prior assumptions about the
   ... problems you are working on, then no search
   strategy, no matter how sophisticated, can be
   expected to perform better than any other Yu-Chi
   Ho and D.L. Pepyne, (2001).
2. No free lunch theorems indicate the importance
   of incorporating problem-specific knowledge into
   the behavior of the optimization or search
   algorithm. David Wolpert William G. Macready
   (1997).

1. Simple explanantion of the No Free Lunch
   Theorem", Proceedings of the 40th IEEE Conference
   on Decision and Control, Orlando, Florida,
2. "No free lunch theorems for optimization", IEEE
   Trans. Evolutionary Computation 1(1) 67-82
   (1997).

19
Therefore...??

• Nothing works better, on the average, than random
  search.
• For a search algorithm like evolutionary search
  to work, we require prescribed exogenous
  information.

20
Evolutionary Search...????

• Evolutionary search is able to adapt solutions
  to new problems and do not rely on explicit human
  knowledge. David Fogel

BUT, the dominoes of an evolutionary program must
be set up before the are knocked down. Recent
results (NFL) dictate there must be implicitly
added information in the crafting of an evolution
program.
(emphasis added D. Fogel, Review of
Computational Intelligence Imitating Life,
IEEE Trans. on Neural Networks, vol. 6,
pp.1562-1565, 1995.
21
Evolutionary Computing????

• e.g. setting up a search requires formulation
  of a fitness function or a penalty function.

Michael Healy, an early pioneer in applied search
algorithms, called himself a penalty function
artist.
22
Can a computer program generate more information
than it is given
?

• If a search algorithm does not obey the NFL
  theorem, it is like a perpetual motion machine -
  conservation of generalization performance
  precludes it. Cullen Schaffer (1994)
  anticipating the NFLT.

• Cullen Schaffer, 1994. A conservation law for
  generalization performance,in Proc. Eleventh
  International Conference on Machine Learning, H.
  Willian and W. Cohen, San Francisco Morgan
  Kaufmann, pp.295-265.

23
Shannon Information Axioms??????

• Small probability events should have more
  information than large probabilities.
• the nice person (common words ? lower info)
• philanthropist (less used ? more information)
• Information from two disjoint events should add
• engineer ? Information I1
• stuttering ? Information I2
• stuttering engineer ? Information I1 I2

24
Shannon Information????
I
p
25
Targeted Search????
Bernoulli's Principle of Insufficient Reason
Maximum Entropy Assumption
26
Available Information??????
?
This is all of the information we can get from
the search. We can get no more.
27
Available Information Interval
Halving???????????
no 0
This is a perfect search
no 0
yes 1
yes 1
4 bits of information 0011
28
Search Probability of Success.Choose a search
algorithm...????????. ????????
Let
be the probability of success of an evolutionary
search.
From NFL, on the average, if there is no added
information
If
information has been added (or were lucky).
29
Added Information Definition???????
Checks

1. For a perfect search,

all of the available information
30
Added Information????
Checks
2. For a blind query,
no added information
31
Added Information ????
?
Prescriptive information can be NEGATIVE
32
EXAMPLES of PRESCRIPTIVE INFORMATION????????
Interval HalvingRandom SearchPartitioned
SearchFOO Search in Alphabet
NucleotidesNegative Added Information
33
Prescribed Exogenous Information in Random
Searches...????????????
For random search, for very small p
Q Number of Queries (Trials) p success of a
trial pS chance of one or more successes
34
Prescribed exogenous Information in Random
Searches...????????????

1. Added information is not a function of the size
   of the space or the probability of success but
   only the number of queries.
2. There is a diminishing return. Two queries gives
   one bit of added information. Four queries gives
   two bits. Sixteen queries gives four bits, 256
   gives 8 bits, etc.

35
2. Prescribed exogenous Information in
Partitioned Search...?????????????

• METHINKSITISLIKEAWEASEL

yada yada yada

• METHINKSITISLIKEAWEASEL

36
2. Prescribed exogenous Information in
Partitioned Search... ?????????????

• METHINKSITISLIKEAWEASEL

For random search
For Partitioned Search
Hints amplify the added information by a factor
of L.
37
2. Prescribed exogenous Information in
Partitioned Search...?????????????
For perfect search using partitioning
information, set
Iterations From Information
Since L28, if we set
it follows that...
L 27 characters, 26 in alphabet
38
2. Prescribed exogenous Information in
Partitioned Search...?????????????

• Comparison
• METHINKSITISLIKEAWEASEL

Reality For Partitioned Search
For Random Search
There is a lot of added information!
L 28 characters, 27 in alphabet
39
Single Agent Mutation (MacKay) ???????
2. Single Agent Mutation (MacKay)

• Specify a target of bits of length L
• Initiate a string of random bits.
• Form two children with mutation (bit flip)
  probability of ?.
• Find the best fit of the two children. Kill the
  parent and weak child. If there is a tie between
  the kids, flip a coin.
• Go to Step 3 and repeat.
• (WLOG, assume target is all ones)

40
2. Single Agent Mutation (MacKay)???????
41
2. Single Agent Mutation (MacKay)
Single Agent Mutation (MacKay) ???????

• If ? ltlt 1 , this is a Markov birth process.

42
? 0.00005, L128 bits
43
3. Prescribed exogenous FOO Information?????????
??

• FOO frequency of occurrence

E 11.1607 M 3.0129
A 8.4966 H 3.0034
R 7.5809 G 2.4705
I 7.5448 B 2.0720
O 7.1635 F 1.8121
T 6.9509 Y 1.7779
N 6.6544 W 1.2899
S 5.7351 K 1.1016
L 5.4893 V 1.0074
C 4.5388 X 0.2902
U 3.6308 Z 0.2722
D 3.3844 J 0.1965
P 3.1671 Q 0.1962
Concise Oxford Dictionary (9th edition, 1995)
Information of nth Letter
Average informationEntropy
44
English Alphabet Entropy?????

• English
• Uniform
• FOO
• Added information

45
Asymptotic Equapartition Theorem??????
?

• A FOO structuring of a long message restricts
  search to a subspace uniform in ?.

Target
T

• For a message with L characters with alphabet of
  N letters...

FOO Subspace
46
Asymptotic Equapartition Theorem??????

• For King James Bible using FOO, the prescriptive
  information is
• I 6.169 MB.
• Available Information
• I? 16.717 MB
• Can we add MORE information?
• digraphs
• trigraphs

47
4. Example of Negative Prescriptive Information
??????????

• The NFL theorem has been useful to address the
  "sometimes outrageous claims that had been made
  of specific optimization algorithms
• S. Christensen and F. Oppacher, "What can we
  learn from No Free Lunch? A First Attempt to
  Characterize the Concept of a Searchable,
  Proceedings of the Genetic and Evolutionary
  Computation (2001).

48
4. Example of Negative Prescriptive Information
??????????
49
Schneiders EV???????
50
Equivalent to inverting a perceptron
51
Where is the source of exogenous
information??????????
The universe is not old enough nor big enough to
allow the evolution of complex life.

• There must be exogenous information being
  prescribed in some way guide this process.
• Where does it come from? Or is there something we
  are not considering?

52
First Possibility 1. The Wrong Problem
Nonteleological Evolution???????????

• Pure Darwinism claims evolution does not have a
  target and is therefore nonteleological.
• This simply displaces the problem to an
  information rich environment wherein evolution
  can thrive. From where do we get the information
  for this information rich environment?

53
Second PossibilityParallel Universes More
Matter???????????(????)

• Parallel Universes
• (Quantum) The big bang was a wave function one of
  whos states was our universe. There exists
  numerous others.
• (String Theory) The big bang was the product of
  the chance touching of multidimensional branes.
• The universe of universes is now big enough to
  explain life with no external information.
• Problem These are pure conjecture and require a
  faith in scientific materialism.

54
Third PossibilityPanspermia More
Time????????(????)

• Panspermia is a theory supported by Sir
  Francis Crick, Nobel Laureate for discovering DNA.

• Panspermia claims life was planted on earth by
  aliens. Directed panspermia, supported by Crick,
  says this was done on purpose.
• Problem This is pure conjecture and requires a
  faith in science fiction.

55
Fourth PossibilityThe Universe was Created by
God???????????

• This is taught in the Bible of the Jew, Christian
  and Muslim. From Genesis
• In the beginning God created the heavens and
  the earth.
• Now the earth was formless and empty, darkness
  was over the surface of the deep, and the Spirit
  of God was hovering over the waters.

No Information
56
Fourth PossibilityThe Universe was Created by
God???????????

• And God said, "Let there be light," ... "Let
  the water under the sky be gathered to one place,
  and let dry ground appear." "Let the land
  produce vegetation." "Let the water teem with
  living creatures, and let birds fly..." "Let the
  land produce living creatures according to their
  kinds..." ""Let us make man in our image, in
  our likeness, and let them rule over the fish of
  the sea and the birds of the air, over the
  livestock, over all the earth, and over all the
  creatures that move along the ground."

Information Structure
57
Science and Theology?????

• My fellow astronomers are scaling the
  mountains of ignorance, conquering the highest
  peak, pulling themselves over the final rock
  to be greeted by a band of theologians who
  have been sitting there for centuries.

Robert Jastrow, founder and director of NASA's
Goddard Institute for Space
58
Summary of Points...????

• Targeted Evolutionary Computing ? Schaffers
  Perpetual Motion Machine for Information
• Prescribed information can be measured
  analytically or through simulation.
• Some models of simulated evolution have negative
  prescriptive information. Random chance works
  better.
• Prescribed exogenous information should be
  reported in all published models of simulated
  targeted evolution.
• What is the source of exogenous information?

59
Finis
Finis
Finis
Finis
Finis
Finis