Artificial%20Neural%20networks%20for%20Robot%20Control

1
Artificial Neural networks for Robot Control

• Neural Networks 15/16

2
Why use ANNs for robotics?
Outputs
Input data
ANN
Sensory Data
Motor outputs

• All (!) we need for robot control is some method
  of transforming sensory input into motor output
  so why use ANNs?
• Argument from existence proof the most
  successful adaptive machines we know of have some
  form of neural network.
• While our nets are impoverished imitations of
  nature this supports the idea that networks
  ofrelatively simple units can generate adaptive
  behaviour over time,
• If we want to reproduce similar types of
  adaptivity, it might seem sensible to start from
  similar types of system.

3
2. ANNs are extremely flexible, with many ways
that architecture can be modified, (changing
weights to changing the entire architecture. 3.
ANNs are well suited to incorporating mechanisms
such as lifetime learning, potentially enabling
agents to adapt to changing environments 4. ANNs
can take input from a variety of sources,
including both continuous and discrete sensor
readings, and similarly produce discrete or
analog motor outputs 5. Memory can easily be
incorporated into the network through retaining
activity over time, 6. ANNs are quite robust to
noisy input data as would be expected from sensor
data from non-trivial environments
4
Training procedures
Majority of the training methods we have seen are
supervised learning methods implies the
existence of a target output for each input
pattern This is true of some robot control tasks
where desired behaviour is specified precisely eg
Case Western cockroach used supervised
techniques to set parameters to produce a
particular gait However, consider a robot
navigating to a target dont necessarily know
correct trajectory trajectory will depend on
starting conditions environment could be
dynamic we need output over TIME inherently
difficult to do function approximation over time
and also how do we know at which point we went
wrong??? Also behaviour may be reliant on
sensory data from previous time-steps ANNs for
robot control are dynamical systems changing over
time.
5
Also gradient descent techniques require
continuously differentiable functions, thus focus
is on feedforward fully-connected nets (though
limited recurrency is possible) and varying
continuous variables (weights) Not always
possible to differentiate error term Also
consider the change in error from eg removing a
node in a network all or nothing procedure and
could be vastly disruptive. VERY bad for gradient
descent Can use unsupervised mtehods like
reinforcement learning but generally need short
trials and arenas where majority of possible
inputs are available eg robot football shooting
6
Use Evolutionary Algorithms!
Eg GAs, simulated annealing, evolutionary
strategies, genetic programming, evolutionary
programming etc (as long as Good points Avoids
many of the problems of gradient descent as we
only need to know networks overall performance.
Also can incorporate lifetime learning, changes
of architecture etc and can work with wide range
of network models (as long as genetic operators
can be defined) VERY useful as we do not know
what types of network are good Here we will focus
on GAs (Sussex - and my - bias) NOT a panacea for
all search problem ills introduce a whole host
of other problems which we shall explore Not the
only approach eg hill climbing, net crawling etc
7
Basic GA
In GA have a population of genotypes which encode
potential solutions to the problem Every
generation they are all tested on the problem and
assigned a score based on their success known as
their fitness Offspring are created for the next
generation of solutions via recombination between
two parents, followed by application of a
mutation operator Generate the probability of a
genotype being picked as a parent proportional to
its fitness (or fitness ranked over the
population) Artificial natural selection where
traditionally recombination emphasised as the
driving force of evolution However recent work
has focussed on role of mutation
8
- Initialise population of N genotypes - Evaluate
initial genotype fitnesses - Repeat until
termination criteria met - Repeat until N
offspring placed in new population - Two
parents P selected probabilistically
(proportional to fitness) - Two offspring O
created through recombination of P - Offspring
O mutated - Offspring O evaluated - Fittest
offspring placed in new population - Replace
current population with new population There are
an enormous number of variations on the canonical
genetic algorithm above, many of which blur the
distinction between GAs and other evolutionary
algorithms
9
Genetic operators mutation and crossover
The genetic operators used to create offspring
must do 2 things There must be a significant
amount of similarity between the parents and
offspring, heredity, so as to allow exploitation
of current solution There must be variation in
order for evolution to discover new solutions
exploration of nearby areas Operators depend
crucially on the solution representation used, cf
binary bit-flips vs real valued Gaussian
mutations. Also want to generate viable
solutions (cf telecoms networks must be
connected) so genetic operators must be matched
to the problem at hand
111 10000
11101111
11101111
11111111
000 01111
mutation
crossover
10
Solution representation Encoding schemes
Before we design our genetic operators, must
first decide how to represent problem
solutions GAs use a string of values (binary,
real, letters etc) which must be used to encode
all the parameters of the network Split into 2
styles (though really a continuum of types)
direct and indirect
11
Direct Encoding
Direct schemes code all parameters directly Eg
Take matrix of weight values (0 for no
connection), and write out as one long string.
Can work well. Grows with network size.
However, can be bad with respect to heredity Eg
What if circled bit is good bit of network how
can we retain this bit without taking the other
nodes Also can have problems if we want networks
to grow and shrink
12
Problems of variable length genotypes
Often want networks to have the capacity to grow
and shrink thus we will have genotypes of
different lengths in our search spaces Can cause
many problems eg if using direct encoding and
basic crossover could get 5x5 matrix crossed with
2x3 matrix at position 19 Also get problems
with connections if above matrices are crossed
at position 5, ie 1st 5 from 2x3 matrix, these
weights are now all weights to neuron 1 this is
NOT what they were in the 2x3 matrix which has no
knowledge of extra nodes of the 5x5 While these
are not insurmountable they illustrate the deeper
problem that children are unlike parents
13
Indirect Encoding
Can avoid some of these problems by use of
indirect encoding schemes Various forms
developmental schemes (where genotype encodes
growth process of phenotype), cellular encoding,
tree structures and various wild and wacky
ideas Can be useful in eg getting heredity or in
getting repeated self structure (cf Gruau
cellular encoding could replicate network
features n-fold) Often developed
task-specifically One problem is that while
fitness is evaluated on phenotype, movement of
population is in genotype space extra layer of
complexity in working out eg how crossover
affects phenotype
14
Some Examples
GasNet encoding scheme to allow for problems of
variable length genotypes and to allow nodes to
have similar proerties across neurons have used a
spatial connection scheme. In this way, node x
will always connect to nodes in the same region
of the plane. Also node properties kept together
so crossover cant mess up a node
2 styles of encoding a telecoms network Indirect
genotype encoded sets of 2x2 matrices which
represented eigenvalues of dynamical systems ie
sets of attractors in 2d space. Connectiosn sent
out from attractors and attracted to others to
form connectivity. Also had self similarity
operator. Result?
15
Rubbish! Didnt work very well at all. Chaotic
dynamical system so smallest change of genotype
could in principle change whole network. No
heredity. Other style semi-direct want to
ensure connected network so made basic genotype a
minimum spanning tree (ensures theres a path
from evry point to every other in least amount of
connections). Then added in extra connections.
Result? OK Not sure about heredity with respect
to spanning tree, but had nice crossover operator
in general add a connection to child with
probability 0.9 if both parents have it, and 0.4
if only one has it. By varying probabilities can
get smaller/bigger nets
16
Fitness functions
Need a fitness function to evaluate robots
performance bit of a black art Problematic as
it is of central importance to evolutionary
computation methods if there is little chance of
differentiating between good and bad solutions,
the evolutionary process cannot hope to succeed.
Basically defines the search space Ideally, there
should be smooth paths in the problem space
leading to the optimal solutions but in reality
may not be possible Basically one should ensure
that there is a gradient for evolution to follow
and avoid having large local optima (though this
process is generally post-hoc ie make fitness
function, population gets stuck, design new
fitness function which avoids local optimum,
population gets stuck again. Repeat ad nauseum
till you regret ever criticising wonderful
gradient-descent techniques
17
Noisy fitnesses
Also often have noisy fitness, often due to not
being evaluated overexactly the same conditions
(eg where sample sets for training are
potentially huge, so fitness evaluated over some
smaller set) EG robot fitnesses are often highly
dependent on the initial conditions
Alternatively environment could be a source of
noise Typically this noise will obscure
differences between the fitnesses of neighbouring
solutions, reducing the performance of the
evolutionary process (although sometimes the
noise can be helpful to eg allow populations to
escape from local optima, or smooth search space)
18
Selection
Related to evaluation of fitness is the selection
applied to solutions If there is not sufficient
selective pressure to drive the evolutionary
process to better parts of the landscape, much
time will be spent evaluating solutions of poor
fitness. By contrast, if the selection pressure
is too strong the evolutionary process will halt
at the first local optimum reached, with little
chance of escaping. Highlights the conflict
between allowing exploration of the problem
space, and exploitation of local regions of the
space
19
Search spaces
As stated earlier, the fitness function defines
the search space of the problem we are looking
at Search space is N-dimensional where N is
(maximum!) length of bit string How we move
through the search space is defined by our
recombination operators. Search space can
therefore be seen to be a connected graph where
connected points are those that can be reached by
crossover and mutation Also depending on
operators will be more likely to get to certain
destinations If operators are well designed in
terms of heredity should be able to get to all
nearby areas of space
20
Fitness landscapes
Often search space viewed as a an N1 dimensional
landscape where extra dimension is the fitness eg
Bit string of length 2 gives us nice landscape
below
Can be a useful metaphor (despite Inmans
protestaions) but ONLY if you reject all cosy
notions of local maxima and minima Eg GasNet
search-space average of 200 dimensions lots of
places to go
Also standard mutation operator can in principle
take us to any part of the space Also have
addition/deletion of nodes difficult to view as
movement Also, noisy fitness how to define
maxima if fitness is a distribution???
21
Epistasis, ruggedness and local optimality
If fitness dependent on a non-linear combination
of the genotype loci, the genotype is said to be
epistatically-linked. Ie individual locus
fitnesses are dependent on the context of other
loci values and inter-locus interactions This
will generally be the case for ANN robot
controllers Epistatically-linked genotypes give
rise to the two major properties of fitness
landscapes thought to influence search dynamics,
ruggedness and local optimality. Ruggedness is
regarded as similar to fitness noise, where
direction to good solutions may be obscured by
local noise By contrast, local optimality is
typically thought of in more global terms, with
landscapes containing numbers of deceptive peaks
However there is no rigorous distinction between
the two properties
22
Search space properties
Smooth vs epistatic
Global vs local optima
Neutrality
Other?
23
Neutrality
Recently much work has gone into analysing
neutrality of landscapes (eg RNA, nkp, evolvable
hardware and some robotics) ie landscapes where
one can move to points of equal fitness moving
along a neutral network Evolution on fitness
landscapes with high levels of neutrality is
characterised by periods when fitness does not
increase (fitness epochs) interspersed by short
periods of rapid fitness increase (epochal
evolution or punctuated equilibrium) Adaptive
evolution on neutral landscapes has shown that
populations tend to move to areas of space which
have more neutral neighbours ie the neutral
evolution of robustness Neutrality may be of use
in escaping from (nearly) locally-optimal
solutions, but in practice in high dimensional
spaces, quite hard to tell if one is moving
neutrally or hovering around a local optimum
24
Neuron Models
Many types of neuron model used in
robotics CTRNNs based on leaky integrator neuron
model from computational neuroscience Spiking
models similar to above but with a spike
generated when activation reaches a
threshold GasNet models incorporate an abstrcat
notion of a diffusible neuromodulator into an
ANN Firing rate based models Etc ..
25

• However, how can we decide what type of neuron
  model to use for a particular task?
• Similarly, how do we know if we have good fitness
  functions/recombination operators to use in
  conjunction with our neuron model?
• Can use intuition, or try several combinations.
  But will our results tell us what we want to know
  about the problem we were working on
• Why did a particular neuron/GA combination work
  well?
• Were our intuitions correct?
• What are the implications for generating a more
  successful model?

26
An Example GasNet evolution
GasNet evolves faster than NoGas over range of
reombination schemes
and mutation rates and connection
architectures and robotic problems
??WHY??
27
GasNets Background

• Classically neurotransmission is viewed as
  occurring Point-to-point at the synapse i.e.
  locally
• Occurs over a short temporal scale
• Overriding metaphor is electrical nodes connected
  by wires
• Inspiration for standard connectionist ANN

?
28
Neuromodulation by nitric oxide (NO)

• Recently neuromodulatory gases have been
  discovered (NO, CO, H2S). By far the most studied
  is NO
• Small and non-polar ? freely diffusing
• Act over a large spatial scale volume signalling
  
• Act over a wide range of temporal scales (ms to
  years)
• Modulatory effects
• Interaction between neurons not connected
  synaptically
• Loose coupling between the 2 signalling systems
  (electrical and chemical) i.e. neurons that are
  connected electrically are not necessarily
  affected by the gas and vice versa.
• ? new style of ANN?

29
Inspiration for new form of ANN GasNets

• Node emits 1 of 2 gases due to high electrical
  activity or high gas concentration
• Computationally fast, crude diffusion method, but
  space and time crucial, local processes

30
Ojt tanhkjt(SwijOit-1 Ij) bj
3. Gases diffuse through the network and alter
slope of transfer functions of other neurons in
concentration-dependent manner. 4. Gas 1
increases gain, gas 2 decreases gain.
31
Analysis of search space properties
If networks evolve faster they must, in some
sense, be making the space of solutions easier to
search in (smoother? More densely packed with
good solutions? Less optima? More
neutral??) Analysis will hopefully tell us what
the search space properties are like and what
features of our networks are good for
search Also can help us to understand the
dynamics of an EA search through high-dimensional
space not well understood Hopeful approach since
many of the intuitions we have about how EAs
search spaces have come from such analyses Eg
work on nk, nkp and royal road landscapes etc
have attempted to address the role of neutrality,
crossover vs mutation and much more
32
Properties to examine
Smooth vs rugged
Global vs local optima
Neutrality
Other?
33
However
Abstract mathematical landscapes like nk and nkp
are generally designed to have tunable
ruggedness, neutrality, local modality
etc. Real-world problems have no direct link
between solution architecture and landscape
properties And maybe no understandable link
between landscape properties and evolutionary
dynamics (how does adding virtual gas affect
neutrality??? What is neutrality in a noisy
space????) Weve found no explanation for GasNet
evolvability in terms of fitness landscape
properties (partly because 99.9 of real spaces
evaluate to 0 fitness meaning standard measures
see them as a homogeneous flat landscape)
34
EG Mutational robustness same
35
What about functional analysis?

• Oscillator sub-networks very commonly evolve
• Node 5 provides electrical stimulation to RMnode,
  this causes gas1 emission from RMnode, this
  causes gas2 emission from node 5, this reduces
  gain of Rmnode which decreases activity which
  shuts off gas emission from RM and hence from 5
  and cycle begins again ..

36
Rmnode when gain high
Rmnode when Gain low
Interaction of chemical and electrical provides
transition between 2 regimes with diffusion
controlled transition timings
37
GasNet and NoGas Timers (1)
38
Timer sub-circuits naturally become active before
object finder circuits
39
GasNet and NoGas Timers (2)
GasNet timer build-up of gas concentration Simpl
e architecture, mechanisms easily tuned?
NoGas timer 3 fully connected nodes. Convoluted
architecture, difficult to tune?
40
Re-evolution in environment with changed time
scales (func )
GasNet (x2) NoGas GasNet (x 0.25) NoGas
Num cases 20 20 20 20
Mean re-eval fitn. 0.17(0.07) 0.15(0.06) 0.36(0.1) 0.21(0.02)
Mean re-evol gens 10(5) 409(336) 30(31) 591(346)
Median re-evol 10 360 19 608
41
Re-evolution in environment with changed time
scales (sample)
GasNet (x2) NoGas GasNet (x 0.25) NoGas
Num cases 20 20 20 20
Mean re-eval fitn. 0.27(0.13) 0.26(0.18) 0.35(0.27) 0.29(0.19)
Mean re-evol gens 107(190) 240(363) 108(229) 116(252)
Median re-evol 36 49 13 21
42
Conclusions

• It seems that easy temporal adaptivity is an
  important feature in GasNet evolvability
• Dynamics used in surprising ways (e.g. sensory
  noise filters), so important even in reactive
  tasks
• Evolution and re-evolution of various kinds of
  rhythmic networks backs this up

43
But

• not whole story
• Have investigated several GasNet variants which
  have improve the performance of the original
• They have same temporal properties so what is
  going on there??
• Maybe need to look at more phenotypic properties
  (eg coupling of gaseous and electrical signalling
  mechanisms)
• To do this must introduce GasNets and variants

44
Diffusion in Original GasNets
Based on the spatial distribution of NO produced
by a single spherical neuron
1. Gas cloud centred on emitting node and builds
up linearly with time (to a maximum) at a
genetically specified rate
2. Gas varies spatially as an inverse
exponential exp(-d2/r).
45

• However, initial gas diffusion model was
  intentionally simplistic
• Cannot capture the rich range of spatio-temporal
  properties seen in real systems
• Therefore decided to develop 2 new versions
  incorporating aspects of NO signalling seen in
  nervous systems
• Will hopefully lead to more powerful/evolvable
  robotic systems
• Also tests the potential utility of the features
  in real nervous systems

46
Mammalian cortical plexus

• NO involved in mediating link between neural
  activity and blood flow
• Fibres are too small to generate an effective NO
  signal individually
• NO from many fibres summate NO signal different
  to that from single neuron

47

• Fineness of fibres leads to a uniform signal
• Combined effect maintains high concentration
  levels over large volume
• Also delay until fibres interact serves to act
  as a noise filter Plexuses of fine fibres signal
  persistent neuronal activity to blood vessels

48
Plexus GasNet model

• 1. Gas cloud has uniform spatial distribution

• 2. Gas clouds centred in a genetically specified
  position in the network plane

49
Receptor GasNet

• In nervous systems, only neurons with one of the
  receptors for NO will be affected
• Each node may have quantities of receptors (none,
  medium, maximum)
• Receptor specific modulations

Gas concentration
Receptor concentration
50
Receptor GasNet modulations

• Increase gain
• Decrease gain
• Activation includes a proportion of previous
  activation
• Transfer function switched

One that was particularly successful was using
only one gas which increased gain
51
Example experiment
Task Evolution of visual object discrimination
under noisy lighting with minimal vision
52
Experiments
Discrimination between
and
and
or
or
Compared Original GasNet, Receptor GasNet and
Plexus GasNet
53
Evolvability results (expt 1)
54
Evolvability results (expt 2)
55
Why do new versions improve evolvability?
In initial GasNet model genotype to phenotype
mapping means that theres quite a tight coupling
between the electrical and chemical processes
Electrical connections depend on spatial
organisation of nodes Also, gas diffuses locally
to all nearby neurons gt if electrically coupled
likely to be chemically coupled New models allow
for a more flexible
56
Coupling (expt 1)
57
Evolvability and coupling

• Gardner and Ashby (70) showed that random systems
  of multiply connected weakly interacting
  components or sparsely connected strongly
  interacting components are more likely to be
  stable
• New GasNets incorporate flexibly coupled
  processes with distinct characteristics
  (electrical, chemical)
• Helps to satisfy the conflicting pressures for
  genotypic instability and phenotypic stability
  needed for successful evolution (Conrad, 90)
• Allows non-destructive tuning of functionality
  of one against the other (eg elec does bright
  object, chem does discrimination)

58
Discussion future directions

• 2 new models presented which significantly
  improve network evolvability
• Both systems are flexibly coupled
• Clearly not the whole story
• A better measure of the degree of coupling is
  needed (allow a continuum of coupling)

For more deatil (on any of past 2 lectures) see
http//www.cogs.susx.ac.uk/ccnr