Strong Random Correlations in Complex Systems

1
Strong Random Correlations in Complex Systems

• Imre Kondor
• Collegium Budapest and Eötvös University,
  Budapest
• Parmenides Foundation, Munich
• ECCS 2008, the annual conference of the European
  Complex Systems Society 
• The Hebrew University, Jerusalem
• September 15-19,  2008

2
This is a report on work in progress.Collaborato
rs

• Albert-László Barabási, Boston
• Alain Billoire, Saclay
• Nándor Gulyás, Budapest
• Jovanka Lukic, Rome
• Enzo Marinari, Rome

3
Summary

• Complex systems are (nearly) irreducible
  (incompressible), depend on a large number of
  variables in a significant way.
• Irreducibility is related to (implies?) large,
  random correlations
• Illustration on two toy models on a spin glass
  and a random cellular automaton
• Some consequences e.g. simulation of such
  systems is a delicate issue, the result depends
  on tiny details, initial and boundary conditions.

4
Preliminary considerations

• It is plausible that in a system that depends on
  a large number of variables the correlations
  between its components must be long-ranged in
  some sense.
• As complex systems are not translationally
  invariant, long-ranged means strong
  correlations between many pairs, though not
  necessarily geometrical neighbours.
• The usual behaviour of correlations in simple
  systems is not like this correlations fall off
  typically exponentially which is why simple
  systems fall apart into small, weakly correlated
  subsystems, and have low effective dimension.

5
The difficulties of defining complexity

• There are nearly as many complexity definitions
  as there are authors in complexity.
• The AIT definition the length of the shortest
  algorithm that is able to produce a given string
  is the measure of the complexity of the string.
• Some authors emphasize emergence, confluence of
  scales, nonlinearity, unpredictability, path
  dependence, historicity, multiple equilibria, a
  mixture of sensitivity and robustness, learning
  and adaptability, and, ultimately,
  self-reflection, self-representation,
  consciousness as characteristics of complex
  systems.

6

• When trying to formulate a common policy of
  sponsoring complexity research in Europe
  Complexity-NET, a network of European funding
  agencies, came to the conclusion that finding a
  compelling definition was a hopeless endeavour on
  which no more time should be wasted.
• A possible alternative is to list examples
  complex systems include the living cell, the
  brain, society, economy, etc.

7
Irreducibility

• G. Parisi at the 1999 STATPHYS Conference in
  Paris A system is complex if it depends on many
  details.
• This suggests the idea of using the degree of
  irreducibility, perhaps the effective
  dimensionality (the number of variables) of the
  simplest model one can construct to describe the
  system to a given level of precision, as a
  measure of complexity.
• NB This definition shares the shortcomings of
  the algorithmic complexity concept it assigns
  maximal complexity to noise, and it is probably
  impossible to decide which model is the simplest.

8
The incompressibility of history ?

• For the want of a nail the shoe was lost
• For the want of a shoe the horse was lost
• For the want of a horse the battle was lost
• For the failure of battle the kingdom was lost
• And all for the want of a horseshoe nail.
• Background The Battle of Bosworth Field in 1485,
  between the armies of King Richard III and Henry,
  Earl of Richmond, that determined who would rule
  England.

9
A more serious example

• 10 days survival probability of patients after a
  heart attack. Depends on some 40 factors. Such a
  model cannot be parametrized even on a population
  of 10 million (overfitting). Yet health policy
  decisions depend on such analyses.
• (Peter Austin at the 2007 AAAS meeting)
• The simplest tool to analyze such problems is
  linear regression.

10
Linear regression

• .

11

• Ideally, the number of dimensions N is small and
  the length of the available time series T is
  long. Then the estimation error is small, and the
  model works fine.
• If the system is complex, however, we will have
  a very large N, and that raises serious
  estimation error and convergence problems.
• When a huge number of regression coefficients are
  roughly equal, we do not have structure, the
  model produces noise.
• It may happen, however, that the regression
  coefficients are not equal, but do not have a
  cutoff beyond which they would become
  insignificant either they may not have a
  characteristic scale, but fall off like a power.

12

• But large regression coefficients imply large
  correlations
• If the independent variables are uncorrelated
  then the regression coefficients are proportional
  to the covariances between the dependent variable
  and the idependent variables

13

• This suggests the idea to look into some toy
  models and see if large correlations may indeed
  be a characteristic feature of complex systems.
• Two toy models will be studied here
• The /-J long range spin glass
• and a
• Random cellular automaton

14
The spin glass A model of cooperation and
competition

• An Ising-like model with random couplings
• where are randomly scattered
  over the lattice or graph. For simplicity we keep
  to the complete graph in the following.
• 
  
  

15
On a small complete graph, e.g

• The red edges represent negative
  (antiferromagnetic)
• couplings. Spins linked by such a negative
  coupling would like to point in opposite
  directions.

16

• The optimal arrangement of the spins is a
  distribution of plus-minus ones, correlated with
  the distribution of couplings in a complicated
  manner. Even the optimal arrangement can contain
  a lot of tension not all the couplings can be
  satisfied simultaneously.

17
Frustration

• The presence of negative couplings leads to
  frustration one may have two friends who hate
  each other. Such a trio cannot be made happy. In
  the little example
• the triangles containing an
• odd number of red edges are frustrated.

18

• Frustration makes the overall bonding much
  weaker the ground state energy is higher than
  for a pure system. At the same time the
  degeneracy of low lying states (the multiplicity
  of states with the same energy) is much enhanced.
  
• For large N, the low temperature structure of
  such a model can be extremely complicated, with
  several nearly degenerate minima and their basins
  of attraction cutting up the set of microscopic
  states into a set of pure states or phase
  space valleys.

19

• A central concept in the characterization of this
  structure is that of the overlap
• which measures the degree of similarity between
  two states.

20
Correlations in ordinary lattice models

• Normally, correlations fall off exponentially
  except
• at the critical point
• in the ordered phase of models with a broken
  continuous symmetry.
• An Ising spin glass does not have any continuous
  symmetry, there is no a priori reason to expect
  long range correlations.

21
Correlations in spin glasses

• Due to the random structure of the model, the
  correlations behave in a
  chaotic, random manner as a function of distance.
  When averaged over the random distribution of the
  couplings they become a trivial Kronecker delta
• For this reason it has been customary to study
  higher order average correlations, often defined
  for a given average overlap.

22

• Some of these correlation functions

23
Correlation in one phase space valley

• A natural combination of the above correlation
  functions, computed in the Gaussian approximation
  via replica field theory, turned out to be
  long-ranged (De Dominicis, Temesvári, I.K.)

24
Correlations between distant valleys

• Remarkably, the overlap between correlation
  functions belonging to phase space valleys with
  zero overlap also was found long-ranged
• So the average correlations are long-ranged in
  spin glasses.

25
Correlations in a given sample

• It may also be of interest to look into the
  distribution of correlations as random variables
  in a given sample. In order to do this, we
  measured all the N(N-1)/2 correlations
  and ranked them according to magnitude. Exact
  enumeration on small systems (up to N20) and
  numerical simulations up to N 2048 indicate
  that the correlations are anomalously large in
  the low temperature spin glass.
• Some preliminary results follow.

26
Sorted correlations for two samples of size
N128, at low temperature (T0.4), averaging over
all microstates
27
The same for two samples of size N 2048, at
T0.4, averaging over all microstates
28

• The sorted distribution is the inverse of the
  cumulative distribution function.
• For large N the sample to sample fluctuations
  disappear, the distribution can, in principle, be
  calculated via replicas.
• The sorted correlations suggest that their
  probability distribution is very broad, maybe
  uniform, or even bimodal!
• Note the apparent symmetry of the sorted
  distribution which does not correspond to any
  exact symmetry of the system.

29
When we go above the critical temperature T1
N128, T1.3

• Note the change
  of scale! Most correlations
  are very small now.

30
N2048, T1.3

• For this large system the
  correlations are even smaller.
  Clearly, for N large the
  number of large correlati
  ons is O(N), which is negligible
  on the scale of the
  figure, O(N²)

31
The main points

• There is a marked difference between the high and
  low temperature phases. Correlations are strong
  all through the low temperature phase.
• A replica calculation shows that the distribution
  of
• is symmetric, and its
  second moment is the same as the second moment of
  the overlap distribution P(q).
  

32
A random cellular automaton RCA

• The model is a finite T variant of the Kauffman
  automaton.
• It is a collection of binary variables again,
  this time living on a 2d lattice. They update
  their state according to the configuration of
  their neighbours.

33
RCA update rule
Table of interactions
Hamilton function with the Ks N(0,1)
distributed
34

• From this point on the simulation of the model
  runs along the usual Monte Carlo path
• The results are shown parallel to the same for
  the Ising model.
• Note that this is a lattice model, so there is a
  geometry behind it (distance, neighbourhood, etc.
  make sense), and we can ask questions about the
  geometric distribution of large correlations.

35
Sorted correlations
36
Distribution functions
37
Density functions
38
RCA vs. Ising model (standard deviation of
correlations)
39
Max correl vs. distance
40

• Strong and weak correlations are randomly
  scattered about the system. Two strongly
  correlated spins may not be connected by strongly
  correlated paths.
• See little demo

41
Linear regression (RCA model)
The sorted coefficients (N100, T2)
42
Concluding remarks

• Randomly distributed large correlations may be a
  general characteristic of complex systems. In
  this sense complex systems may be regarded as
  critical in a wide region of parameter space.
• This property may explain their sensitivity to
  changes in the control parameters, boundary
  conditions, initial conditions and other details,
  even for large sizes (e.g. chaos in spin
  glasses).
• It also calls for caution when doing, and drawing
  conclusions from, simulations of such systems.
• Strong random correlations redefine the geometry
  of the system. Problems of the RG and the
  thermodynamic limit.

43
Thank you!
44
Appendix
45
The Ising model a model of cooperation

• N spins i 1,2,,N, having a binary choice
• . The spins are coupled by
  ferromagnetic interaction, they want to minimize
  the energy
• The magnetic field h wants to align all the
  spins with itself.

46

• This is a simple description of magnetism and a
  host of other cooperative phenomena.
• The total number of microscopic arrangements of
  the spins is . The model has two optimal
  states (ground states) All spins 1 (up), or -1
  (down).
• Finite temperature some spins fail to comply.

47
Averaging

• Averages at temperature T are calculated over the
  whole ensemble of microscopic states, with the
  Boltzmann-weight exp-H/T.
• Alternatively, we define a Monte Carlo dynamics
  on the system, and measure time averages.
• Pick initial state, calculate its energy .
  Flip randomly chosen spin, calculate new energy
  . Accept new state if lt 0,
  
• and accept new state with
• probability , if
  gt 0.

48
The underlying geometry

• Such a model can be implemented on a regular
  lattice, like the 2d square lattice shown here

49

• on a random graph

50

• or on a complete graph
• Full circles mean spins 1, empty ones -1.

51
Phase transition

• At high temperatures the acceptance rate of bad
  moves is nearly as large as that of the good
  moves, the system is totally disordered. As T is
  lowered, the tendency of cooperation gradually
  overcomes thermal agitation. If the graph is
  sufficiently large and connected, at a critical
  temperature a sharp transition takes place to a
  spontaneously ordered state, with the majority of
  spins pointing, say, up, even without the help of
  the external field h.
• For the 2d square lattice the value of this
  critical temperature is .

52
Correlations in the Ising model

• The correlations between the
  spins at lattice sites i and j are short-ranged
  (fall off exponentially with distance) above the
  critical temperature. (The angular brackets
  denote the thermal average.)
• A typical formula is
• where ? is the coherence length.

53

• Below the critical temperature the system is
  polarized, so K tends to a constant, but its
  connected part
  is decaying exponentially again.

54
The critical state

• As the temperature goes to its critical value,
  ,
• the coherence length diverges
  .
• Right at the critical point correlations in the
  system become long-ranged. There is no
  characteristic distance beyond which they would
  become negligible, they fall off like a negative
  power of the distance

55

• As a direct consequence, the system becomes
  extremely sensitive to changes in the control
  parameters, such as the external field even an
  infinitesimal h provokes a large response.
• Note, however, that in order to reach the
  critical point one has to fine tune the
  parameters of the model, this is an exceptional
  point where even the humble Ising model becomes
  complex.

56
Models with broken continuous symmetry

• If instead of the binary Ising spins we consider
  little vectors that can rotate in 3d space and
  interact via a scalar product-like coupling, we
  arrive at the Heisenberg model. This has a
  continuous (rotation) symmetry. When the system
  orders, it develops a macroscopic magnetization
  and the rotation symmetry is broken.
• We can now define two different correlation
  functions the longitudinal one corresponding to
  fluctuations parallel to the magnetization, and
  the transverse one that is perpendicular to it.

57
Goldstone modes

• The transverse correlation function can exactly
  be shown to fall off like a power all through the
  ordered phase
• ,
• Such long-ranged behaviour always appears when a
  continuous symmetry is broken.
• Complex systems are typically very inhomogeneous,
  they do not display any symmetry. There seems to
  be no reason to expect them to have long-range
  correlations.