ENTROPY

1
ENTROPY INFORMATION

• a physicist point of view

Jean V. Bellissard Georgia Institute of
Technology Institut Universitaire
de France
2
ENTROPY

• Some history

3
Carnots Principle

• Sadi CARNOT
• 1825
• Reflexions sur la Puissance Motrice du Feu

4
Carnots Principle

• Sadi CARNOT
• 1825
• Reflexions sur la Puissance Motrice du Feu
• A steam machine needs 2
• sources of heat
• a hot one temperature Th
• a cold one temperature Tc
• Th gt Tc

5
Carnots Principle

• Sadi CARNOT
• 1825
• Reflexions sur la Puissance Motrice du Feu
• The proportion of thermal
• energy that can be
• transformed into mechanical
• motion depends only on the
• temperatures of the two
• sources

6
Steam Engines

• Any steam engine has
• a heat source (burner)
• and a cold source
• (the atmosphere).

7
Thermal engines are everywhere

• in power plants (coal, nuclear, )
• in cars, airplane, boats,
• in factories,

8
Entropy definition

• Rudolf CLAUSIUS
• 1865
• Definition of entropy
• d S d Q/T
• 2nd Law of Thermodynamics
• Entropy cannot decrease
• over time

9
Gas are made of molecules

• Clausius showed that gas were made of molecules,
  explaining the slow diffusion of dust and the
  origin of viscosity

10
Statistical Thermodynamics

• Ludwig BOLTZMAN
• 1872
• Kinetic theory
• 1880
• Statistical interpretation
• of entropy
• disorder in energy space

11
Statistical Mechanics

• Josiah Willard GIBBS
• 1880s
• Thermodynamical equilibrium corresponds to
  maximum of entropy
• 1902 book
•  Statistical Mechanics 

12
Information theory

• Claude E. SHANNON
• 1948
•  A Mathematical Theory
• of Communication 
• -Information theory
• -Entropy measures the
• lack of information of a
• system

13
Second Law of Thermodynamics

• Over time, the information contained in an
  isolated system can only be
• destroyed
• Equivalently, the entropy can only
• increase

14
MORPHOGENESIS

• how does nature produces information ?

15
Conservation Laws

• In an isolated system, the Energy, the Momentum,
  the Angular Momentum, the Electric Charge,. are
  conserved.

16
Conservation Laws
Angular momentum
17
Conservation Laws

• At equilibrium, the only information available on
  the system are the values of conserved
  quantities!
• Example elementary particles are characterized
  by their mass (energy), spin (angular momentum),
  electric charge
• Electron m 9.109x10-31 kg, s 1/2, e
  1.602 x10-19 C,

18
Out of Equilibrium

• Variations in time or space force transfer of
  conserved quantities
• Transfer of Energy (Heat), Mass, Angular
  Momentum, Charges, creates current flows.

19
Out of Equilibrium

• Transfer of Energy (Heat), creates heat current
  like in flames and fires.

20
Out of Equilibrium

• Transfer of Mass, creates fluid currents like in
  rivers or streams.

21
Out of Equilibrium

• Transfer of Charges, creates electric currents.

22
Out of Equilibrium

• Transfer of Angular Momentum creates vortices
  like this hurricane seen from a satellite.

23
Out of Equilibrium

• Pattern Formation
• A shallow horizontal liquid heated from below
  exhibits instabilities and formation of rolls and
  patterns, as a consequence of fluids equations

24
Out of Equilibrium
Explosions produce interstellar clouds Collapses
produces stars The Sun, the Moon, The Planets,
and the Stars have been used as sources
of information measure of time, localization on
Earth
25
Beating the 2nd Principle

• Without variations in time and space the only
  information contained in an isolated system is
  provided by conservation laws
• Motion and heterogeneities allow Nature to create
  a large quantity of information.
• All macroscopic equations (fluids, flame,)
  describing it are given by conservation laws

26
CODING INFORMATION
the art of symbols
27
Signs

• Signs can be visual
• color, shape, design

28
Signs

• Signs can be a sound
• ring, noise, applause musical, speech

29
Signs

• Signs can be a smell

30
Signs

• Signs can be a smell

31
Signs

• Signs can be a smell
• plants can warn their neighbors with phenols

32
Signs

• Signs can be a smell
• female insects can attract males with
  pheromones

33
Writings
34
Writings

• More than 80,000 characters are used to code the
  Chinese language

35
Writings

• Ancient Egyptians used hieroglyphs to code
  sounds and words

36
Writings

• Japanese language is also using the 96 Hiragana
  character coding syllables

37
Writings

• the Phoenicians and the Greeks found the alphabet
  simpler to code elementary sounds with 23
  characters

• b g d e z h q
• i k l m n o p r
• s t u f c y w

38
Writings

• Modern numbers are coded with 10 digits created
  by Indians and transmitted to Europeans through
  the Arabs

39

• George BOOLE (1815-1864)
• used only two characters to code logical
  operations

Writings
0 1
40

• John von NEUMANN (1903-1957)
• developed the concept of programming using
  also binary system to code
• all possible information

Writings
0 1
41
Writings

• Nature uses 4 molecules

42
Writings

• Nature uses 4 molecules to code

43
Writings

• Nature uses 4 molecules to code the genetic
  heredity

44

• Proteins uses 20 amino acids to code their
  functions in the cell

Writings
molecule of Tryptophan, one of the 20 amino acids
45
Unit of information

• Following Shannon (1948) the unit is the
• bit
• A system contains N-bits of information
• if it contains 2N possible characters

46
TRANSMITTING INFORMATION
redundancy
47

• Coding theory uses redundancy to transmit binary
  bits of information

Transmitting
0 coding 1
48

• Coding theory uses redundancy to transmit binary
  bits of information

Transmitting
0 000 coding 1 111
49

• Coding theory uses redundancy to transmit binary
  bits of information

Transmitting
0 000 coding 1 111
Transmission
50

• Coding theory uses redundancy to transmit binary
  bits of information

Transmitting
0 000 coding 1 111
010 110
Transmission errors (2nd Principle)
51

• Coding theory uses redundancy to transmit binary
  bits of information

Transmitting
0 000 coding 1 111
010 110
Transmission errors (2nd Principle)
Reconstruction
52

• Coding theory uses redundancy to transmit binary
  bits of information

Transmitting
0 000 coding 1 111
010 110
000 111
Transmission errors (2nd Principle)
Reconstruction at reception (correction)
53

• Humans use also redundancy to make sure they
  receive the correct information

Transmitting
54

• Humans use also redundancy to make sure they
  receive the correct information

Transmitting
55

• Humans use also redundancy to make sure they
  receive the correct information

Transmitting
say it again !
56

• A cell is a big factory designed to duplicate the
  information contained in the DNA

Transmitting
57

• Prior to the cell fission the DNA molecule is
  unzipped

Transmitting
58

• Prior to the cell fission the DNA molecule is
  unzipped by another protein

Transmitting
59

• A cell is a big factory designed to duplicate the
  information contained in the DNA

Transmitting
60

• A cell is a big factory designed to duplicate the
  information contained in the DNA

Transmitting
mitosis
61

• A cell is a big factory designed to duplicate the
  information contained in the DNA

Transmitting
mitosis
62

• A cell is a big factory designed to duplicate the
  information contained in the DNA

Transmitting
mitosis
63

• A cell is a big factory designed to duplicate the
  information contained in the DNA

Transmitting
mitosis
64

• A cell is a big factory designed to duplicate the
  information contained in the DNA

Transmitting
mitosis
65

• A cell is a big factory designed to duplicate the
  information contained in the DNA

Transmitting
mitosis
66

• A cell is a big factory designed to duplicate the
  information contained in the DNA

Transmitting
mitosis
67
Beating the 2nd Principle

• The cell divides before the information contained
  in the DNA fades away
• In this way, cell division and DNA duplication at
  fast pace, conserve the genetic information for
  millions of years.

68
THE MAXIMUM ENTROPYPRINCIPLE REVISITED
The scary art of extrapolation
69

• A physical system of particle reaches equilibrium
  when all information but the one that must be
  conserved have vanished

Equilibrium
70

• A physical system of particle reaches equilibrium
  when all information but the one that must be
  conserved have vanished

Equilibrium
In a gas the chaotic motion produced by
collisions is responsible for the loss of
information
71

• By analogy other systems involving a large number
  of similar individuals can be treated through
  statistics and information

Equilibrium
72

• By analogy other systems involving a large number
  of similar individuals can be treated through
  statistics and information

Equilibrium
Like bureaucracy
73

• By analogy other systems involving a large number
  of similar individuals can be treated through
  statistics and information

Equilibrium
Like bureaucracy 1837 J. S. MILL in Westm. Rev.
XXVIII. 71 That vast net-work of administrative
tyrannythat system of bureaucracy, which leaves
no free agent in all France, except the man at
Paris who pulls the wires. (Oxford English
Dictionary)
74

• China (3rd century BC)
• Confucius
• France (18th century)
• USSR (1917-1990)
• European Community (1952)

Bureaucracy
The French ENA National School of Administration
75
Bureaucracy
76
Bureaucracy

• Rules Conserved
• quantities

77
Bureaucracy

• Rules Conserved
• quantities
• Individuals particles
• undiscernable

78
Bureaucracy

• Rules Conserved
• quantities
• Individuals particles
• undiscernable
• Removal Shocks
• of an individual
• Loss of information

79
Bureaucracy

• Rules Conserved
• quantities
• Individuals particles
• undiscernable
• Removal Shocks
• Loss of information
• Maximum of entropy
• No evolution

80
Bureaucracy

• A bureaucratic system is stable (its entropy is
  maximum).
• Example China empire lasted for 2000 years.
• It cannot be changed without a major source of
  instability.
• Example collapse of the USSR

81
COMPUTERS
machines and brains
82

• Alan TURING
• (1912-1954)
• 1936
• Description of a computing machine

Computers

• Computers execute logical operations
• They produce information, memorize them, treat
  them,

83

• A Turing machine is sequential operations are
  time ordered

Computers
rules
states
Left-Right
tape
84

• The von NEUMANN computer repeatedly performs the
  following cycle of events
• 1. fetch an instruction from memory.
• 2. fetch any data required by the instruction
  from memory.
• 3. execute the instruction (process the data).
• 4. store results in memory.
• 5. go back to step 1.

Computers
85

• February 14th 1946
• ENIAC
• the first computer
• Los Alamos NM

Computers
86

• Cellular automata produce patterns as in shells

Computers
rule change pattern from layer to layer
computer simulation
87

• Nature has also produced brains
• Brain does not seem to follow the von Neumann nor
  Turing schemes

Computers
88

• In brain signals are not binary but activated by
  thresholds
• The operations are not performed sequentially

Computers
89

• Brain can learn
• It can adapt itself plasticity
• Brain memory is associative it recognizes
  patterns by comparison with pre-stored ones

Computers
90
TO CONCLUDE
Entropy Information
91

• The Second Law of Thermodynamics leads to global
  loss of information
• Systems out of equilibrium produce information
  to the cost of the environment
• Information can be coded, transmitted, memorized,
  hidden, treated.
• Life is a way of producing information genetic
  code, proteins, chemical signals, pattern
  formation, neurons, brain.
• Machines can produce similar features

92
Is Nature a big computer ?
93
THE END