John Doyle

1

• John Doyle
• Control and Dynamical Systems
• Caltech

2
Perspective

• Mathematicians and systems engineers (and systems
  biologists) have much in common, but have
  viewpoints that tend to be surprisingly different
  from
• Physicists and device engineers, who also have
  much in common.
• This work is the result of interactions with
  physicists interested in complex systems
  (Carlson, UCSB and Mabuchi, Caltech)

3
Outline

• Complexity and robustness
• Motivation from biology and engineering
• Connections with foundations of physics
• Power laws (Carlson)
• Shear flow turbulence (Bamieh, Bobba)
• Thermodynamic arrow of time, dissipation
• Quantum/classical transition, quantum measurement

4
Engineering Trends

• 21st Century
• Heterogeneous, diverse, and versatile physical
  substrates
• Convergence and integration of computation,
  communication, control in
• Complex, ubiquitous networks of networks
• Biology
• Global economic and environmental systems
• Applications demand more integration
• Promising new mathematics

• 20th Century
• Explosion of information technology
• Devices and fabrication Moores law and more
• Two key abstractions
• Separate computation, communication, control
  (theory and application)
• Abstract systems from physical substrate

5
Biochemical Network E. Coli Metabolism
Regulatory Interactions
From Adam Arkin
from EcoCYC by Peter Karp
6
Biochemical Network E. Coli Metabolism

• Constraints
• Mass balance
• Energy balance
• Entropy

from EcoCYC by Peter Karp
7
(No Transcript)
8
500Kv
350Kv
250Kv
9

• Constraints
• Mass balance
• Energy balance
• Entropy

10
Biochemical Network E. Coli Metabolism
Regulatory Interactions
Constraints?
Robustness
Distributed Asynchronous
from EcoCYC by Peter Karp
11
During flight test, a partial system state is
saved at the rate of 1e8 bits (100 Mbits) per
second.
The human genome can be stored with 1e10 bits (lt
2 CDs).
12
Themes

• The source of complexity is robustness tradeoffs
• The robustness challenge is cascading failure
  events
• The solution is to build barriers (in state
  space) to stop cascading events
• Tradeoffs lead to robust, yet fragile systems
• These themes are universal and ubiquitous
• Central to engineering and biological networks
• Illustrate with problems familiar in physics
  power laws and turbulence

13
Robustness and watches

• Early technology
• precise time
• precise components
• Modern digital watches
• more precise time, much greater functionality
• much cheaper, less precise components
• many more degrees of freedom

14
Uncertainty (environment, users)

• Full system is
• robust,
• far from equilibrium,
• nonlinear.

Robust Mesoscale
Uncertainty (CMOS) (transistors, capacitors,
Resistors)
15
If reconfigured at the CMOS level without careful
design
Robust Mesoscale

• You can get complex-emergent-edge-of-chaos-spatio-
  temporal-far-from-equilibrium-etc-etc
• You wont get a watch.
• Robustness depends on extremely specialized,
  structured designs.

16
My First Clock (ages 5)
17
(No Transcript)
18
Cascading events in car crashes
Normal
Danger
Crash
Contact w/car
Trauma
Barriers in state space
19
Normal
Danger
Crash
Contact w/car
Trauma
Normal
Sense/ Deploy
Contact w/bag
Trauma
20
Full state space
Desired
Worse
Bad
21
Full state space
Robust
Yet Fragile
22
Robust, yet fragile

• Robust to uncertainties
• that are common,
• the system was designed for, or
• has evolved to handle,
• yet fragile otherwise
• This is the most important feature of complex
  systems (the essence of HOT).

23
Humans supply most feedback control
Normal
Danger
Crash
Contact w/car
Trauma
Lanes Laws Lights Ramps
Collision avoidance Anti-lock brakes
Heavy metal Seat belts Airbags
Helmets
24
Fully automated systems?
Normal
Danger
Crash
Contact w/car
Trauma
Lanes Laws Lights Ramps
Collision avoidance

• Internally unimaginably more complex.
• Superficially much simpler?

25
Uncertainty
Basic functionality
Sensors
Robustness
26
Uncertainty
Sensors
Actuators
Actuators
Basic functionality
Sensors
Complexity is dominated by Robustness (through
regulatory feedback networks)
27
Uncertainty
Sensors
Actuators
Actuators
Basic functionality
Sensors
But scientific research has ignored almost all
real complexity.
28
Uncertainty
Sensors
Actuators
ic functiona ces, compone
materials
Actuators
Basic functionality
Sensors
But scientific research has ignored almost all
real complexity.
29
Uncertainty
Sensors
Actuators
Basic functionality Devices, components, material
s
Ators
Actuators
Basic functionality
Sensors
Sensors
But scientific research has ignored almost all
real complexity.
30
Towards a more balanced view
Uncertainty
Sensors
Actuators
Actuators
Basic functionality
Sensors
Control networks

• What happens when we get some balance?
• New answers, completely foreign to physics
• But very resonant with engineering, biology, and
  mathematics.
• New math leading to an integrated theory of
  complexity.
• Surprise has deep implications for physics after
  all!?!?!

31
Control, communications, computing
Uncertainty
Sensors
Actuators
Actuators
Basic functionality
Sensors
Control networks

• Sense data
• Communications
• Information Focus on what is surprising in data
• Reliably store or transmit information
• Control
• Extract what is useful (not merely surprising)
• Compute decisions from useful information
• Take appropriate action

32
Theoretical foundations

• Control theory feedback, optimization, games
• Information theory source and channel coding
• Computational complexity decidability,
  P-NP-coNP-
• Dynamical systems dynamics, bifurcation, chaos
• Statistical physics phase transitions, critical
  phenomena, multiscale physics
• These are largely fragmented within isolated
  technical disciplines.
• Unified theory would be both intellectually
  satisfying and of enormous practical value.

33
Highly Optimized Tolerance (HOT)

• Complex systems in biology, ecology, technology,
  sociology, economics,
• are driven by design or evolution to
  high-performance states which are also tolerant
  to uncertainty in the environment and components.
• This leads to specialized, modular, hierarchical
  structures, often with enormous hidden
  complexity,
• with new sensitivities to unknown or neglected
  perturbations and design flaws.
• Robust, yet fragile!

34
From molecules to organisms to ecosystems

• Extraordinary robustness to uncertainty in
  environment and components, yet
• Catastrophically disabled from tiny perturbations
• Eg. from single base pair mutations, to
  introduction of a single exotic specie,
• Robust, yet fragile

35
Boeing 777

• Robust to large scale atmospheric disturbances,
  variations in cargo loads and fuels, turbulent
  boundary layers, inhomogeneities and aging of
  materials, etc
• ...but could be catastrophically disabled by
  microscopic alterations in a handful of
  components (eg. 4 carefully chosen transistors).
• This is, fortunately, very unlikely.

36
Complex systems challenges

• Biological networks
• Web/Internet and convergent, ubiquitous
  networking
• Power and transportation systems
• Turbulence in shear flows
• Financial and economic systems
• Natural and man-made disasters
• Ecosystems and global change
• Quantum networks and computation
• Integrated networks of networks

37
Collaboratorsand contributors(partial list)

• Biology Csete, Simon, Arkin,Yi, Borisuk,
  Bolouri, Kitano,
• HOT Carlson, Zhou
• Theory Lall, Parrilo, Paganini, Barahona,
  DAndrea,
• Physics Mabuchi, Doherty, Marsden,
  Asimakapoulos,
• Web/Internet Low, Effros, Zhu,Yu, Chandy,
  Willinger,
• Turbulence Bamieh, Dahleh, Gharib, Marsden,
  Bobba,
• Engineering CAD Ortiz, Murray, Schroder,
  Burdick, Barr,
• Disturbance ecology Moritz, Carlson, Robert,
• Power systems Verghese, Lesieutre,
• Finance Primbs, Yamada, Giannelli,
• and casts of thousands

38
Themes

• Complexity ? Robustness
• Robustness ? barriers to cascading events
• Tradeoffs lead to robust, yet fragile systems

39
Criticality and power laws

• The orthodox view
• Tuning 1 or 2 parameters ? critical point
• In certain model systems (percolation, Ising, )
  power laws and universality iff criticality.
• Physics power laws are suggestive of
  criticality, but the tuning is an annoyance (does
  Nature tune parameters?)
• Solution self-organized criticality (SOC)

40
Criticality and power laws

• Mathematicians and engineers would tend to
• Associate power laws with tuning
• View criticality as an extreme special case
• What happens when more parameters are tuned?
• What happens in highly optimized systems?
• Both points of view (criticality vs. tuning) are
  natural from their respective viewpoints.
• Which perspective has greater explanatory power
  for power laws in natural and man-made systems?

41
Web/internet traffic
web traffic
Is streamed out on the net.
Web client
Creating internet traffic
Web servers
42
Network protocols.
Files
HTTP
TCP
IP
packets
packets
packets
packets
packets
packets
Routers
43
web traffic
Lets look at some web traffic
Is streamed out on the net.
Web client
Creating internet traffic
Web servers
44
6
5
4
Cumulative
3
Frequency
WWW files Mbytes (Crovella)
2
(Rank)
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
2
Decimated data Log (base 10)
Size of events
45
6
Data compression (Huffman)
WWW files Mbytes (Crovella)
5
4
Cumulative
3
Frequency
Forest fires 1000 km2 (Malamud)
2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
2
Decimated data Log (base 10)
Size of events
(codewords, files, fires)
46
Size of events x vs. frequency
log(Prob gt size)
log(size)
47
1e3 samples
log10(P)
x integer
log10(x)
48
log10(P)
log10(x)
49
x
50
log10(p)
log10(x)
51
log10(p)
Slope -(?1)
log10(x)
52
?1
log10(P)
?0
log10(x)
1e3 samples
53
Log(freq.) cumulative
Fat tails
Log(event sizes)
54
Examples of fat tail distributions

• Power outages, forest fires
• Air traffic cascading congestion events
• Meteor impacts, earthquakes
• Deaths and dollars lost due to man-made disasters
• Deaths and dollars lost due to natural disasters
• Ecosystem and specie extinction events?
• Variations in stock prices and federal budgets
• All of these involve frequencies of events

55
Examples of fat tail distributions

• Web files, UNIX files, CPU utilization
• Word rank (Zipfs law)
• Species per genera, populations of cities
• Income and wealth of individuals and companies
• Masses or sizes of objects in this room
• Paper citation and actor collaboration networks
• Publications per author, patents per inventor
• Documents in libraries

56
The mystery of power laws?

• If the world is microscopically exponential,
• And the central limit theorem yields Gaussians,
• Why are there so many power laws?
• Engineers, mathematicians, and physicists
  naturally have very different reactions to this.

57
Statistics of fat tail distributions

• The central limit theorem produces Gaussians and
  power laws (though rarely taught this way).
• Power laws have additional statistical features
  that make them even more likely to arise as
  laws.
• This has nothing to do with any specific model or
  systems but is purely statistical.
• Thus models that produce power laws are no more
  explanatory a priori than models that produce
  Gaussians.
• We must go further and explain how the
  distributions change as, say, conditions change.

58
3
10
2
10
Frequency of outages gt N
1
10
US Power outages 1984-1997
0
10
4
5
6
7
10
10
10
10
N of customers affected by outage
59
Models of fat tail distributions

• There are general probability/statistics results
  that suggest there are purely mathematical
  reasons for the ubiquity of Gaussian, Poisson,
  and fat tail distributions
• Success-breeds-success is most widely studied
  model (at least since Simon, 1955) and robustly
  produces power laws (Information science
  literature).
• Criticality and SOC are promoted by some
  advocates as a general model (Bak, physics
  literature), and specifically as relevant to
  internet traffic and to forest fires.
• There are many other possibilities, but lets
  compare SOC with HOT in these contexts.

60
6
Data compression (Huffman)
WWW files Mbytes (Crovella)
5
Cumulative
4
3
Frequency
Forest fires 1000 km2 (Malamud)
2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
2
Decimated data Log (base 10)
Size of events
(codewords, files, fires)
61
Heavy tails in networks?
Heavy tails are everywhere in networks.
There is a large literature since 1994 Leland,
Taqqu, Willinger, Wilson Paxson, Floyd Crovella,
Bestavros Harchol-Balter,
Well review some of this literature.
62
Typical web traffic
Heavy tailed web traffic
? gt 1.0
log(freq gt size)
p ? s-?
Is streamed out on the net.
log(file size)
Creating fractal Gaussian internet traffic
Web servers
63
Fat tail web traffic
time
creating long-range correlations with
Is streamed onto the Internet
64
Consequences of fat-tail web traffic

• Most web file transfers are small, but
• Most packets are in very large files!
• With current protocols (TCP Reno with drop
  tails), during congestion
• Small file packets are queued behind large
• Unnecessary delays
• Exactly the opposite of what you want
• Promising alternatives
• Start with web layout

65
Issues in web layout design

• Logical and aesthetic structure determines rough
  graph topology
• Navigability, manageability, and download times
  drive geometry of links and files
• Navigability and manageability
• Low diameter
• Low out-degree
• Download time small files
• These objectives are in conflict

66
A toy website model( 1-d grid HOT design)
document
67
links files
68
Source coding for data compression
69

• 2 key abstractions in Shannon formulation
• Ignore value of information, consider only
  surprise
• Compress stochastic ensembles of source words
  rather than actual files

70
Control, communications, computing
Uncertainty
Sensors
Actuators
Actuators
Basic functionality
Sensors
Control networks

• Sense data
• Communications
• Information Focus on what is surprising in data
• Reliably store or transmit information
• Control
• Extract what is useful (not merely surprising)
• Compute decisions from useful information
• Take appropriate action

71
Control, communications, computing
Uncertainty
Sensors
Actuators
Actuators
Basic functionality
Sensors
Control networks

• Sense data
• Communications
• Information Focus on what is surprising in data
• Reliably store or transmit information
• Control
• Extract what is useful (not merely surprising)
• Compute decisions from useful information
• Take appropriate action

72
Communications theory

• Robust storage and transmission of information
• Data compression, rate distortion theory
• Error-correcting codes
• Temporal, frequency, and spatial multiplexing and
  coding
• All can be interpreted as building barriers in
  abstract state spaces
• Recall simple data compression

73
Shannon source coding
Minimize expected length
Krafts inequality
74
Minimize
Leads to optimal solutions for codeword lengths.
With optimal cost
Equivalent to optimal barriers on a discrete
tree (zero dimensional).
75

• Compressed files look like white noise.
• Compression improves robustness to limitations
  in resources of bandwidth and memory.
• Compression makes everything else much more
  fragile
• Loss or errors in compressed file
• Statistics of source file
• Information theory also addresses these issues at
  the expense of (much) greater complexity

76
Generalized coding for web layout

• Based on probability of hit,
• choose file sizes (locations of cuts)
• to minimize average download time
• (assumed proportional to file size)

links files
77
6
5
Cumulative
4
3
Frequency
2
Forest fires 1000 km2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
2
Size of events
78
Forest fires dynamics
Intensity Frequency Extent
79
A severe abstraction
Fire suppression mechanisms must stop a 1-d front.
80
Generalized coding problems
Data compression
Optimizing d-1 dimensional cuts in d dimensional
spaces.
Web
81
PLR optimization
Minimize expected loss
82
d-dimensional
li volume enclosed ri barrier density
pi Probability of event
Resource/loss relationship
83
PLR optimization
? 0 data compression ? 1 web layout ?
2 forest fires
? dimension
84
PLR optimization
? 0 data compression
? 0 is Shannon source coding
85
Minimize average cost using standard Lagrange
multipliers
Leads to optimal solutions for resource
allocations and the relationship between the
event probabilities and sizes.
With optimal cost
86
Minimize average cost using standard Lagrange
multipliers
Leads to optimal solutions for resource
allocations and the relationship between the
event probabilities and sizes.
With optimal cost
87
To compare with data.
88
To compare with data.
89
(No Transcript)
90
Data
6
DC
5
WWW
4
3
FF
2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
2
91
Data Model
6
DC
5
WWW
4
3
FF
2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
2
92
(No Transcript)
93
What can we learn from this simple model?

• P uncertain events with probabilities pi
• R limited resources ri to minimize
• L loss li due to event i

• Be cautious about simple theories that ignore
  design.
• Power laws arise easily in designed systems due
  to resource vs. loss tradeoffs.
• Exploiting assumptions, makes you sensitive to
  them.
• More robustness leads to sensitivities elsewhere.
• Robust, yet fragile.

94
More complete website models(Zhu, Yu, Effros)

• For trees, ??1
• Additional hyperlinks increases ?
• Random graphs have ??2
• Newer data from web has ??1.4-1.6
• Heavy tail traffic appears to be a permanent
  feature of any media
• Suggests alternative control strategies (TCP)

95
Forest fires?
Fire suppression mechanisms must stop a 1-d front.
96
Forest fires?
Geography could make ? lt2.
97
California geographyfurther irresponsible
speculation

• Rugged terrain, mountains, deserts
• Fractal dimension ? ? 1?
• Dry Santa Ana winds drive large (? 1-d) fires

98
Data Model
6
5
California brushfires
4
3
FF (national) ? 2
2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
2
99
18 Sep 1998
Forest Fires An Example of Self-Organized
Critical Behavior Bruce D. Malamud, Gleb Morein,
Donald L. Turcotte
4 data sets
100
HOT FF ? 2
2
10
1
10
0
10
-2
-1
0
1
2
3
4
10
10
10
10
10
10
10
Additional 3 data sets
101
SOC
HOT
102
SOC
HOT
103
California brushfires
? 1
104
SOC vs. HOT fires

• Critical fires fractal regions of
  system-spanning extent but infinitesimal area
• HOT fires compact, bounded regions of possibly
  large area
• These are very large differences
• Real fires have statistics and appearance that
  are strikingly like toy HOT models predict
• This is not the end, but just the beginning

105
Forest fires dynamics
Intensity Frequency Extent
106
Santa Monica Mountains
107
SAMO Fire History
108
SAMO Fires Over Time
109
SAMO Fires Over Time
110
SAMO Fires Over Time
111
SAMO Fires Over Time
112
SAMO Fires Over Time
113
SAMO Fires Over Time
114
SAMO Fires Over Time
115
SAMO Fires Over Time
116
SAMO Fires Over Time
117
SAMO Fires Over Time
118
SAMO Fires Over Time
119
SAMO Fires Over Time
120
SAMO Fires Over Time
121
SAMO Fires Over Time
122
SAMO Fires Over Time
123
SAMO Fire History
124
Robust
Log(freq.) cumulative
yet fragile
Log(event sizes)
125
Power laws are inevitable.
Gaussian
log(probgtsize)
log(size)
126
Power laws summary

• Power laws are ubiquitous
• HOT may be a unifying perspective for many
• Criticality, SOC is interesting, but very rare in
  the lab, and even much rarer still outside it
• Viewing a complex system as HOT is just the
  beginning of study

127
Universal network behavior?
Congestion induced phase transition.
throughput

• Similar for
• Power grid?
• Freeway traffic?
• Gene regulation?
• Ecosystems?
• Finance?

demand
128
Web/Internet?
129
Networks

• Making a random network
• Remove protocols
• No IP routing
• No TCP congestion control
• Broadcast everything
• ? Many orders of magnitude slower

log(thru-put)
log(demand)
130
Networks
HOT
log(thru-put)
log(demand)
131
Turbulence
flow
HOT
pressure drop
132
streamlined pipes
flow
HOT
HOT turbulence? Robust, yet fragile?
random pipes
pressure drop

• Through streamlined design
• High throughput
• Robust to bifurcation transition (Reynolds
  number)
• Yet fragile to small perturbations
• Which are irrelevant for more generic flows

133
Shear flow turbulence summary

• Shear flows are ubiquitous and important
• HOT may be a unifying perspective
• Chaos is interesting, but may not be very
  important for many important flows
• Viewing a turbulent or transitioning flow as HOT
  is just the beginning of study

134
The yield/density curve predicted using random
ensembles is way off.

• Similar for
• Power grid
• Freeway traffic
• Gene regulation
• Ecosystems
• Finance?

135
Turbulence in shear flows
Kumar Bobba, Bassam Bamieh
wings
channels
Turbulence is the graveyard of theories. Hans
Liepmann Caltech
pipes
136
Chaos and turbulence

• The orthodox view
• Adjusting 1 parameter (Reynolds number) leads to
  a bifurcation cascade to chaos
• Turbulence transition is a bifurcation
• Turbulent flows are chaotic, intrinsically
  nonlinear
• There are certainly many situations where this
  view is useful.

137
velocity
high
low
equilibrium
periodic
chaotic
138
random pipe
139
bifurcation
laminar
flow (average speed)
turbulent
pressure (drop)
140
Random pipes are like bluff bodies.
141
flow
Typical flow
pressure
142
wings
Streamline
channels
pipes
143
theory
laminar
log(flow)
experiment
turbulent
Random pipe
log(pressure)
144
log(flow)
Random pipe
log(Re)
145
This transition is extremely delicate (and
controversial).
Random pipe
It can be promoted (or delayed!) with tiny
perturbations.
log(Re)
146
Transition to turbulence is promoted (occurs at
lower speeds) by
Surface roughness Inlet distortions Vibrations The
rmodynamic fluctuations? Non-Newtonian effects?
147
None of which makes much difference for random
pipes.
Random pipe
148
Shark skin delays transition to turbulence
149
log(flow)
It can be reduced with small amounts of polymers.
log(pressure)
150
streamlined pipes
flow
HOT
HOT turbulence? Robust, yet fragile?
random pipes
pressure drop

• Through streamlined design
• High throughput
• Robust to bifurcation transition (Reynolds
  number)
• Yet fragile to small perturbations
• Which are irrelevant for more generic flows

151
Macro Flow Properties
Lift and drag Mixing Flow induced
vibrations Control of transition
Robust Mesoscale
Robust, yet fragile
Micro Flow Perturbations
Surface characteristics Inlet distortions
Vibrations Fluid composition Thermodynamic
fluctuations? Non-Newtonian effects?
152
(No Transcript)
153
streamwise
Couette flow
154
high-speed region
From Kline
155
Streamwise constant perturbation
Spanwise periodic
156
Streamwise constant perturbation
Spanwise periodic
157
y
flow
position
z
x
158
y
v
flow
flow
x
u
position
velocity
z
w
159
y
v
flow
flow
x
u
position
velocity
z
w
160
(No Transcript)
161
y
flow
x
position
z
2 dimensions
2d-3c model
162
These equations are globally stable!
2d-3c model
163
Linearize around
2d-3c model
164
energy
(Bamieh and Dahleh)
t
165
energyN10R1000t1000alpha2
5
10
Total energy
0
10
energy
vortices
-5
10
-10
10
0
200
400
600
800
1000
t
166
What youll see next.
Log-log plot of time response.
167
Random initial conditions on
concentrated at lower boundary.
168
Streamwise streaks.
Long range correlation.
169
streamlined pipes
flow
HOT
HOT turbulence? Robust, yet fragile?
random pipes
pressure drop

• Through streamlined design
• High throughput
• Robust to bifurcation transition (Reynolds
  number)
• Yet fragile to small perturbations
• Which are irrelevant for more generic flows

170
Plans

• Detailed experimental verification
• Model reduction
• Nonlinear analysis
• 2d/3c to 3d/3c
• Integration with variational methods (Marsden et
  al)

171
Complexity, chaos and criticality

• The orthodox view
• Power laws suggest criticality
• Turbulence is chaos
• HOT view
• Robust design often leads to power laws
• Just one symptom of robust, yet fragile
• Shear flow turbulence is noise amplification
• Other orthodoxies
• Dissipation, time irreversibility, ergodicity and
  mixing
• Quantum to classical transitions
• Quantum measurement and decoherence

172
Epilogue

• HOT may make little difference for explaining
  much of traditional physics lab experiments,
• So if youre happy with orthodox treatments of
  power laws, turbulence, dissipation, quantum
  measurement, etc then you can ignore HOT.
• Otherwise, the differences between the orthodox
  and HOT views are large and profound,
  particularly for
• Forward or reverse (eg biology) engineering
  complex, highly designed or evolved systems,
• But perhaps also, surprisingly, for some
  foundational problems in physics