Title: Modelling%20of%20dynamic%20systems:%20Part%202%20Solving%20differential%20equations%20using%20computers
1Modelling of dynamic systems Part 2Solving
differential equations using computers
Robert N. Shorten Douglas Leith The Hamilton
Institute NUI Maynooth
2Contacting me?
- How can you contact me?
- email robert.shorten_at_may.ie
- My office is in the Hamilton Institute
- When can you ask me questions?
- During lectures
- Contact hours (any time)
- Books
- Booklist on the web (www.hamilton.may.ie/course/)
3Overview
- Lecture 1 Introduction
- What are ordinary differential equations?
- Why are they important?
- Why are they difficult to solve?
- Lecture 2 Solving differential equations
- Eulers method
- Lecture 3 Solving differential equations
- Runge Kutta method
- Lecture 4 Systems of differential equations,
stiff systems, and some examples
Introduction
4Mathematics!
- Show no fear - Maths is just another language!!
-
Relax, dont panic, and think about
the equation. - Pay attention to detail and pay particular
attention to Theorems - and to the proofs
5Building our first model
- Example Malthuss law of population growth
-
- Government agencies use population models to
plan. - What do you think be a good simple model for
population growth? - Malthuss law states that rate of an unperturbed
population (Y) growth is proportional to the
population present.
Introduction
6(No Transcript)
7(No Transcript)
8(No Transcript)
9Modelling
- Modelling is usually necessary for two reasons
to predict and to control. However to build
models we need to do a lot of work. - Postulate the model structure (most physical
systems can be classified as belonging to the
system classes that you have already seen) - Identify the model parameters (later)
- Validate the parameters (later)
- Solve the equations to use the model for
prediction and analysis (now)
Introduction
10Introduction How do we predict?
- To predict we need to solve our equations
- continuous time systems discrete time systems
and hybrid systems. - Sometimes, when we are very lucky, we can find an
analytic solution to these equations. - More often, an analytic solution cannot be found,
and we must use a computer to approximate the
solution to the equation. - This is usually not a problem for discrete time
systems. However, for continuous systems,
modelled using differential equations, life is
not so simple. Solving differential equations is
the subject of the next four lectures.
Introduction
11What are differential equations?
- Calculus, the science of change, was invented by
Newton to help him in his investigations of
natural phenomena. - He discovered that lots of systems evolve
according to rate equations. - Example Application of Newtons law of cooling
- Experiments show that the time rate of change of
temperature of an object and its surrounding is
proportional to the difference between the
temperature of the object and the surrounding
medium
Introduction
12What are differential equations?
- Calculus, the science of change, was invented by
Newton to help him in his investigations of
natural phenomena. - Many other types of systems can be modelled by
writing down an equation for the rate of change
of some variable - bandwidth utilisation in TCP networks
- acceleration of car
- mortgage repayments
- population increases
- chemical change of some kind
- flight of a football
- All of the above behaviour can be captured by
very simple differential equations
Introduction
13What are ODEs?
- The previous examples are examples of ordinary
differential equations. - Mathematically speaking, an ODE is an equation
relating the derivatives of an unknown function
of an dependent variable with known functions of
an independent variable. - ODEs are used to model many physical systems and
are therefore capable of predicting the behaviour
of a system for all time into the future, only
requiring an input and some activation. - Consider a net constant longtitudinal force F
acting on a body of mass M. From Newtons laws F
Ma -
-
Introduction
14What is a differential equations telling us to do?
- Notice that the velocity at time t depends on the
initial velocity. - Sometimes we are asked to find the solution of an
ODE that satisfies some initial value this is an
initial value problem. - More often we are interested in the qualitative
behaviour of the system for all initial
conditions.
Introduction
15Cant we solve them analytically?
- Many differential equations can be solved
analytically. In fact, the theory underlying
linear time-invariant systems is well-understood.
Most linear systems can be modelled using linear
differential equations of the following form - Linearity means that the solution is specified by
an equation involving only linear combinations
derivatives of the dependent variable with
respect to the independent variable. (the order
of the system is given by the power of the
highest derivative).
Introduction
16Linear time-invariant ODEs
- Consider the RL circuit depicted below. The
source is a - sinusoid of frequency w and of unit amplitude.
Introduction
17Cant we solve them analytically?
- It is convenient to express the output of a
system, modelled by a differential equation, as
the sum of two components one associated with no
input the other associated with the input to the
system. - The first component is called the natural
response of the system and is the system output
when the input is zero. - The second component is the component of the
output due to the input only and is known as the
forced response of the system.
Introduction
18The natural response
- Consider the RL circuit depicted below. The
source is a - sinusoid of frequency w and of unit amplitude.
- The natural response is obtained by setting the
input to zero.
Introduction
19The forced response
- The forced response is obtained by setting the
input to zero.
Introduction
20The combined response
- What is the combined response if the current is 2
amps when t 0, - and if w RL1?
Introduction
21The combined response
Introduction
22What do we do when analytical solutions cannot be
found?
- Mathematical analysis is not the grand thing it
is made out to be! - Maths solves only the simplest equations. As soon
as the equations become a little more
complicated, we are in trouble. - Analytical solutions to differential equations
arising in the real world can only rarely be
found. Linear time-invariant systems are well
understood, but most real systems are
time-varying, non-linear and hybrid. In such
cases we must use numerical techniques and a
computer to investigate the system under study.
Introduction
23What do we do when we cannot find a soln?
- We begin our study of numerically approximating
differential equations with - What is this equation telling us to do?
- The equation is saying the following. Take any
point in the x-t plane. If a solution to the
above equation passes through this point then the
tangent to the solution is given by f(t,y).
Introduction
24Numerical analysis A direction field
Introduction
25Introduction
26Introduction
27Introduction
28Introduction
29The direction field
- Another equation
- What is this equation telling us to do?
- The equation is saying the following. Take any
point in the x-y plane. If a solution to the
above equation passes through this point then the
tangent to the solution is given by f(x,y).
Introduction
30Numerical analysis A direction field
Introduction
31Introduction
32Introduction
33Introduction
34Introduction
35Numerical analysis Direction fields
- We have depicted graphically the directions in
which solutions should proceed. - Direction fields generate the flow associated
with differential equations. - For such a plot to make sense, the function
f(x,y) should satisfy certain properties it
should be defined in some region of the plane it
should be single valued in this region and
solutions to the equation in this region should
be differentiable.
Introduction
36Numerical analysis Word of warning
- We have been a little bit hard on mathematics.
However, we know from maths that a solution to a
differential equation need not always exist. It
need not even be unique.
- Example Does a solution to the following
equation exist? - Example Is the solution to the following
equation unique?
Introduction
37Numerical analysis Word of warning
- We have been a little bit hard on mathematics.
However, we know from maths that a solution to a
differential equation need not always exist. It
need not even be unique. - Computers dont care. Always be careful about the
output of a program. Generally, the computer will
give you an answer not necessarily the one that
you want. - Use mathematical theory to verify simulation. In
particular, verify uniqueness and existence
conditions are satisfied.
Introduction
38Existence and uniqueness
- Existence and uniqueness theorems for ordinary
differential equations. You should have done
them .What are they? - Roughly speaking if f(t,y) is continuous in some
region of the t-y plane then at least one
solution to the differential equation exists if
the partial derivative of f w.r.t y exists and is
continuous then the solution is unique. - Revise the concept of Lipschitzness and the
existence and uniqueness theorems for ODEs from
part one of the course (if it has been covered).
Introduction
39Modelling of dynamic systems Part 2Eulers
method
Robert N. Shorten Douglas Leith The Hamilton
Institute NUI Maynooth
40Overview
- We have already spoken about the poor job that
mathematics has done in studying differential
equations. - Most engineering and scientific problems involve
differential equations that cannot be solved
analytically. - We can use the direction field to get a solution
if the order of the problem is not too high, or
if there are not too many independent variables. - What do we do in other cases?
Lecture 2
41Eulers method
- There are many methods for solving ODEs
numerically. We begin with the simplest and most
intuitive method Eulers method. - Eulers method is the easiest algorithm for
numerically solving a differential equation. - Unfortunately, it is useless. Nevertheless, it is
useful to illustrate some pertinent points and
introduce numerical methods. Let us assume that
we are dealing with a differential equation that
satisfies our existence and uniqueness
conditions.
Lecture 2
42Eulers method
- Then, using Taylors theorem, one can write any
function about t0 a point as - Terms left out of the expansion are of the form
- where n is greater than 2. Hence, for h t-t0
small enough the function can be approximated
Lecture 2
43Eulers method
- In other words, for small h, function y(t) can be
approximated from knowledge of the function and
derivative at t0. - This observation suggests the following
approximation to the solution to a differential
equation. Suppose that we want to approximate the
solution to a well posed initial value problem
over the interval a,b. To construction this
solution we divide the interval into a number of
mesh points
Lecture 2
44Eulers method
- At the first mesh point we can approximate the
solution as - The beauty of Eulers method is that this
procedure can be repeated at intermediate mesh
points.
Lecture 2
45Example
- Use Eulers method to solve approximately the
initial value problem - over the interval t in the interval 0,2 with
y(t0) 1. Use a step size - of 0.2.
k t y_eul
y_act 0 0 1.0000
1.0000 1.0000 0.2000 0.8000 0.8375
2.0000 0.4000 0.6800 0.7406 3.0000
0.6000 0.6240 0.6976 4.0000
0.8000 0.6192 0.6987 5.0000 1.0000
0.6554 0.7358 6.0000 1.2000 0.7243
0.8024 7.0000 1.4000 0.8194
0.8932 8.0000 1.6000 0.9355 1.0038
9.0000 1.8000 1.0684 1.1306 10.0000
2.0000 1.2147 1.2707
Lecture 2
46k t y_eul y_act
0.0000 1.0000 2.0000 3.0000 4.0000
5.0000 6.0000 7.0000 8.0000 9.0000
10.0000 0.0000 0.1000 0.2000 0.3000
0.4000 0.5000 0.6000 0.7000 0.8000
0.9000 1.0000 1.0000 1.1000 1.2100
1.3310 1.4641 1.6105 1.7716 1.9487
2.1436 2.3579 2.5937 1.0000 1.1002
1.2013 1.3042 1.4097 1.5185 1.6312
1.7484 1.8707 1.9984 2.1321
Lecture 2
47k t y_eul y_act
0.0000 1.0000 2.0000 3.0000 4.0000
5.0000 6.0000 7.0000 8.0000 9.0000
10.0000 0.0000 0.1000 0.2000 0.3000
0.4000 0.5000 0.6000 0.7000 0.8000
0.9000 1.0000 1.0000 1.1000 1.2100
1.3310 1.4641 1.6105 1.7716 1.9487
2.1436 2.3579 2.5937 1.0000 1.1002
1.2013 1.3042 1.4097 1.5185 1.6312
1.7484 1.8707 1.9984 2.1321
Lecture 2
48k t y_eul y_act
0.0000 1.0000 2.0000 3.0000 4.0000
5.0000 6.0000 7.0000 8.0000 9.0000
10.0000 0.0000 0.1000 0.2000 0.3000
0.4000 0.5000 0.6000 0.7000 0.8000
0.9000 1.0000 1.0000 1.1000 1.2100
1.3310 1.4641 1.6105 1.7716 1.9487
2.1436 2.3579 2.5937 1.0000 1.1002
1.2013 1.3042 1.4097 1.5185 1.6312
1.7484 1.8707 1.9984 2.1321
Lecture 2
49Eulers method Geometric interpretation
- Errors accumulate over time!!!!!
50Eulers method
- What happens if the step size is decreased? What
happens if the step size is increased?
51Eulers method
- What happens if the step size is decreased? What
happens if the step size is increased?
52Eulers method Errors
- We have seen that our approximation to the
solution to the differential equation is not
exact. - Errors are introduced into the procedure as a
result of two mechanisms - Round-off
- Discretisation errors.
- Round-off errors arises from numbers being stored
in computers using finite precision. This is
usually not a concern as far more error is
introduced as a result of truncation.
53Eulers method Local truncation error
- There are two types of truncation error local
truncation errors and global truncation errors.
The local truncation error is the error after
just one step (show) - Hence
54Global discretisation error and convergence
- The global error at a certain value just what we
would ordinarily call the error the difference
between the true value and the approximation.
Roughly speaking, the global error can be
written - So halving the step size should, roughly
speaking, halve the global discretisation error.
55Eulers method Errors
56Improving Eulers method
- The modified Eulers method is based on including
more terms in the Taylor expansion. - We can approximate the first two terms in the
expansion. How do we get the higher order term?
We approximate this term as
57Improving Eulers method
- With this approximation for the second derivative
we obtain the following approximation - It can be shown that for this method
58Improving Eulers method
59Improving Eulers method
- Investigate the effect of step-size on the
accuracy of the solution. - Show for the example that the global truncation
error is approximately - What order is the local truncation error?
60Modelling of dynamic systems Part
2Runge-Kutta methods
Robert N. Shorten Douglas Leith The Hamilton
Institute NUI Maynooth
61Overview
- In the last lecture we have looked at the Euler
method. Eulers method is usually not good
enough. The main reason is that the error is
global O(h). - An improvement in the Euler method was obtained
through the modified Euler method. - Lets take a another approach to finding the
numerical solution of the differential equation.
Lecture 2
62Second order RK method
Lecture 2
63Fourth order RK method
- This results in the following algorithm
Lecture 2
64Fourth order RK method
- This results in the following algorithm
- This is the second order Runge-Kutta method. This
algorithm is an improvement on what we have
looked at already. However, it is based on simple
trapezoidal integration. There are more
sophisticated numerical integration algorithms.
One of these is Simpsons rule. Application of
Simpsons rule leads to the commonly used 4th
order Runge-Kutta method.
Lecture 2
65Fourth order RK method
- The fourth order Runge-Kutta yields
- where
Lecture 2
66Example
- Use the RK method to solve approximately the
initial value problem - over the interval t in the interval 0,1 with
y(t0) 1. Use a step size - of 0.1.
k t y_rk y_act
0 0 1.0000 1.0000
1.0000 0.2000 0.8375 0.8375 2.0000
0.4000 0.7406 0.7406 3.0000 0.6000
0.6976 0.6976 4.0000 0.8000 0.6987
0.6987 5.0000 1.0000 0.7358
0.7358 6.0000 1.2000 0.8024 0.8024
7.0000 1.4000 0.8932 0.8932 8.0000
1.6000 1.0038 1.0038 9.0000
1.8000 1.1306 1.1306 10.0000 2.0000
1.2707 1.2707
Lecture 2
67 k t y_eul
y_act 0 0 1.0000
1.0000 1.0000 0.2000 0.8000 0.8375
2.0000 0.4000 0.6800 0.7406 3.0000
0.6000 0.6240 0.6976 4.0000
0.8000 0.6192 0.6987 5.0000 1.0000
0.6554 0.7358 6.0000 1.2000 0.7243
0.8024 7.0000 1.4000 0.8194
0.8932 8.0000 1.6000 0.9355 1.0038
9.0000 1.8000 1.0684 1.1306 10.0000
2.0000 1.2147 1.2707
Lecture 2
68 k t y_eul
y_act 0 0 1.0000
1.0000 1.0000 0.2000 0.8000 0.8375
2.0000 0.4000 0.6800 0.7406 3.0000
0.6000 0.6240 0.6976 4.0000
0.8000 0.6192 0.6987 5.0000 1.0000
0.6554 0.7358 6.0000 1.2000 0.7243
0.8024 7.0000 1.4000 0.8194
0.8932 8.0000 1.6000 0.9355 1.0038
9.0000 1.8000 1.0684 1.1306 10.0000
2.0000 1.2147 1.2707
Lecture 2
69Lecture 2
70Runge-Kutta method
- What happens if the step size is decreased? What
happens if the step size is increased?
71Fourth order Runge-Kutta method
- The fourth order Runge-Kutta is very popular for
a variety of reasons. - 1) It is simple to program.
- 2) It is very accurate. The global truncation
error is O(h4). The local truncation error
is O(h5). - 3) It is very efficient.
72A few words on stability
- All of the methods that we have considered were
of the form - This is a first order difference equation and
will be unstable (when)? - If we keep the step size small enough then we can
usually ensure stability.
73Efficiency of the RK schemes
- We have said that the 4th order RK method is more
efficient than other techniques that we have
considered. - How do me measure efficiency? There are two
components to consider. - Computational effort
- Truncation errors.
- We get a rough measure of computational effort by
counting how many times the function f(t,y) is
evaluated as we make a prediction to a desired
accuracy.
74Efficiency of the RK schemes
- The 4th order Runge Kutta method is much more
efficient than the Euler method as we usually
require much less computational effort to obtain
the same accuracy. - For our running example, the error of the RK4
method with a step size of 0.1 is 9.3771e-006.
There total number of computations to obtain this
estimate is 40. To obtain a similar error with
Euler we need a step-size of 1e-5. The total
computational requirement is therefor 1e5.
75Runge-Kutta-Fehlberg
- The big problem when using any method is to
estimate the accuracy of the approximation (the
global truncation error). - Normally we do not have access to the solution so
the problem is not straight forward. - One approach to estimate the accuracy of the
solution was proposed by Fehlberg.
76Runge-Kutta-Fehlberg
- Fehlbergs basic idea was to run two RK methods
of different order. For example, using a 4th
order RK method, the error introduced by
truncation may be estimated as - If we use a 5th order RK method
- Hence an approximation to the truncation error
(small h)
77Runge-Kutta-Fehlberg
- At each time step we can estimate the truncation
error. If this is too large, we simple decrease
the step size. If it is too small, we can
increase the step size. - In fact, given a tolerance C, the new step size
can be estimated as (show)
78Multi-step methods A qualitative discussion
- All of the methods that we have considered have
been of the form - It is of course possible to develop methods where
the prediction depends on more than one previous
value of prediction. These are called multi-step
methods.
79Adams-Bashforth Moulton method
- The ABM predictor-corrector method is a
multi-step method. It is obtained by
approximating the integral in the formula - by an interpolation polynomial of third
degree. This polynomial is chosen so that is
passes through the points - This produces an ABM prediction
80Adams-Bashforth Moulton method
- Having obtained the prediction, a second
polynomial is constructed to fit the points - and the point
- This polynomial is then integrated to obtain the
final prediction at time index k1.
81Adams-Bashforth Moulton method
- The ABM formulas are
- The advantage of this technique is that the
difference between the predictor and corrector
gives an estimate of the truncation error.
82Adams-Bashforth-Moulton method
- Disadvantages of the technique include
- (1) The first 4 predictions must be obtained
using some other method. - (2)The technique results in a high order
difference equation. Care must be taken to
ensure that this equation remains stable
(zeros inside the unit circle).
83Adams-Bashforth-Moulton method
84Adams-Bashforth-Moulton method
85Examples
- Eulers method
- Modified Euler method
- 4th order Runge-Kutta method
86Modelling of dynamic systems Part 2Systems of
differential equations
Robert N. Shorten Douglas Leith The Hamilton
Institute NUI Maynooth
87Is the world first order?
- Up to now we have considered approximating the
solution to first order ordinary differential
equations using numerical methods. Our methods
work for non-linear systems and for time-varying
systems. - What about second order systems?
- What about higher order systems in general?
- What about systems of ODEs?
88A system of differential equations
- The predator-prey model of Volterra (Foxes and
Rabbits) - Let x denote the rabbit population and let y
denote the fox population. Volterra postulated
the following. In the absence of foxes we have - and in the absence of rabbits we have
89A system of differential equations
- Now let the populations of rabbits and foxes
intermingle with one another. We assume that the
number of contacts is proportional to the total
population - We now have a system of differential equations.
90The predator-prey model of Volterra
- Lets assume that all parameters are unity.
Assume that the initial condition is (2,4).
91Higher order equations
92How do we simulate complicated systems?
- Fortunately, systems of differential equations
can be treated in the same way. To illustrate the
concepts consider the initial value problem
93How do we simulate complicated systems?
- Formally speaking, a solution to the system of
differential equations is a pair of function
f(x,y,t) and g(x,y,t) that satisfy - How do we numerically approximate such solutions?
94How do we simulate complicated systems?
- A numerical solution to the system of
differential equations over some alttltb interval
is found by considering the differentials - Can you guess Eulers method for systems of
differential equations?
95Eulers method for systems of ODEs
- We approximate the differentials as
- By subdividing the time interval into M
sub-intervals we obtain the recursive formulae
96The predator-prey model of Volterra
- Lets assume that all parameters are unity.
Assume that the initial condition is (2,4).
97The predator-prey model of Volterra
- We can also plot x against y - giving us the
phase plane (more about this later).
98Runge-Kutta for systems of ODEs
- Virtually all of the ODE solvers that we have
looked at can be used to solve systems of ODEs. - The Runge-Kutta solver for a pair of ODEs is
analagous to the solver for a first order ODE.
99Runge-Kutta for systems of ODEs
100Systems of differential equations
- Virtually all ODE solvers can be extended to
solve systems of differential equations. - We have considered (briefly) systems of two
equations. Exactly the same methodology applies
to systems of more than two equations. - When comparing ODE solvers, the issues that are
relevant for first order equations are also
relevant for systems of equations. Can you think
of any issues that are special to systems of
equations?
101Higher order equations
- We have dealt with systems of equations. How do
we deal with higher order differential equations?
- The equation
- can also be written
- i.e. as a system of differential equations.
102Some more examples
- Systems of two first order equations (second
order systems) are very special. Consider again - The plane with x and y as co-ordinates is known
as the phase plane. The phase plane can be
derived from the direction field of
103The phase plane
- Consider the system
- This system is unstable.
104The phase plane
- Consider the system
- This system is stable and has a fixed point at
the origin.
105A word of warning Stiff systems
- We have already encountered stability problems
when looking at first order ODEs. - Roughly speaking, stability is maintained by
choosing a very small step size. - However, very small step-sizes lead to large
computational burdens.
106A word of warning Stiff systems
- Consider the differential equation
- We can write
- This decays very fast and therefore
107A word of warning Stiff systems
- So, given that y(t) is not that important, we
should choose the step-size to approximate - WRONG!!!!! The stability of the numerical
integration is determined by the largest
eigenvalue of the system.Always choose a
step-size to guarantee stability. - This is an example of a stiff systema number of
variables evolving at different time scales.
There are special solvers for such systems.
108Examples
- To finish this part of the course we give two
examples where numerical solutions to systems of
ODEs are required. - 1) Physics Modelling a pendulum
- A non-linear continuous time system
- 2) Computer Science Network congestion
control - A system with continuous/discrete state A
hybrid systems model (Hespanha)
109Modeling pendulums
- Pendulums appear everywhere!
110Modeling pendulums
- This is a second order differential equation.
Hence, it can be written in the form.
111Modeling pendulums
112TCP congestion control
- Mathematical modelling of the internet is a hot
topic!! Hopefully, we can use these models to
understand what is going on and avoid congestion.
Typically we model and analyse simple scenarios!
113TCP congestion control
- Roughly speaking, TCP operates as follows
- Data packets reaching a destination are
acknowledged by sending an appropriate message
to the sender. - Upon receipt of the acknowledgement, data sources
increase their send rate, thereby probing the
network for available bandwidth, until congestion
is encountered. - Network congestion is deduced through the loss of
data packets (receipt of duplicate ACKs or non
receipt of ACKs), and results in sources
reducing their send rate drastically (by half).
114TCP congestion control
- Congestion control is necessary for a number of
reasons, so that - catastrophic collapse of the network is avoided
under heavy loads - each data source receives a fair share of the
available bandwidth - the available bandwidth B is utilised in an
optimal fashion. - interactions of the network sources should not
cause destabilising network side effects such as
oscillations or instability
115TCP congestion control
- Hespanhas hybrid model of TCP traffic.
- Loss of packets caused by queues filling at the
bottleneck link. - TCP sources have two modes of operation
- Additive increase
- Multiplicative decrease
- Packet-loss detected at sources one RTT after
loss of packet.
116TCP congestion control
117TCP congestion control
118Modelling the queue not full state
- The rate at which the queue grows is easy to
determine. - While the queue is not full
119Modelling the queue full state
- When the queue is full
- One RTT later the sources are informed of
congestion
120TCP congestion control
121TCP congestion control Example (Hespanha)
122TCP congestion control Example (Fairness)