Lecture Outline:

1

• Lecture Outline
• Random variables and probability distributions
• Functions of a random variable, moments
• Multivariate probability
• Marginal and conditional probabilities and
  moments
• Multivariate normal distributions
• Application of probabilistic concepts to data
  assimilation

2
A random variable is a variable whose possible
values are distributed throughout a specified
range. The variables probability density
function (PDF) describes how these values are
distributed (i.e. it gives the probability that
the variable value falls within a particular
interval).
Continuous PDFs
A Discrete PDF
3
Probability that x falls in interval (x1, x2
Continuous PDF
Discrete PDF

Probability that y takes on some value in the
range (- ? , ?) is 1.0
That is, area under PDF must 1
4
Historical data indicate that average rainfall
intensity y during a particular storm follows an
exponential distribution
a0.1 mm -1
y (mm)
What is the probability that a given storm will
produce greater than 10mm. of rainfall if a 0.1
mm-1 ?

5
f y(y)
?

y
Note that F y (?) ? 1.0 !
6
How are these 50 monthly streamflows distributed
over range of observed values?
1

0.8
0.6
Rank data from smallest to largest value and
divide into bins (sample PDF or histogram) or
plot normalized rank (rank/50) vs. value (sample
CDF)
0.4
0.2
0
-3
-2
-1
0
1
2
Sample CDF may be fit with a standard function
(e.g. Gaussian)
2
7
The expectation of a function z g(y) of the
random variable y is defined as
Expectation is a linear operator
Note that expectation of y is not a random
variable but is a property of the PDF f y(y ).
8
Non-central Moments of y
Mean
Second moment
Integrals are replaced by sums when PDF is
discrete
9
The mean and variance of a random variable
distributed uniformly between 0 and 1 are
Variance
Standard deviation
10
Plot table as discrete joint PDF with two
independent variables y1 and y2
11
In multivariate problems interval probabilities
are replaced by the probability that the n
random variables fall in a specified region (R)
of the n-dimensional space with coordinates ( y1
, y2 , , yn ) .
Bivariate case -- Probability that the pair of
variables ( y1 , y2 ) lies in a region R in the
y1 - y2 plane is
y2
Continuous PDF (contour plot)
Region R
Discrete PDF (discrete contour plot)
y1
0.15
Region R
12
The mean of a vector of n random variables y
y1, y2 , , yn is an n vector
Second non-central moment of a vector y is an n
by n matrix, called the covariance matrix
The correlation coefficient between any two
scalar random variables (e.g. two elements of the
vector y) is
If Cyiyk ?ik 0 then yi and yi are
uncorrelated.
13
The marginal PDF of any one of a set of jointly
distributed random variables is obtained by
integrating joint density over all possible
values of the other variables. In the bivariate
case marginal density of y1 is
Continuous PDF
Discrete PDF
The conditional PDF of a random variable yi for a
given value of some other random variable yk is
defined as
The conditional density of yi given yk is a
valid probability density function (e.g. the area
under this function must 1).
14
For the discrete example described earlier the
marginal probabilities are obtained by summing
over columns to get f y1 ( y 1 ) or rows to
get f y2 ( y 2 )
Marginal densities shown in color (last row and
last column)
The conditional density of y1 (June storms) given
that y2 1 (one storm in July) is obtained by
dividing the entries in the y2 1 column by f
y2 ( y21) 0.3
15
Conditional moments are defined in the same way
as regular moments, except that the unconditional
density e.g. f y1 ( y1 ) is replaced by the
conditional density e.g. f y1y2 (y1 y121)
in the appropriate definitions.
For discrete example, unconditional mean and
variance of y1 may be computed directly from f
y1 ( y1) table
The conditional mean and variance of y1 given
that y2 1 may be computed directly from f y1y2
(y1 y121) table
Note that the conditional variance (uncertainty)
of y1 is smaller than the unconditional variance.
This reflects the decrease in uncertainty we
gain by knowing that y121.
16
Two random vectors y and z are independent if any
of the following equivalent expressions holds
Independent variables are also uncorrelated,
although the converse may not be true.
In the discrete example described above, the two
random variables y and y are not independent
because
For example, for the combination (y1 0, y2 0
) we have
17
A function z g(y) of a random variable is also
a random variable, with its own PDF f z(z).
The basic concept also applies to multivariate
problems, where y and z are random vectors and z
g (y) is a vector transformation.
18
The PDF f z(z) of the random variable z g(y)
may be sometimes be derived in closed form from
g(y) and f z(z). When this is not possible Monte
Carlo (stochastic simulation) methods may be
used. If y and z are scalars and z g(y) has a
unique solution y g -1(z) for all permissible
y, then
where
If z g(y) has multiple solutions the
right-hand side term is replaced by a sum of
terms evaluated at the different solutions.
This result extends to vectors of random
variables and a vector transformation z g(y) if
the derivative g is replaced by the Jacobian of
g(y).
19
The definition of the conditional PDF may be
applied twice to obtain Bayes Theorem, which is
very important in data assimilation. To
illustrate, suppose that we seek the PDF of a
state vector y given that a measurement vector
has the value z. This conditional PDF may be
computed as follows.
This expression is useful because it may be
easier to determine f zy( zy) and then compute
f yz( yz) from Bayes Theorem than to derive f
yz( yz) directly. For example, suppose that
Then if y is given (not random) f z y(z y)
f ? (z - y). If the unconditional PDFs f ? (?)
and f y(y) are specified they can be substituted
into Bayes Theorem to give the desired PDF f
yz( yz). The specified PDFs can be viewed as
prior information about the uncertain measurement
error and state.
20
The only widely used continuous joint PDF is the
multivariate normal (or Gaussian)
Multivariate normal PDF of the n vector y y1,
y2 , , yn is completely determined by mean
and covariance C yy of y
Where C yy represents determinant of C yy
and C yy-1 represents inverse of C yy .
f y1 y2 ( y1 , y2 )
Bivariate normal PDF .
Mean of normal PDF is at peak value. Contours of
equal PDF form ellipses.
y1
y2
21
The following properties of multivariate normal
random variables are frequently used in data
assimilation

• A linear combination z a1 y1a2 y2 an yn a
  T y of jointly normal random variables y y1 ,
  y2 , , ynT is also a normal random
  variable. The mean and variance of z are

• If y and z are multivariate normal random vectors
  with a joint PDF fyz(y, z) the marginal PDFs fy
  (y) and fz(z) and the conditional PDFs f y z
  (y z) and f z y (z y) are also multivariate
  normal.

• Linear combinations of independent random
  variables become normally distributed as the
  number of variables approaches infinity (this is
  the Central Limit Theorem)

In practice, many other functions of multiple
independent random variables also have nearly
normal PDFs, even when the number of variables is
relatively small (e.g. 10-100). For this reason
environmental variables are often observed to be
normally distributed.
22
Consider two vectors of random variables which
are all jointly normal y y1, y2 , , yn
(e.g. a vector of n states) z z1, z2 , , zm
(e.g. a vector of m measurements)
The conditional PDF of y given z is
Where
The conditional covariance is smaller than the
unconditional y covariance (since the difference
matrix Cy y - Cyy z is positive definite).
This decrease in uncertainty about y reflects the
additional information provided by z
23

• Data assimilation seeks to characterize the true
  but unknown state of an environmental system.
  Physically-based models help to define a
  reasonable range of possible states but
  uncertainties remain because the model structure
  may be incorrect and the models inputs may be
  imperfect. These uncertainties can be accounted
  for in an approximate way if we assume that the
  models inputs and states are random vectors.

• Suppose we use a model and a postulated
  unconditional PDF f u ( u) for the input u to
  derive an unconditional PDF f y ( y ) for the
  state y . f y ( y ) characterizes our knowledge
  of the state before we include any measurements.
• Now suppose that we want to include information
  contained in the measurement vector z . This
  measurement is also a random vector because it
  depends on the random state y and the random
  measurement error ? . The measurement PDF is f
  z ( z ).

• Our knowledge of the state after we include
  measurements is characterized by the conditional
  PDF f yz (y z). This density can be derived
  from Bayes Theorem. When y and z are
  multivariate normal f yz (y z) can be readily
  obtained from the multivariate normal expressions
  presented earlier. In other cases approximations
  must be made.

• The estimates (or analyses) provided by most data
  assimilation methods are based in some way on the
  conditional density f yz (y z) .