Title: Model Predictive Uncertainty
1Model Predictive Uncertainty
2Sensitivity analysis ..
3(No Transcript)
4GM Seam Inflows
5Permian Inflows
6Tertiary Sands Inflow
7Hydraulic property heterogeneity
correlation length
8Hydraulic property correlation decreases with
distance
correlation length
C(K1 , K2 )
distance
9Hydraulic property correlation decreases with
distance
correlation length
variogram
C(K1 , K2 )
distance
10Hydraulic property measurement points
11Hydraulic property correlation decreases with
distance
correlation length
variogram
C(K1 , K2 )
distance
12Hydraulic property realisation
13Hydraulic property realisation
14constrained by point measurements
15constrained by point measurements
16Particularly useful in pathline analysis ...
17...
18Sensitivity analysis calibrated model ..
19Estimated parameter values
p2
Objective function minimum
p1
20Estimated parameter values
Objective function minimum
p2
Maximum probability for p1 and p2
p1
21Estimated parameter values
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
22Estimated parameter values
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
23Estimated parameter values nonlinear case
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
24Field or laboratory measurements and model
output-
Model output
value
calibration dataset
prediction
q2
q1
q3
etc
distance or time
25Field or laboratory measurements and model
output-
Model output
value
Lower predictive limit
calibration dataset
q2
q1
q3
etc
distance or time
26Field or laboratory measurements and model
output-
Model output
value
Upper predictive limit
calibration dataset
q2
q1
q3
etc
distance or time
27Field or laboratory measurements and model
output-
Model output
value
Confidence interval for prediction
calibration dataset
q2
q1
q3
etc
distance or time
28Estimated parameter values nonlinear case
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
29Estimated parameter values nonlinear case
knowledge constraints
p2
Allowed parameter values
p1
30A certain model prediction
p2
Increasing value
p1
31Defining a confidence interval
p2
The critical points
p1
32Residuals
Model output
value
calibration dataset
prediction
q2
q1
q3
etc
distance or time
33The variance of the residuals is- ?2
? / (m - n)
m number of observations n number of
parameters
34Field or laboratory measurements and model
output-
Model output
value
Confidence interval for prediction
calibration dataset
q2
q1
q3
etc
distance or time
35Field or laboratory measurements and model
output-
Model output
Predictive uncertainty interval
value
calibration dataset
q2
q1
q3
etc
distance or time
36Software for predictive uncertainty analysis
- UCODE
- assumes model linearity
- only works with a few parameters
- PEST
- full nonlinear predictive analysis
- unlimited number of parameters
37For linear models
38Estimated parameter values-
p2
Extreme values of p1 and p2
p1
39A simple lumped parameter model
40A simple lumped parameter model
par1 par2
par5 par6
par3 par4
41The covariance matrix of the estimated parameter
set is given by C(p) ?2 (Mt QM)-1
For a nonlinear model replace M by J, the
Jacobian matrix. C(p) ?2 (Jt QJ)-1
42Let M (ie. green M) represent the action of the
model in predictive mode and o the model outputs
in predictive mode. Then C(o) M C(p)Mt
For a nonlinear model- C(o) J C(p)Jt
Notice that predictions can be correlated.
43probability
value of prediction 1
44Maximum probability
Bivariate probability density function.
45For linear and nonlinear models
46PESTs predictive analyzer
p2
Initial parameter estimates
The critical point
p1
47PESTs predictive analyzer
p2
The critical point
Initial parameter estimates
p1
48Major problem with this approach
- assumes that there is an objective function
minimum - assumes that this defines a unique set of
parameters - thus it assumes that parameters are lumped and
that there arent many of them
49objective function contours
50A Confined Aquifer
head
Fixed Inflow
T1
T3
T2
Fixed head
51objective function contours
52Hillside and Piezometers
53Transmissivity distribution - I
100 m2/day
54Transmissivity distribution - II
12 m2/day
360 m2/day
55SNOW section of the PERLND module of HSPF
56PWATER section of the PERLND module of HSPF
57PWATER section of the PERLND module of
HSPF (continued)
58Daily Flow
59Monthly Volume
60Exceedence fraction
61In cases such as these (and most real-world cases
are the same), quantification of predictive
uncertainty must take place by other means.
62The linear method of confidence interval
estimation is impossible to apply because
parameter covariance matrices are singular and
the mathematics breaks down.
63PESTs ability to maximise/minimise a prediction
while still maintaining calibration constraints
can still be applied. This is an excellent method
to test the wiggle room of a prediction.
However quantification of uncertainty limits
becomes mathematically more difficult.
64Reality
Exit time 3256 Exit point 206
65Calibration to 12 observations (no noise)
Exit time 7122 true3256 Exit point 241
true206
66Use predictive analysis to minimize travel time
K ranges from 1.9e-5 to 8813 m/day
Exit time 280 true3256 Exit point 226
true206
67Mathematically, because of parameter
nonuniqueness, the parameter and predictive
uncertainty limits are huge. Limits will then be
set by parameter plausibility and parameter
relationships plausibility, rather than simply by
model-decalibration considerations.
68Special calibration-constrained monte carlo
methods are often used (see later).
69Another major problem is this.
70Field or laboratory measurements and model
output-
Model output
Predictive uncertainty interval
value
calibration dataset
q2
q1
q3
etc
distance or time
71We know the residual variance when our
predictions are of the same type as the data
against which we calibrate. But what if we are
making a prediction of a different type or at a
different place? That is why we are using a
physically based model in the first case. The
residual noise can be enormous (remember our
travel time example from earlier lecture).
72Some examples
73A surface water modelling example
74(No Transcript)
75Observed and modelled flows over part of
calibration period.
76Observed and modelled monthly volumes
77Observed and modelled exceedence fractions
78(No Transcript)
79(No Transcript)
80(No Transcript)
81(No Transcript)
82(No Transcript)
83(No Transcript)
84(No Transcript)
85Points joined only as a method of enhancing
visual comparison of field measurements with
model outputs
86Observations used in calibration process
- Individual TSS values
- mean TSS (observed and modelled after
interpolation) - std dev of TSS (observed and modelled after
interpolation - bed sand, silt and clay unchanged over
calibration period
87Points joined only as a method of enhancing
visual comparison of field measurements with
model outputs
88Measured and model-generated TSS
time-interpolated to measurement times
89Measured and model-generated TSS
90Measured and model-generated TSS
91Aim of predictive analysis
Maximise/minimise a key model prediction while
maintaining the model in a calibrated state. In
our case we will maximise/minimise the total
sediment mass carried by the creek over the whole
calibration period of 1970 to 1995.
92Calibration limits
Objective function under calibration conditions
4.2E4 Model deemed to be uncalibrated when
objective function 4.8E4. So very tight
calibration constraints.
93Maximum/minimum sediment mass
Maximum sediment mass 1.5E6 tonnes Minimum
sediment mass 7.9E5 tonnes
94Measured and model-generated TSS
Minimum phi
95Measured and model-generated TSS
Minimum phi
Minimized prediction
96Measured and model-generated TSS
Minimum phi
Minimized prediction
97a groundwater modelling example
98A hillside
99A hillside and finite difference grid
100Observation points
101Transmissivity field- average100m2/day exponenti
al variogram X-correlation length twice
Y-correlation length Recharge
100mm/yr Concentration of Effluent 100
units Effective porosity 5 Long. dispersivity
10m Trans dispersivity 1m
102Log transmissivity distribution average
100m2/day green is higher
103Piezometric surface
Contour interval 5m
104Solute concentration after 3 years leakage
Maximum concentration 20 units
105Remediation strategy interception by pumping
106Solute concentration after 3 years leakage
Maximum concentration 20 units
Well pumps at 600m3/day
107Solute concentration after 30 years remediation
Maximum concentration 2.6 units /m3
Contaminant outflow to river is 83000 units over
30 years
Well pumps at 600m3/day
108However in real life we do not know what is in
the ground. So we must calibrate a model.
109What is the worst-case scenario ie. how
ineffective could pumping be while ensuring that
our model is still calibrated?
110Parameterisation using pilot points
Pilot point
Observation bore
111Observed and model-generated water levels at bores
Contour interval 2m
112Concentration residuals
113Solute concentration after 30 years remediation
Maximum concentration 1.6 units /m3
Contaminant outflow to river is 8.0?106 units
over 30 years
Well pumps at 600m3/day
114Log transmissivity distribution green is higher
Contaminant outflow to river is 8.0?106 units
over 30 years
Well pumps at 600m3/day
115What is the best-case scenario ie. how effective
could pumping be?
Make sure that the model is still calibrated.
116Observed and model-generated water levels at bores
Contour interval 2m
117Concentration residuals
118Solute concentration after 10 years remediation
Maximum concentration 5.9 units (minimum
contour at .01 units)
Contaminant inflow to creek is 14400 units over
30 years
Well pumps at 600m3/day
119Log transmissivity distribution green is higher
Contaminant inflow to creek is 14400 units over
30 years
Well pumps at 600m3/day
120Fail-safe Remediation
- Plot worst-case river pollution against pumping
rate - Choose minimum pumping rate that meets regulatory
requirements
121Worst-case outflow
Results from individual predictive analysis runs
122Solute concentration after 30 years of pumping
Add a second bore
123Worst-case outflow
One extraction bore
Two extraction bores
124This method of predictive uncertainty analysis
has the advantage that it is computationally
reasonably efficient. However in some cases
predictive uncertainty limits could have been
narrower if known or suspected parameter
relationships are enforced. However this is very
difficult where these relationships are
smoothed or modified by the regularisation
process which is necessary to achieve
calibration. This is an area of active research.
125Calibration-constrained Monte-Carlo ..
126GLUE Method (Lancaster University)
127Estimated parameter values nonlinear case
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
128Estimated parameter values nonlinear case
p2
Allowed parameter values based on linear
approximation
p1
129Advantages-
- does not rely on linearity assumption
- sometimes predictive probabilities can be
estimated - robust
Disadvantages-
- Only a few parameters can be examined so model
may be over-simplified - Hit to miss ratio probably extremely low
- Extremely computationally expensive
130Bayes Theorem
p(?,?yi) ? p(yi ?,?) p(?,?)
131Bayes Theorem
p(?,?yi) ? p(yi ?,?) p(?,?)
132Markov-Chain Monte Carlo
p2
Allowed parameter values based on linear
approximation
p1
133Advantages-
- two orders of magnitude more efficient than GLUE
- Provides posterior parameter probability
distribution (from which predictive distribtution
can be derived) - robust
Disadvantages-
- Only a few parameters can be examined so model
may be over-simplified - Computationally very expensive
134Warping
135A model grid
136The model domain
Fixed head
Recharge 1.0e-4 m/day
Transmissivity 100 m2/day
137Calculated heads
138Observation bore locations
139Pilot point locations
140Methodology
- Generate a random field (rather, the log of a
random field). - Multiply random field by field interpolated from
pilot points (use regularisation to make that
field as smooth as possible). - Use multiplied field in model.
- Estimate field multipliers at pilot points
through calibration process (ie.estimate field
multipliers so as to minimise head residuals
calculated using multiplied field.)
Use of PESTs regularisation mode essential
141Generated random field
Calibrated multiplier field
Field used by model
142Generated random field
Calibrated multiplier field
Field used by model
143Calibration Process
- Measurement Objective function
- comprised of differences between measured and
calculated heads at bores - maximum permitted measurement objective function
supplied by user
- Regularisation Objective function
- rises with heterogeneity of multipliers at pilot
points - weights assigned to homogeneity constraints
determined geostatistically - regularisation process maximises homogeneity of
multiplier field
144The following fields all calibrate the model with
an rms error of 0.12m (ie. 12 cm).
145Calibrated transmissivity field
Transmissivity ranges from 11 m2/day to 667 m2/day
146Calibrated transmissivity field
Transmissivity ranges from 13 m2/day to 617 m2/day
147Calibrated transmissivity field
Transmissivity ranges from 15 m2/day to 994 m2/day
148Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 815 m2/day
149Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 471 m2/day
150Calibrated transmissivity field
Transmissivity ranges from 7 m2/day to 643 m2/day
151Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 524 m2/day
152Calibrated transmissivity field
Transmissivity ranges from 13 m2/day to 572 m2/day
153Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 488 m2/day
154Calibrated transmissivity field
Transmissivity ranges from 11 m2/day to 1809
m2/day
155Calibrated transmissivity field
Transmissivity ranges from 7 m2/day to 722 m2/day
156Study area
157Model domain
158Observed heads
159Model grid
160Pilot points
161rms 2in
The heterogeneity that MUST exist.
162rms 3.5in
The heterogeneity that MUST exist.
163rms 2.9in
The heterogeneity that MAY exist.
164rms 2.9in
The heterogeneity that MAY exist.
165rms 2.9in
The heterogeneity that MAY exist.
166Generate field and warp them to enforce
calibration constraints
565
593
675
506
167perform stochastic analysis
Exit points
168evaluate probabilities.
Exit times
169Objective function contours
p2
Objective function minimum
Initial parameter estimates
p1
170Objective function contours
p2
Objective function minimum
Initial parameter estimates
p1
171Objective function contours
p2
Objective function minimum
Final parameter estimates
p1
172(No Transcript)
173Observed and modelled flows over part of
calibration period.
174Observed and modelled monthly volumes
175Observed and modelled exceedence fractions
176Parameter LZSN 2.0 UZSN
2.0 INFILT 0.0526 BASETP 0.200 AGWETP 0.00108 LZET
P 0.50 INTFW 10.0 IRC
0.677 AGWRC 0.983
177Observed and modelled flows over part of
calibration period.
178Observed and modelled monthly volumes
179Observed and modelled exceedence fractions
180Parameter Set 1 Set 2 Set 3 Set 4
Set 5 Set 6 LZSN 2.0 2.0
2.0 2.0 2.0 2.0 UZSN
2.0 1.79 2.0 2.0 1.76
2.0 INFILT 0.0526 0.0615 0.0783 0.0340
0.0678 0.0687 BASETP 0.200 0.182 0.199
0.115 0.179 0.200 AGWETP 0.00108 0.0186
0.0023 0.0124 0.0247 0.0407 LZETP 0.50
0.50 0.20 0.72 0.50
0.50 INTFW 10.0 3.076 1.00 4.48
4.78 2.73 IRC 0.677 0.571
0.729 0.738 0.759 0.320 AGWRC 0.983
0.981 0.972 0.986 0.981 0.966
181Advantages-
- simple to implement
- provides posterior parameter probability
distribution (from which predictive distribtution
can be derived) - can generate pre-warped parameter values
according to known prior probability and
correlation relationships - robust
Disadvantages-
- can computationally expensive (but getting more
efficient) - Convenient but not quite theoretically correct
182Some Conclusions
183Some conclusions
- The potential for model predictive uncertainty is
often very high especially when a model needs
to predict something which is different from the
data used in its calibration. - In most cases there is no such thing as the
model. - Not only is the model just one of many models,
but the regularisation that allows uniqueness to
exist will probably introduce bias into
predictions. - Predictive uncertainty analysis should be an
essential part of model deployment
184- In view of this, the separation between the
calibration and predictive process is artificial.
A model is never calibrated. - Calibration is simply the imposition of a set of
constraints on parameter values when making a
prediction, only use parameters which respect
what we know about the system, and which allow
the model to replicate past measurements - In most cases (especially those involving
complexity) the level of parameter nonuniqueness
is still very high - There is thus potential for high predictive
uncertainty as well.