Quantify prediction uncertainty(Book, p. 174-189)

1
Quantify prediction uncertainty(Book, p. 174-189)

• Prediction standard deviations (Book, p. 180)
• A measure of prediction uncertainty
• Calculated by translating parameter uncertainty
  through to the predictions
• Activate all parameters when calculating !!!
  
• Calculate parameter var-cov matrix with all
  parameters
• Calculate prediction sensitivities for all
  parameters

2
Quantify prediction uncertainty

• Linear confidence and prediction intervals (p.
  176-177)
• Intervals can be individual or simultaneous
• Form confidence interval prediction
  interval
• Prediction intervals account for measurement
  error. Use to compare simulated results to field
  measurements.
• a is the significance level, c(a) is the critical
  value and is different for different types of
  intervals (Table 8.1, p. 176).

3
Individual vs. Simultaneous Intervals

• Individual linear intervals
• Defined as an interval that has a specified
  probability of containing the true predicted
  value.
• Exact for correct, linear models with normally
  distributed residuals.
• The more these requirements are violated, the
  less accurate the intervals become.
• Simultaneous linear intervals
• On two or more predictions, each has a specified
  probability of containing the true value.
• Always linear intervals, because of greater
  difficulty in defining intervals that
  simultaneously include true values of two or more
  predictions. Largest intervals are for case where
  of predictions of parameters
• Common types Bonferoni Sheffé

4
Exercise 8.2a Calculate linear confidence
intervals on predicted advective transport

• Linear confidence intervals can be computed in
  UCODE_2005 using program Linear_Uncertainty.exe.
• Linear_Uncertainty uses V(b) from the regression
  run output, along with information from an extra
  ucode run with the prediction conditions (for
  computing prediction sensitivities) to calculate
  prediction standard deviations.
• Then it calculates the different types of
  individual and simultaneous intervals using the
  appropriate statistics.

5
Calculating linear intervals with
UCODE_2005. From Poeter , 2005, p. 158)
6
Linear Intervals

• Do Exercise 8.2a (p. 208-209) and the Problem,
  including answering Question 5 What is the
  uncertainty in the predictions?
• Correction to book p. 208, second line from the
  bottom, should read Answer Question 5

7
Figure 8.15a, p. 210
Linear Individual
Results of Exercise 8.2aLinear Confidence
Intervals for Question 5 What is the prediction
uncertainty?
Linear Simultaneous (Scheffe dNP)
Figure 8.15b, p. 210
8
Results of Exercise 8.2a(continued)Linear
Confidence Intervals for Question 5 What is the
prediction uncertainty?
Figure 8.16, p. 211
9
Nonlinear Intervals
Method involves finding the minimum and maximum
predicted value on a confidence region for the
parameters, which is defined as (book, p.
178) S(b) ? S(b) (s2 x crit) a
critcritical value
Maximum prediction
Minimum prediction
Developed by Vecchia and Cooley (1987, WRR) Each
limit of each interval requires a regression run
that is often more difficult than the regression
runs used for calibration.
10
Calculating nonlinear intervals with
UCODE_2005. Modified from Poeter , 2005, p. 193)
11
Nonlinear Intervals

• Do exercise 8.2b
• Computer instructions the input files are
  provided for you in initial\ex8\ucode-opr-ppr-runs
  \ex8.2b directory, as noted in the computer
  instructions.
• The nonlinear intervals are in ex8.2b._intconf

12
Figure 8.15c, p. 210
Nonlinear Individual
Results of Exercise 8.2bNonlinear Confidence
Intervals for Question 5 What is the prediction
uncertainty?Do the Problem on p. 212
Nonlinear Simultaneous (Scheffe dNP)
Figure 8.15d, p. 210
13
Linear Individual
Linear Simultaneous (Scheffe dNP)
Figure 8.15a, p. 210
Figure 8.15, p. 210
Nonlinear Simultaneous (Scheffe dNP)
Nonlinear Individual
14
Our Final Analysis and the County Decision

• Our Analysis
• Though it looks likely that the particle goes to
  the well, results are not conclusive.
• Consider using parameter values for which the
  particle goes to the river in an
  advective-dispersive model to analyze
  concentrations at the well. If concentrations
  high, results become more conclusive.
• County decision
• No additional modeling right now
• Wait for the new data and use it to recalibrate

15
Monte Carlo Analysis (Book, p. 185-189)

• Change some aspect of model input, run model,
  evaluate selected changes in model results.
• Can change parameter values, definition of
  hydrogeology, etc.
• When changing parameter values, can generate new
  sets from V(b) if model was calibrated by
  regression. For changing hydrogeology, a common
  geostatistical approach is simulation, which
  uses kriging as part of the method.
• Can just do forward simulations, or can involve
  inverse modeling as well.
• Commonly need to do numerous model runs to obtain
  enough data to make supportable conclusions.
  This is now often feasible, with the level of
  computational power in PCs.
• Results commonly displayed as histograms showing
  distribution of model output values can also
  calculate statistics from the results, such as
  means and variances.
• Suggestion only use sets of generated parameter
  values that produce a reasonable fit to the
  calibration data (Beven)

16
Can confidence intervals replace traditional
sensitivity analysis? (p. 184-185)

• Traditional sensitivity analysis
• quantify uncertainty in the calibrated model
  caused by uncertainty in the estimated parameter
  values
• change hydraulic conductivity, storage, recharge
  and boundary conditions systematically within
  previously established plausible range
• Weaknesses of traditional method
• Plausible range does not reflect significant
  information provided through model calibration.
  Results exaggerate uncertainty.
• Suggested method to account for parameter
  correlation exacerbates this exaggeration.

17
Can confidence intervals replace traditional
sensitivity analysis?

• Weaknesses of both methods
• Only consider uncertainty in the parameter
  values.
• Uncertainty in model construction generally
  neglected entirely
• Advantages of confidence intervals
• Account for information provided through the
  modeling process.