Uncertainty Analysis Using GEM-SA

1
Uncertainty Analysis Using GEM-SA
2
Outline

• Setting up the project
• Running a simple analysis
• Exercise
• More complex analyses

3
Setting up the project
4
Create a new project

• Select Project -gt New, or click toolbar icon

• Project dialog appears
• Well specify the data files first

5
Files

• Using Browse buttons, select input and output
  files

• The Inputs file contains one column for each
  parameter and one row for each model training run
  (the design)
• The Outputs file contains the outputs from
  those runs (one column, in this example)

6
Our example

• Well use the example model1 in the GEM-SA DEMO
  DATA directory
• This example is based on a vegetation model with
  7 inputs
• RESAEREO, DEFLECT, FACTOR, MO, COVER, TREEHT, LAI
• The model has 16 outputs, but for the present we
  will consider output 4
• June monthly GPP

7
Number of inputs

• Click on Options tab
• Select number of inputs using
• Or click From Inputs File

8
Define input names

• Click on Names
• The Input parameter names dialog opens
• Enter parameter names
• Click OK

9
Complete the project

• We will leave all other settings at their default
  values for now
• Click OK
• The Input Parameter Ranges window appears

10
Close and save project

• Click Defaults from input ranges button
• Click OK

• Select Project -gt Save
• Or click toolbar icon
• Choose a name and click Save

11
Running a simple analysis
12
Build the emulator

• Click to build the emulator
• A lot of things now start to happen!
• The log window at the bottom starts to record
  various bits of information
• A little window appears showing progress of
  minimisation of the roughness parameter
  estimation criterion
• The Main Effects tab is selected, in which
  several graphs are drawn
• Progress bar at the bottom

13
Focus on the log window

• The Main Effects and Sensitivity Analysis
  tabs are concerned with SA, and will be
  considered in the next session
• We are interested just now simply in Uncertainty
  Analysis (UA)
• The Output Summary tab contains all we need and
  more
• But the key things can be seen more simply in the
  log window at the bottom
• Diagnostics of the emulator build
• The basic uncertainty analysis results

14
Emulation diagnostics

• Note where the log window reports
• The first line says roughness parameters have
  been estimated by the simplest method
• The values of these indicate how non-linear the
  effect of each input parameter is
• Note the high value for input 4 (MO)

Estimating emulator parameters by maximising
probability distribution... maximised posterior
for emulator parameters precision
sigma-squared 0.342826, roughness 0.217456
0.0699709 0.191557 16.9933 0.599439 0.459675
1.01559
15
Uncertainty analysis mean

• Below this, the log reports
• So the best estimate of the output (June GPP) is
  24.1 (mol C/m2)
• This is averaged over the uncertainty in the 7
  inputs
• Better than just fixing inputs at best estimates
• There is an emulation standard error of 0.062 in
  this figure

Estimate of mean output is 24.145, with variance
0.00388252
16
Uncertainty analysis variance

• The final line of the log is
• This shows the uncertainty in the model output
  that is induced by input uncertainties
• The variance is 73.9
• Equal to a standard deviation of 8.6
• So although the best estimate of the output is
  24.1, the uncertainty in inputs means it could
  easily be as low as 16 or as high as 33

Estimate of total output variance 73.9033
17
Exercise
18
A small change

• Run the same model with Output 11 instead of
  Output 4
• Calculate the coefficient of variation (CV) for
  this output
• NB the CV is defined as the standard deviation
  divided by the mean

19
More complex analyses
20
Input distributions

• Default is to assume the uncertainty in each
  input is represented by a uniform distribution
• Range determined by the range of values found in
  the input file or separately input

• A normal (gaussian) distribution is generally a
  more realistic representation of uncertainty
• Range unbounded
• More probability in the middle

21
Changing input distributions

• Reopen Project dialog by Project -gt Edit or
  clicking on
• Select Options tab
• Click All unknown, product normal
• Then OK
• A new dialog opens to specify means and variances

22
Model 1 example

• Uniform distributions from input ranges
• Normal distributions to match
• Range about 4 std deviations
• Except for MO
• Narrower distribution

23
Effect on UA

• After running the revised model, we see
• It runs faster, with no need to rebuild the
  emulator
• The mean is changed a little and variance is
  halved

The emulator fit is unchanged
Estimate of mean output is 26.2698, with variance
0.00784475 Estimate of total output variance
38.1319
24
Reducing MO uncertainty further

• If we reduce the variance of MO even more, to 49
• UA mean changes a little more and variance
  reduces again
• Notice also how the emulation uncertainty has
  increased (0.004 for uniform)
• This is because the design points cover the new
  ranges less thoroughly

Estimate of mean output is 26.3899, with variance
0.0108792 Estimate of total output variance
27.1335
25
A homework exercise

• What happens if we reduce the uncertainty in MO
  to zero?
• Two ways to do this
• Literally set variance to zero
• Select Some known, rest product normal on
  Project dialog, check the tick box for MO in the
  mean and variance dialog
• What changes do you see in the UA?

26
Cross-validation

• Reopen the Project dialog and select the Options
  tab
• Look at the bottom menu box, labelled
  Cross-validation
• There are 3 options
• None
• Leave-one-out
• Leave final 20 out
• CV is a way of checking the emulator fit
• Default is None because CV takes time

27
Leave-one-out CV

• After estimating roughness and other parameters,
  GEM predicts each training run point using only
  the remaining n-1 points
• Results appear in log window

Cross Validation Root Mean-Squared Error
0.907869 Cross Validation Root Mean-Squared
Relative Error 4.34773 percent Cross Validation
Root Mean-Squared Standardised Error
1.15273 Largest standardised error is 4.32425 for
data point 61 Cross Validation variances range
from 0.18814 to 3.92191 Written cross-validation
means to file cvpredmeans.txt Written
cross-validation variances to file cvpredvars.txt
(Model 1, output 4, uniform inputs)
28
Leave final 20 out CV

• This is an even better check, because it tests
  the emulator on data that have not been used in
  any way to build it
• Emulator is built on first 80 of data and used
  to predict last 20
• Standardised error a bit bigger
• But not bad for just 24 runs predicted

Cross Validation Root Mean-Squared Error
1.46954 Cross Validation Root Mean-Squared
Relative Error 7.4922 percent Cross Validation
Root Mean-Squared Standardised Error
1.73675 Largest standardised error is 5.05527 for
data point 22 Cross Validation variances range
from 0.277304 to 4.886
29
Output Summary tab

• The Output Summary tab presents all of the key
  results in a single list
• Tidier than searching for the details in the log
  window
• Although the log window actually has more
  information
• Can print using

30
Other options

• There are various other options associated with
  the emulator building that we have not dealt with
• See built in help facility for explanations
• Also slides at the end of session 3
• But weve done the main things that should be
  considered in practice
• And its enough to be going on with!

31
When it all goes wrong

• How do we know when the emulator is not working?
• Large roughness parameters
• Especially ones hitting the limit of 99
• Large emulation variance on UA mean
• Poor CV standardised prediction error
• Especially when some are extremely large
• In such cases, see if a larger training set helps
• Other ideas like transforming output scale
• A suite of diagnostics is being developed in MUCM
• See Bastos and OHagan on my website
• http//tonyohagan.co.uk/academic/pub.html
• Not implemented in GEM-SA yet