Quantifying Uncertainty using Classical Methods Likelihood Profile, Bootstrapping

1
Quantifying Uncertainty using Classical
Methods(Likelihood Profile, Bootstrapping)

• Fish 458, Lecture 12

2
Quantifying Uncertainty(an overview)

• Uncertainty comes in several forms
• Process uncertainty (e.g. recruitment
  variability, natural mortality variability,
  birth-death processes).
• Observation uncertainty (e.g. CVs for abundance
  estimates).
• Model uncertainty (is the model we chose correct
  how many alternative models fit the data
  adequately?)
• Estimation uncertainty given a model and some
  data, how well do the data determine the
  parameters (and predictions) of the model.
• Implementation uncertainty given a management
  decision, it be enforced?

3
Quantifying Uncertainty(an overview-II)

• The various types of uncertainties can be
  distinguished by
• Can they be reduced by additional research or are
  they inherent to the system.
• Can we quantify them using classical statistical
  methods.
• Today we address estimation uncertainty. We
  defer the other types of uncertainties to future
  lectures.

4
Estimation Uncertainty

• We are going to quantify uncertainty about the
  estimates of the model parameters (and its
  predictions of state variables) under the
  assumption that the model (and likelihood) are
  correct.
• Typical ways to quantify estimation uncertainty
  include computing standard errors and confidence
  intervals.

5
Hint Generic Solver Macro-I
Sub ApplySolver(Minpars, TheSheet,
TheFunctionValue) Worksheets(TheSheet).Activate
SolverReset ' Small precision and
automatic scaling SolverOptions
Precision0.00001, ScalingTrue ' Specify
the cell to minimize (MaxMilVal2) and the
parameters to change SolverOK
SetCellRange(TheFunctionValue), MaxMinVal2,
ByChangeRange(Minpars) Add a constraint
(in this case the cells must be positive)
SolverAdd CellRefRange(Minpars), Relation3,
FormulaText0 ' Don't ask anything at the
end of the call SolverSolve
UserFinishTrue End Sub
6
Hint Generic Solver Macro-II

• Notes
• Many EXCEL versions do not have all the Visual
  Basic libraries needed to call SOLVER from a
  macro loaded.
• Within the Visual Basic editor you will need to
  click Tools - References - Solver.xls to
  make this library accessible.

7
Likelihood Profile (one parameter)

• Fit the model to find the ML parameter estimates
  and the corresponding negative log-likelihood.
• Select a set of fixed values for the parameter of
  interest.
• Minimize the negative log-likelihood fixing the
  parameter to each value in turn.
• Plot the difference between the negative
  log-likelihood from step 1 and those from step 3.

8
A First Example-I
Note t00 and ? is assumed known
The problem We are fitting a growth curve to
some age and length data. We want to compute the
likelihood profile for ?
9
A First Example-II
We can compute confidence intervals from
likelihood profiles
Approximate 95 CI
10
Likelihood profiles and confidence intervals

• An 100-x confidence interval for p parameters is
  determined by finding the values for the
  parameter(s) for which
• is the negative log-likelihood
  corresponding to the maximum likelihood
  estimates.

11
Likelihood Profile (multiple parameters)

• Fit the model to find the ML parameter estimates
  and the corresponding negative log-likelihood.
• Select a set of fixed parameter combinations for
  the set of parameters of interest.
• Minimize the negative log-likelihood fixing the
  values for the set of parameters of interest to
  each set of values in turn.
• Plot the difference between the negative
  log-likelihood from step 1 and those from step 3
  (this creates a surface).

12
What about State Variables-I

• Model outputs often include population size,
  harvest rate, etc. We are usually more interested
  in these quantities than about the parameters
  themselves.
• However, the state variables are seldom
  parameters of the model (and cannot be made to be
  parameters of the model). This makes computing a
  likelihood profile for them difficult.

13
What about State Variables-II

• For each (target) value of the State Variable
• Add a penalty to the negative log-likelihood that
  increases as the difference between the target
  value and the model estimate is larger.
• It is often a good idea to change the size of the
  penalty, w, as we get closer to the target (i.e.
  apply Solver several times, each time increasing
  w)

14
A Likelihood Profile for Current Biomass
15
Bootstrapping-I

• Likelihood profile has some major disadvantages
• Dealing with derived quantities (e.g. state
  variables) can be difficult.
• Dealing with multiple parameters simultaneously
  is very computationally intensive.
• It is impossible to compute likelihood profiles
  for all of the state variables simultaneously.
• These problems can be overcome through
  bootstrapping.

16
Bootstrapping-II

• Bootstrapping deserves a course of its own but,
  in (very) simple terms, it involves
• Generate a large number of pseudo data sets, each
  based on the original data set.
• Fit the model to each such data set.
• Compute summary statistics of interest (standard
  deviations, confidence intervals, etc.) from the
  results for each model fit.

17
Bootstrapping III

• The most common form of bootstrapping involves
  developing the pseudo data set by resampling the
  residuals (with replacement) and adding these to
  the model predictions
• is the index for year y in pseudo data set U,
• is the model prediction of the index for year
  y, and
• y is selected at random from 1n.

18
Bootstrapping IV(the biomass trajectory for
Cape Hake)
19
Readings

• Hilborn and Mangel, Chapter 7
• Haddon, Chapter 3