Calibration, verification and validation ,

1
Calibration, verification and validation(????????
????, ??????????? ? ?????????)
Jos van Dam
2
Terms (???????)

• Verification (???????????)
• Show that the simulation program performs
  according to the purpose (?????????????
  ???????????? ????????????? ????)
• Sensitivity analysis (?????? ????????????????)
• Determination of the change of the model results
  due to change of a parameter value (???????????
  ??????? ????????? ???????? ?????????? ??
  ?????????? ?????????????)
• Calibration (????????????)
• Adjustment of the model by optimization of the
  parameter values, determination of systematic
  errors, improvement of the model concept, etc.
  (???????? ?????? ?? ???? ??????????? ????????
  ??????????, ?????????? ??????????????? ??????,
  ????????????????? ????????? ?????? ? ?.?.)
• Validation (?????????)
• Show that the model is able to predict the
  systems behaviour

3
Verification (???????????)

• More than no compilation and link errors of the
  compiler or reasonable results (??? ?????? ???
  ????? ?????????????? ?????? ? ?????? ???
  ?????????????????? ??????????)
• Preferably comparison to analytical or
  independent numerical solution (???????????????
  ?????????? ? ????????????? ??? ???????????
  ????????????? ????????)
• Check with debugger all steps (???????? ?
  ?????????? ???? ?????)
• Test extremes (???????????? ???????????)
• Requires systematic and creative analysis of the
  code (??????? ????????? ??????????????? ?
  ????????????? ?????? ???? ??????)
• Let other people work with your code (????
  ??????????? ?????? ?????????? ? ????? ??????)

4
Sensitivity analysis (?????? ????????????????)

• Conditions (???????)
• Choose reference situation close to model
  application (??????? ???????? ??????? ?
  ?????????? ??????)
• Adjust parameter according to its accuracy
• (????????? ????????? ???????? ?? ????????)
• Results (??????????)
• The most sensitive parameters (????????
  ?????????????? ?????????)
• Sensitivity of model results (????????????????
  ??????????? ??????)
• Required accuracy of input data (?????????
  ???????? ??????? ??????)

5
Example sensitivity field application (??????
???????????????? ????????????? ? ????)
Experimental period 1 October 1992 - 30 September
1993 (?????? ???????????? )
Maize crop from 1 May until 30 September (?????
???????? )
Optional application bromide tracer 250 (?g
cm-2) at 1 October 1992 (????????????? ??????
???????? ??????? )
Top soil loam (??????? ???? ????? - ????????)
-0.40
Sub soil loam (?????? ???? ????? - ????????)
Subsurface drains (???????????? ?????)
-0.80
8.0 m
-1.20
Impermeable clay (????????????? ?????)
6
Calculated infiltration flux and groundwater
levels (???????????? ???????????????? ????? ?
??????? ????????? ???)
7
Calculated drainage flux and root zone pressure
head (???????????? ????????? ????? ? ????????
????????? ????? ? ?????????????? ????)
8
Root-mean-square-error (RMSE)(????? ?????????
??????????)
with yref,I the simulated value in the
reference situation (????????, ???????????????
???????????? ????????) ysim,I the simulated
value with the adapted parameter (????????,
???????????? ?? ??????) n number of data of the
considered quantity (????? ????????
???????????????? ?????? ????????)
9
Results sensitivity analysis (?????????? ???????
????????????????)
10
Calibration search processes(???????
?????????????? ??????)
Model concept ????????? ??????
Model input ???? ??????
Measurements ?????????
Model parameters ????????? ??????
Unknown parameters ??????????? ?????????
Residuals ???????
Calculated parameters ????????? ?????????
Statistical information ?????????????? ??????????
External search process ??????? ???????? ??????
Measurement errors and acceptable model
errors ?????? ????????? ? ?????????? ?????? ??????
Yes
No
11
Field measurements De Waag (???????
???????????? )
Soil profile (????????? ???????)
Measurements (?????????)

• Water transport (????????????)
• rain, groundwater tables, drain discharges, soil
  water contents (??????, ??????? ????????? ???,
  ????????? ????, ??????????????? ?????)
• water retention and hydraulic conductivity
  function (??????? ??????????????? ?
  ?????????????? ????????????????)
• Non-reactive solute transport (Br-) (???????
  ????????????? ???????? Br)
• concentrations in soil profile and drainage water
  (???????????? ? ????????? ??????? ? ?????????
  ?????)
• Reactive solute transport (dichloorpropene)
  (??????? ??????????? ???????? )
• concentrations in soil profile and drainage water
  (???????????? ? ????????? ??????? ? ?????????
  ?????)
• soil temperatures (??????????? ?????)
• decomposition rate (??????????? ??????????)
• Adsorption (?????????)

95 cm
15 cm
12
Calibration procedure (????????? ????????????)
Step 1 Soil water flow (??? 1 ???????????? ?
?????) Fitting groundwater levels, drainage
discharges and soil moisture content (??????
??????? ????????? ???, ????????? ???? ?
?????????? ????????? ?????) Adapting soil
hydraulic functions (????????? ??????????????
??????? ?????) Step 2 Non-reactive solute
transport (Br-) (??? 2 ??????? ?????????????
???????? ) Fitting concentrations in soil and
drainage water (?????? ???????????? ? ????? ?
????????? ?????) Adapting dispersion coefficient
(????????? ??????????? ?????????) Step 3
Reactive solute transport (Dichloorpropene) (???
3 ??????? ??????????? ???????? ) Fitting
concentrations in soil and drainage water
(?????? ???????????? ? ????? ? ?????????
?????) Adapting adsorption and decomposition
parameters (????????? ????????? ????????? ?
??????????)
13
Observed and simulated drainage discharges
(?????????? ? ?????????????? ???????? ??????????
?????)
14
Calculated and measured bromide concentrations
(?????????? ? ?????????????? ???????????? ???????)
15
Variability for 30 month period (????????? ?
??????? ???? ???????)
16
Automatic calibration field-scale pesticide
transport (?????????????? ???????????? ????????
?????????? ?? ???????? ????)
Input data ??????? ??????
Agro-hydrological model SWAP ????-???????????????
?????? SWAP
Simulated data ?????????????? ??????
Fitting parameters ?????? ??????????
Comparison ?????????
Parameter estimation program PEST ?????????
?????? ?????????? PEST
New iteration ????? ????????
Observed data ?????????? ??????
After convergence ????? ??????????
Residuals Optimal parameters Confidence
intervals ??????? ??????????? ????????? ??????????
??? ?????????
17
Object function ?(?) (??????? ??????? )
With (???) ? vector with fitting parameters
(?????? ? ???????????? ???????????) yj(ti)
measurement of type j at time ti (??????????
???????? ???? ? ?????? ???????
) yj(ti,?) corresponding model
prediction (??????????????? ????????
????????????? ???????) wj weighting factor for
quantity j (??????? ??????????? ??? ????????
???? )
18
Analysis measurement program (?????????????
????????????????? ?????????)
Experimental period 1 October 1992 - 30 September
1993 (?????? ?????????? ???????????? )
Maize crop from 1 May until 30 September (?????
???????? )
Top soil loam (??????? ???? ????? - ????????)
-0.40
Sub soil loam (?????? ???? ????? - ????????)
-0.80
Subsurface drains (???????????? ?????)
8.0 m
-1.20
Impermeable clay (????????????? ?????)

• Reference situation (as depicted above) (???????,
  ????????? ????)
• Reference situation with cultivation of maize
  (??????? ? ??????? ????????)
• Reference situation with short, intensive
  measurement periods (??????? ?????????
  ???????????? ??????? ?????????)
• Reference situation with use of a tracer (???????
  ? ?????????????? ????????)

19
Procedure prior analysis (?????????
?????????????? ???????)

• Set up a measurement program (type and frequency
  of measurements) (????????? ????????? ?????????
  (??? ? ??????? ?????????))
• Apply the model to expected experimental
  conditions and with estimated values for the
  fitting parameters (????????? ?????? ? ?????????
  ????????????????? ???????? ? ? ??????????
  ?????????? ??????????? ??????????)
• Add a random measurement error to the simulated
  observations (???????? ????????? ??????
  ????????? ? ?????????????? ???????????)
• Optimize with PEST the fitting parameters, using
  the simulated observations corrupted with
  measurement error (?????????????? ? ???????????
  ?????????, ???????????? ???????? ?????????
  ??????????)
• Analyse the accuracy of the optimized parameters
  and their correlations (?????? ????????
  ???????????????? ?????????? ? ?? ??????????)
• Large confidence regions or high correlations
  indicate (??????? ????????????? ????????? ???
  ??????? ???????? ????????????? ??????????
  ????????)
• insufficient information in the proposed
  measurements (??????????????? ?????????? ?
  ???????????? ??????????)
• too many fitting parameters (??????? ?????
  ??????????? ??????????)
• unsuitable combination of fitting parameters
  (????????????? ?????????? ??????????? ??????????)

20
Optimization results for reference situation
(?????????? ?????????? ??? ???????????? ????????)
Parameter (????????)
Mean
CV
Correlation coefficient (????. ??????????)
-1
Crop resistance (s m
)
70.0
0.0075
1.000
........
........
........
Scaling factor (-)
1.00
0.0593
-0.421
1.000
........
........
3
-3
Sat. water content (cm
cm
)
0.40
0.0102
-0.070
0.425
1.000
........
-1
Sat. conductivity (cm d
)
25.0
0.1123
0.119
0.082
0.104
1.000
21
Variation coefficient for 4 measurement programs
(???????????? ?????????? ??? 4-? ?????????
?????????)
Variation coefficient (??????????? ????????)
Parameter (????????)
Sim. 1
Sim. 2
Sim. 3
Sim. 4
Crop resistance
0,007
-
0,009
0,008
Scale factor
0,059
0,055
0,047
0,005
q
0,010
0,008
0,010
0,006
Saturated
top soil
Conductivity sub soil
0,112
0,116
0,030
0,007
Dispersion length solutes
-
-
-
0,012
Sim. 1 Reference situation (as depicted above)
(???????, ????????? ????) Sim. 2 Reference
situation with cultivation of maize (??????? ?
??????? ????????) Sim. 3 Reference situation
with short, intensive measurement periods
(??????? ?????????, ???????????? ???????
?????????) Sim. 4 Reference situation with use
of a tracer (??????? ? ?????????????? ????????)
22
Data use (?????????????? ??????)
Calibration (????????????)

• Use default input data (????????????? ???????
  ?????? ?? ?????????)
• Calibrate system specific measurements
  (???????????? ????????? ????????????? ?????????)

Data set 1 (????? ?????? 1)
Validation (?????????)

• Use default input data and system specific
  measurements (????????????? ??????? ?????? ??
  ????????? ? ????????? ????????????? ?????????)
• Data set 2 should be independent from data set 1
  (????? ?????? 2 ?????? ???? ????????? ?? ??????
  ?????? 1)
• Analyse relevant model results (??????
  ??????????????? ????????? ???????????)

Data set 2 (????? ?????? 2)
Model application (?????????? ??????)

• Calibrate system specific parameters using all
  the data available! (???????????? ?????????
  ????????????? ????????? ? ?????????????? ????
  ????????? ??????)
• Use experiences from earlier model applications
  (????????????? ????????????? ?? ??????????
  ?????????? ?????? )

Data set 1 (????? ?????? 1)

Data set 2 (????? ?????? 2)
23
Published SWAP applications (1994 - 2000)
?????????????? ?????????? ?????????? ?????? SWAP
(1994-2000)