1. Uncertainty analysis
and
Model Validation

2. Summary of Results & Conclusions
Final Project
Summary of Results
&
Conclusions

3. In a real-world problem we need to establish model specific calibration criteria and define targets including associated error.
Calibration Targets
associated error
calibration
value
0.80 m
20.24 m
Target with smaller associated error.
Target with relatively
large associated error.

4. Smith Creek Valley (Thomas et al., 1989)
Calibration Objectives
Heads within 10 ft of measured heads. Allows for
Measurement error and interpolation error.
Absolute mean residual between measured and
simulated heads close to zero (0.22 ft) and standard
deviation minimal (4.5 ft).
Head difference between layers 1&2 within 2 ft of
field values.
4. Distribution of ET and ET rates match field estimates.

5. Also need to identify calibration parameters and their
reasonable ranges.

6. Calibration Prediction Group ARM h ARM ET (x10e7) (at targets)
Prediction
Group
ARM h
ARM ET (x10e7)
(at targets)
(at pumping wells) 1
0.92
1.38
1.60
4.16 2
0.73
1.11
1.99
3.03 3
0.69
0.51
0.95
1.76 4
1.34
1.27
1.46
2.57 5
1.56
0.89
2.79
1.43 6
1.29
0.16
2.58
2.92

7. Calibration to Fluxes
When recharge rate (R) is a calibration parameter, calibrating to fluxes can help in estimating K and/or R.

8. In this example, flux information helps calibrate K.
q = KI
K = ? H1 H2

9. In this example, discharge information helps calibrate R.

10. In our example, total recharge is known/assumed to be 7
In our example, total recharge is known/assumed to be 7.14E08 ft3/year and discharge = recharge.
All water discharges to the playa.
Calibration to ET merely fine tunes
the discharge rates within the playa
area.

11. Calibration Prediction Group ARM h ARM ET (x10e7) (at targets)
Prediction
Group
ARM h
ARM ET (x10e7)
(at targets)
(at pumping wells) 1
0.92
1.38
1.60
4.16 2
0.73
1.11
1.99
3.03 3
0.69
0.51
0.95
1.76 4
1.34
1.27
1.46
2.57 5
1.56
0.89
2.79
1.43 6
1.29
0.16
2.58
2.92

12. Includes results from 2000, 2001, 2003

13. Includes results from 2000, 2001, 2003

14. Particle Tracking Group Truth 802 PW1 1913 playa 620 playa 310 PW4
3.93E6 PW4
252 PW4
1084 playa
1576 playa 3
1.21E6 PW1
2.15E6 PW2
3.90E6 playa
1110 PW4
1.58E6 playa
1860 playa
893 playa 4
1200 PW1
1900 PW2
6.7E6 PW3
290 PW4
2800 PW5
760 PW1
1200 PW2 5
1295 PW1
3160 PW2
503 playa
986 PW4
605 PW4
316 PW1
3100 PW2 6
3100 PW1
982 PW1
 4.9E5 playa
 603 PW4
1450 PW4
 2000 PW1
1380 PW2
Truth
802 PW1
1913 playa
620 playa
310 PW4
1933 PW5
690 playa
2009 PW2

15. Observations Predicted ARM > Calibrated ARM
Predicted ARM at pumping wells >
Predicted ARM at nodes with targets
Flow predictions are more robust (consistent among different calibrated models) than transport (particle tracking) predictions.

16. Conclusions Calibrations are non-unique.
A good calibration (even if ARM = 0)
does not ensure that the model will make
good predictions.
You can never have enough field data.
Modelers need to maintain a healthy skepticism
about their results.
Need for an uncertainty analysis to accompany
calibration results and predictions.

17. Uncertainty in the Calibration
Involves uncertainty in:
Targets
Parameter values
Conceptual model including boundary conditions,
zonation, geometry, etc.

18. Ways to analyze uncertainty
in the calibration
Sensitivity analysis
Use an inverse model (automated calibration) to quantify uncertainties and optimize the calibration.

19. Uncertainty in the Prediction
Reflects uncertainty in the calibration.
Involves uncertainty in how parameter values
(e.g., recharge) will vary in the future.

20. Ways to quantify uncertainty
in the prediction
Sensitivity analysis
Stochastic simulation

21. MADE site – Feehley and Zheng, 2000, WRR 36(9).

23. A Monte Carlo analysis considers 100 or more realizations.

25. Stochastic modeling option
in GW Vistas

26. Ways to quantify uncertainty
in the prediction
Sensitivity analysis
Scenario analysis
Stochastic simulation

27. Model Validation How do we “validate” a model so that
we have confidence that it will make
accurate predictions?

28. Modeling Chronology 1960’s Flow models are great!
1970’s Contaminant transport models are great!
1975 What about uncertainty of flow models?
1980s Contaminant transport models don’t work.
(because of failure to account for heterogeneity)
1990s Are models reliable? Concerns over
reliability in predictions arose over efforts to model
a geologic repository for high level radioactive waste.

29. “The objective of model validation is to determine how well the mathematical representation of the processes describes the actual system behavior in terms of the degree of correlation between model calculations and actual measured data”
(NRC, 1990)

30. What constitutes validation? (code vs. model)
NRC study (1990): Model validation is not possible.
Oreskes et al. (1994): paper in Science
Calibration = forced empirical adequacy
Verification = assertion of truth
(possible in a closed system, e.g., testing of codes)
Validation = establishment of legitimacy (does not contain
obvious errors), confirmation, confidence building

31. How to build confidence in a model
Calibration (history matching)
steady-state calibration(s)
transient calibration
“Verification”
requires an independent set of field data
Post-Audit: requires waiting for prediction to occur
Models as interactive management tools

32. HAPPY MODELING!

33. Have a good summer!