VIII: Methods for Evaluating Model Predictions 1. Define predictive quantity and calculate sensitivities and standard deviations (Ex8.1a) 2. Assess data needs for the predictions Which parameters are important to predictions? Use composite and prediction scaled sensitivities (css and pss) and parameter correlation coefficients (pcc) (Ex8.1b) Use parameter-prediction (ppr) statistic (Ex8.1c) What existing observations are important to predictions? Use observation-prediction (opr) statistic (Ex8.1d) How important are the proposed new observations? Use dss, css, pcc (Ex8.1e) Use opr (Ex8.1f) 3. Quantify prediction uncertainty Linear confidence and prediction intervals Nonlinear confidence and prediction intervals Monte Carlo – no exercise

Exercise files MFI2005 does not support predictions All the files have been constructed for you, and are in initial\ex8.i Copy the files to exer\ex8 Execute batch files as described in the input instructions. Look at output files For other problems, you can use the class files as a beginning point.

Define predictive quantity We typically make predictions to assess the state of the simulated system at a future time. Often, predictions occur under different conditions than those under which the model was calibrated – addition of pumping, change in boundary conditions, etc. The predictive quantity might differ from the types of quantities used as observations. You as the modeler need to help define what model-calculated values are best to use as predictions in the problem of concern The predictions available can depend on the software used MODFLOW-2005: any quantity that can be an observation can also be a prediction. Heads, temporal changes in head, some flows, advective transport. UCODE_2005 or PEST: any quantity that can be calculated using values in the application model output files.

Prediction sensitivities (Book, p. 159) Used to calculate measures for assessing: Parameters important to predictions Observations important to predictions Prediction uncertainty Data needs Prediction sensitivities: Usually need to scale these sensitivities to produce useful measures.

Prediction scaled sensitivities (pss) (Book, p ) Scaling depends on prediction and purpose. Four useful scalings are: 1. pss is percent change in prediction caused by a 1-percent change in the parameter value b j (in _sppp file of UCODE_2005): pss j = (  z /  b j ) (b j /100) (100/z ) 2. pss is percent change in prediction caused by a change in b j equal to 1 percent of its standard deviation s b j (in _spsp file) : pss j = (  z /  b j ) (s b j /100) (100/z ) 3,4. For both these scalings, (100/z ) can be replaced by (100/a ), where a’ is some other meaningful quantity. On p. 161, using a reference value is suggested. (in _sppr and _spsr file of UCODE_2005) ps pr In _sp**, third letter identifies parameter value or standard deviation on the parameter; fourth letter identifies prediction or reference value)

Parameter Correlation Coefficients (pcc) in the Context of Predictions (Book, p ) Calculate pcc using different combinations of observations, prior, and predictions to evaluate the existence and need for unique parameter estimates For example, two parameters are extremely correlated given a set of observations. This is only a problem if the predictions need unique values. Next: Use pss, css, and pcc together to assess the importance of parameters to the predictions.

Use pss and css to identify insensitivity problems Are imprecise parameter estimates important to the predictions? (from Hill & Tiedeman, Figure 8.2a, p. 166)

Use pcc to identify uniqueness problems Are non-unique parameter estimates important to the predictions? (from Hill & Tiedeman, Figure 8.2b, p. 166)

Developer claims landfill effluent will go to river, not to wells. Water supply wells are being completed and a pump test could be used to collect more data on the system. Developer claims the model is inadequate for two reasons: 1.Model was calibrated using heads and flows and no pumping, but is being used to predict advective transport under pumping conditions. 2.Need for prior suggests the observations are inadequate County government officials want to know: Is existing model adequate? Wait for new data? We, as the modelers, suggest using sensitivity analysis to: Evaluate the developer’s claims. Plan new data collection Prediction Exercises – Book, p

Prediction Exercises

Will effluent go to well or river? Thorough analysis requires full transport model (advection, dispersion, etc.). We will do preliminary analysis using Advective Transport. Use MODPATH. ? ? Predicted transport from landfill

Predictions simulated using MODPATH (Pollock 1996) Calculates motion in the x, y and z directions. When used for observations: Observed advective transport inferred from concentrations. Three entries are added to the objective function for each advective-transport observation. When used for predictions: Three predictions for each advective transport prediction – transport along rows (y), columns (x), and vertically (z). Calculates sensitivities; these are used here to calculate pcc, parameter standard deviations, and linear confidence intervals.

Start of path Particle path simulated using MODPATH. The horizontal and vertical bars show  one standard deviation at selected travel times. End of path

Predictive transport: Questions to address (p. 194) 1. Transport destination and time? Ex. 8.1a. Forward model with MODPATH 2. Consider the parameters important to predictions. Are they precisely and uniquely estimated? Prediction and composite scaled sensitivities and parameter correlation coefficients (pss, css, pcc). Ex. 8.1b. PPR statistic. Ex. 8.1c 3. What existing observations are important? OPR statistic. Ex. 8.1d 4. Are potential new observations worth waiting for? Dimensionless and prediction scaled sensitivities, parameter correlation coefficients (dss, pss, pcc). Ex. 8.1e OPR statistic. Ex. 8.1f 5. What is the prediction uncertainty? Linear confidence intervals. Ex. 8.2a. Nonlinear confidence intervals. Ex 8.2b.

Exercise 8.1a: Predict Advective Transport Read general Exercises description on p Do Exercise 8.1a on p In the simulation, where does the landfill effluent go, and how long does it take to get there?

MODPATH Output (Equivalent to ADV output on Fig. 8.7a, p. [ MODPATH Version 4.00 (V4, Release 3, ) (TREF= E+00 ) ] E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E Distance along rows from left Distance along columns from bottom Level above bottom of model Row, column, layer Level above bottom of model layer Time, in seconds

Exercise 8.1b: Determine problematic parameters using css and pss 1.Compare prediction and composite scaled sensitivities: Are pss large for any parameters with small css? Use pss scaling so that pss represent the percent change in distance traveled caused by a 1-percent change in a parameter value: For UCODE_2005, these pss are in ex8.1b \ ex8.1b_sppp. pss j = (  z /  b j ) (b j /100) |100/z | One-percent scaled sens

Exercise 8.1b: Determine problematic parameters using pcc 2.Compare two different sets of parameter correlation coefficients: pcc calculated using only calibration observations pcc calculated with calibration observations AND predictions. Use parameter correlation coeffeicients in ex8.1b \ ex8.1b._mc or _pcc ex8.1b-obs-pred\

Weights for the advective travel predictions are calculated from specified standard deviations, in meters (Table 8.3, p. 198). Weights in this analysis can reflect desired prediction precision. Time of advective travel Direction10 years50 years100 years X200 m600 m1000 m Y200 m600 m1000 m Z10 m15 m25 m Exercise 8.1b: Prediction weights needed for pcc with predictions Do Exercise 8.1b (p ) and the Problems. Question 2: Are parameters important to predictions precisely and uniquely estimated?

Results of Exercise 8.1b css-pss analysis for Question 2: Are parameters important to predictions precisely estimated? Figure 8.8, p. 198

Results of Exercise 8.1b css-pss analysis for Question 2: Are parameters important to predictions precisely estimated? Figure 8.8, p. 198 Not all of them

Observations only With predictions pcc analysis for Question 2: Are parameters important to predictions uniquely estimated? Results of Exercise 8.1b Tables 8.4 & 8.5, p. 199 No correlations larger than 0.85 Not all of them

Exercise 8.1c: PPR Individual Parameters Which parameters rank as most important to the predictions by the ppr statistic? Average ppr statistic for all predictions Figure 8.9a, p. 201 How does this differ from the rankings by the pss? What causes these differences?

Response to developer concerns Developer claims the model is inadequate for two reasons: Model was calibrated using heads and flows and no pumping, but is being used to predict advective transport under pumping conditions. Need for prior suggests the observations are inadequate Our response based on sensitivity analysis Indeed, the model is inadequate. The calibration data does not provide information on some parameters important to predictions Also, we stress that we need to consider the results of an uncertainty analysis