1 Annual Report by Work Group 2 on Uncertainty Analysis and Parameter Estimation Tom Nicholson, U.S. Nuclear Regulatory Commission Mary Hill, U.S. Geological Survey 2014 ISCMEM Annual Public Meeting February 26, 2014 DOE Headquarters, Washington, DC

2 Outline Work Group 2 (WG2) Objective and Goals Members and Participants Activities and Technical Projects Seminars at the WG2 Meetings Methodologies, Tools and Applications Forward Strategy Recommendations for FY2014

3 Work Group Objective  Coordinate ongoing and new research conducted by U.S. Federal agencies on: parameter estimation uncertainty assessment in support of environmental modeling and applications  Focus on strategies and techniques  Includes sensitivity analysis What is needed to achieve this objective? Coordination of research staff and their management thru efficient and targeted use of our limited resources.

4 Work Group Goals Basics: Develop a creative, collaborative environment to advance  parameter estimation in the context of model development.  sources of uncertainty in the context of model predictions. Develop a common terminology. Identify innovative applications. Existing Tools: Identify, evaluate, and compare available analysis strategies, tools and software. New Tools: Develop, test, and apply new theories and methodologies. Exchange: Facilitate exchange of techniques and ideas thru teleconferences, technical workshops, professional meetings, interaction with other WGs and ISCMEM Web site links: Communicate: Develop ways to better communicate uncertainty to decision makers (e.g., evaluation measures, visualization). Intermediate scale (2m) Plume scale (2000m) Batch scale (0.01m) Butler et al. Electrical Conductivity Geophysics (2-200m) Column scale (0.1 m) Tracer test scale (1-3m)

5 Members and Participants from U.S. Federal agencies, universities, and industry Tom Nicholson, NRC, co-Chair Mary Hill, USGS, co-Chair Ming Zhu, DOE Tommy Bohrmann, EPA Gary Curtis, USGS Bruce Hamilton, NSF Yakov Pachepsky, USDA-ARS Tom Purucker, EPA-Athens Yoram Rubin, UC Berkeley Brian Skahill, USACOE Matt Tonkin, SSPA Gene Whelan, EPA-Athens Steve Yabusaki, PNNL Ming Ye, Florida State U Sanja Perica, NOAA/NWS Larry Deschaine, HydroGeologic, Inc. Boris Faybishenko, LBNL Pierre Glynn, USGS Philip Meyer, PNNL Debra Reinhart, NSF You?

6 Activities: Seminars We conduct seminars to: review and discuss ongoing research studies and software development formulate proposals for field applications from 6/26/2013 seminar by Sanja Perica, NOAA/NWS on NOAA Atlas 14 How to retrieve Precipitation Frequency estimates with confidence limits?

7 Seminar at WG2 in FY2013 NOAA Atlas Precipitation – Frequency Atlas of the U.S.: Products and Methods by Sanja Perica, Director, Hydrometeorological Design Studies Center, Office of Hydrology, National Weather Service, NOAA. Current NOAA/NWS Precipitation Frequency (PF) documents by State location: Access NOAA/NWS’s Precipitation Frequency Data Server (PFDS) at:

8 Seminars at WG2 Teleconferences (continued) NOAA Atlas 14 Simulation used to construct 90% confidence intervals (i.e., 5% and 95% confidence limits) at stations. Algorithm adjusted to account for inter-station correlation. Estimates interpolated on 30 arc-sec grid. Uncertainty analysis and sensitivity analyses are included in the simulation result analyses.

WG2 Activities UCODE_2014 UCODE - A Computer Code for Universal Inverse Modeling Eileen P. Poeter, Colorado School of Mines and the Integrated Groundwater Modeling Center, Emeritus Mary C. Hill, U.S. Geological Survey Dan Lu, Oak Ridge National Laboratory Steffen Mehl, California State University-Chico

Selected New Features Evaluate Uncertainty with Markov-Chain Monte Carlo (MCMC) –Use for non-Gaussian distributions. Often this means nonlinearity produces local minima New stacked sensitivity graphs –Show parameter importance and contributions from different types of observations 10

Evaluate Uncertainty with Markov- Chain Monte Carlo (MCMC) The DREAM algorithm used (Vrugt et al., 2009) is able to search very complicated distributions This example is for an very nonlinear analytic function from Lu Ye Hill, 2012, WRR 11

Quantifying Uncertainty: Tests using a groundwater model of a synthetic system Lu, Ye, Hill, 2012, WRR Investigate methods that take 10s to 100,000s of model runs (Linear to MCMC) New version of UCODE with MCMC 12

Test Case MODFLOW 3.8 km x 6 km x 80m Top: free-surface No-flow sides, bottom Obs: 54 heads, 2 flows, lake level, lake budget Parameters: 5 for recharge KRB confining unit KV net lake recharge, vertical anisotropy K : 1 (HO), 3 (3Z), or 21 (INT) Data on K at yellow dots True 3Z Int 13

Predictions, Confidence Intervals HO3ZInt Standard error of regression 1.49 ( ) 1.27 ( ) 1.00 ( ) 1 Intrinsic nonlinearity RN2RN2 True Model runs 100 1, ,000 Statistics to use when the truth is unknown HO 3Z INT Name of alternative model…….. Lesson about uncertainty  May be more important to consider the uncertainty from many models than use a computationally demanding way to calculate uncertainty for one model 14

New stacked sensitivity graphs Here, parameter T16 is most important, P110 is least. (T is a transmissivity, P is porosity) Head obs ( ) contribute to the T parameters. Time of travel ( ), concentration( ), and source ( ) (fraction of water from different locations) are broadly important Hanson, Kauffman, Hill, Dickinson, Mehl, 2013, MODPATH-OBS documentation, USGS 15

The following slides show the problem for which the stacked graph was created 16

Observations 15 Heads 5 Proximity to expected location by a defined time 14 Time of travel 50 “Concentrations” 25 Source water type Parameters 5 K 5 Porosities Particle path figures are provided to show simulation tests 17

Trench Pit Farm Well 3 Obs well 1 Well 2 Obs well 1 Particle paths 18

PCE concentrations simulated to reach well 2 from the trench and the pit Starting paramete r values Estimated paramete r values 19

Constant Head=11 Constant Head=0 Local grid refinement (LGR) to simulate pumping. Particles transported through the locally refined grid. Parameters defined in zones: K, porosity. 20

21

Sensitivity analysis New DELSA. Compare to global SA methods USGS-NCAR collaborative study with Mary Hill, USGS; Martyn Clark, NCAR; Olda Rakovec, U of Wageningen, and others from his university. Rakovec WRR New DELSA multi-scale statistic (1,000 model runs) DELSA: Distributed Evaluation of Local Sensitivity Analysis Compare to global variance-based Sobol’ statistics (10,000 model runs) Rainfall-runoff problem –FUSE modular model (Clark et al. 2008, WRR) 22

SOILR GFLWR GWATR IZONE LAYR2 LAYR1 PCTIM ADIMP ADIMC UZTWCUZFWC LZTWC LZFSCLZFPC IMPZR RZONE IFLWR GFLWR LZONE PRMSSACRAMENTO ARNO/VICTOPMODEL FUSE supports quantifying uncertainty using multiple models… 23

Define model building decisions—1 SOILR GFLWR GWATR IZONE LAYR2 LAYR1 PCTIM ADIMP ADIMC UZTWCUZFWC LZTWC LZFSCLZFPC IMPZR RZONE IFLWR GFLWR LZONE PRMSSACRAMENTO ARNO/VICTOPMODEL UPPER LAYER ARCHITECTURE 24

SOILR GFLWR GWATR IZONE LAYR2 LAYR1 PCTIM ADIMP ADIMC UZTWCUZFWC LZTWC LZFSCLZFPC IMPZR RZONE IFLWR GFLWR LZONE PRMSSACRAMENTO ARNO/VICTOPMODEL LOWER LAYER ARCHITECTURE / BASEFLOW Define model building decisions—2 25

SOILR GFLWR GWATR IZONE LAYR2 LAYR1 PCTIM ADIMP ADIMC UZTWCUZFWC LZTWC LZFSCLZFPC IMPZR RZONE IFLWR GFLWR LZONE PRMSSACRAMENTO ARNO/VICTOPMODEL PERCOLATION Define model building decisions—3 26

SOILR GFLWR GWATR IZONE LAYR2 LAYR1 PCTIM ADIMP ADIMC UZTWCUZFWC LZTWC LZFSCLZFPC IMPZR RZONE IFLWR GFLWR LZONE PRMSSACRAMENTO ARNO/VICTOPMODEL SURFACE RUNOFF Define model building decisions—4 27

SOILR GFLWR GWATR IZONE LAYR2 LAYR1 PCTIM ADIMP ADIMC UZTWCUZFWC LZTWC LZFSCLZFPC IMPZR RZONE IFLWR GFLWR LZONE PRMSSACRAMENTO ARNO/VICTOPMODEL and mix architecture + parameterizations… TO CREATE HUNDREDS OF MODELS, ALL WITH DIFFERENT STRUCTURE 28

Field problem Belgian watershed Consider RMSE 29

Parameters common to all 5 alternative models constructed using FUSE Global method produces averages. DELSA provides a distribution DELSA first-order sensitivities [-] 30

Dig deeper into DELSA for FUSE-016 Sobol DELSA DELSA first-order sensitivity 31

from Kavetski and Kuczera 2007 WRR Potential reason for variable sensitivities 32

SOILR GFLWR GWATR IZONE LAYR2 LAYR1 PCTIM ADIMP ADIMC UZTWCUZFWC LZTWC LZFSCLZFPC IMPZR RZONE IFLWR GFLWR LZONE PRMSSACRAMENTO ARNO/VICTOPMODEL FUSE supports quantifying uncertainty using multiple models… 33

Lesson from DELSA Utility of detailed SA, including evaluation at many sets of parameter values and many models. Also, over time. 34

35 Forward Strategy Energize the science and technology thru closer linkage to decision making:  better understand the methods being used in parameter estimation and uncertainty analyses  establish a base set of model sensitivity analysis and uncertainty evaluation measures, in addition to the other performance measures  use and compare different methods in practical situations

36 Recommendations for FY2014 Assist development and creation of other working groups –Take advantage of the relevance of uncertainty and parameter estimation to all environmental modeling and monitoring fields. –Develop and conduct joint ISCMEM teleconferences WG1 (Software System Design; design of uncertainty and parameter estimation software and data fusion) WG3 (Reactive Transport Models and Monitoring; support decision making) –Act as an incubator to build support for new ideas Proposed WG on monitoring based on the importance of monitoring to uncertainty and parameter estimation, and visa versa Sponsor technical workshops on endorsed studies ISCMEM Website –Use EPA’s iemHUB to enhance Information Transfer of Technical Reports and Data Sources

37