1 Modeling Complex Systems – How Much Detail is Appropriate? David W. Esh US Nuclear Regulatory Commission 2007 GoldSim User Conference, October 23-25, 2007, San Francisco CA

2 Overview Background Model development process Model complexity Model abstraction Examples Conclusions

3 Background The issue of how much detail to include in models of complex systems is not new. 14 th century philosophers were considering different approaches to explain the world around them. Decisions regarding model complexity apply to all fields of study. Modern tools and computational capabilities present unique opportunities.

4 Model – representation of essential aspects of a system (existing or planned)

5 Model Development Process: Key Questions Why are you using a model? What is the purpose of your model? Who is your audience? What are your resources?

6 Model Development Process: Key Questions Why are you using a model? –Developing understanding (integrating, generalizing, testing) –Directing research (identify data gaps, propose new lines of research) –Representing reality (prohibitively costly or can’t observe) What is the purpose of your model? –Is the decision controversial? –Is it high risk? ($, safety, etc.)

7 Model Development Process: Key Questions Who is your audience? –Technical, lay person, policy –High competency, low competency, mix What are your resources? –Now and future –Computational –Time –For collection of additional information

8 Model Development Process: Example Site Assessment Site Selection and Characterization NRC would require a Performance Assessment to: Provide site and design data Describe barriers that isolate waste Evaluate features, events, and processes that affect safety Provide technical basis for models and inputs Account for variability and uncertainty Evaluate results from alternative models, as needed What is Performance Assessment? Systematic analysis of what could happen at a site Collect Data Combine Models and Estimate Effects Develop Conceptual Models Develop Numerical and Computer Models Performance Assessment: a learning process Site Characteristics Design and Waste Form Overview of Performance Assessment Why use it? Complex system Systematic way to evaluate data Internationally accepted approach How is it conducted? Collect data Develop scientific models Develop computer code Analyze results What is assessed? What can happen? How likely is it? What can result?

9 Model Complexity Goals: Simple is better (all things equal) Broader scope Systematic approach Metrics: Accuracy Explanatory Power Reliability and Validity “Theories should be as simple as possible, but no simpler.”

10 Can improve model fit (But does it improve explanatory power?) Can identify the need for enhancements Increases difficulty in understanding Increases difficulty in working with it Increases computational burden Model Complexity So how do I decide?

11 Model Complexity – How Much? Complexity and Effort Comparison of features Mass balance (watershed) GIS based analysis Model comparisons Analogs Long-term field experiments Isotopic studies

12 Model Complexity – How Much? No complete methodologies (generally) –Iteration (+/- interactions) –Statistical analysis of results –Visualization (data and output) –Metamodels Most modelers put too much in to manage the risk of leaving something out If complexity is not inexorably linked with accuracy, there may exist an opportunity to simplify

13 Model Complexity – How Much? 1 Prices go up, farmers produce more (too much) 2 Prices go down, farmers produce less 3 Repeat 1 2 3

14 Models provide information to think about, they don’t do your thinking for you Decision makers need to reason about the issues Model abstraction approaches can and should be used Model Complexity – How Much? Complexity Effort P(decision)

15 Model Abstraction Example NUREG/CR-6884 Model Abstraction Techniques for Soil-Water Flow and Transport

16 Model Abstraction Need to start with a broad model space – allows exploratory analysis essential to abstraction Reduce complexity – maintain validity Show the abstraction represents the complex model Benefits Less $ Fewer inputs Easier to integrate Easier to interpret Types (not exhaustive) Drop unimportant parts Replace with simpler part Coarsen ranges of values Group parts together

17 Model Abstraction: Example Benefit Uncertainty analysis Simpler model yielder stronger results (6 variables identified compared to 3) Allowed focused refinement of model Complexity can have many unintended consequences Variable Description Import ance Factor Grout_deg_st art Time at which degradation of the wasteform can begin 0.98 Nm MacMullin number. The effective diffusion coefficient is a product of Nm and the molecular diffusion coefficient Degraded_gr out_Kh Hydraulic conductivity for degraded region of the wasteform TransFactor_i ndoor Factor to account for shielding of radiation when an individual is inside a residence Se_solubility Solubility of Se in the pore fluid of the wasteform Kd_waste_Sr _ox Distribution coefficient for Sr in the oxidized region of wasteform Vent_light_ac tivity Breathing rate for an individual during light activity SZ_dispersivi ty_factor Used with the transport length in the saturated zone to develop the saturated zone dispersivity Kd_Waste_E u Distribution coefficient for Eu in the intact portion of the wasteform. 0.08

18 Conclusions Methodologies to address the level of model complexity continue to evolve Model abstraction can have many benefits when done properly Simple is better (all things being equal)