Verification and Validation of Agent-based Scientific Simulations Xiaorong Xiang, Ryan Kennedy, Gregory Madey Computer Science and Engineering University of Notre Dame Steve Cabaniss Department of Chemistry University of New Mexico 9/22/2018

Overview Introduction A Case Study Concepts of Verification and Validation Research Objectives and Methods A Case Study Apply Verification and Validation Methods to the Case Study Conclusion Future Work 9/22/2018

Model Verification & Validation (V & V) Verification: get model right Validation: get right model The cost and value influence confidence of model acceptance level Model verification is a process determining if the implmentation of model have sufficient accuracy Model validation is a process determing if the model represent the real world phenomena The purpose of V & V is t o increase the model confidence. According to the relationships of cost, value and model confidence, it is critical to find most cost-effictive way for conducting the process , the verification and validation process are conducted until sufficient confidence is obtained The purpose of V & V is to increase confidence of the model. Users’ belief of the model is high. How good and useful the model for users. V&V can be go forever, need to decide when should stop the V & V process. Get the highest model confidence with low cost. Find optimal approach for V & V of ABS. DES is the most closely approach as ABS. It is too costly and time consuming to determine that a model is absolutely valid to be used practically. Instead, the verification and validation process are conducted until sufficient confidence is obtained Model verification: the process of determining whether a computational model of a physical event and the code implementing the model represent the model with sufficient accuracy Model validation: the process of determining whether a mathematical model of a physical event represents the actual event with sufficient accuracy *Adapted from Sargent: “Verification and Validation of Simulation Models” 9/22/2018

V & V for Agent-based Simulation Agent-based modeling is a new approach Different than Queuing Models Entities: large number of heterogeneous active objects vs. passive objects Space: continuous or discrete grid space vs. network of servers and queues Interactivity: high vs. low Active components: agents vs. queues and servers Goal: discovery vs. design and optimization Few literature to date address the formalized methodology for V & V of Agent-based Simulations Agent-based modeling is only 10 years old, and recently become popular for modeling the complex system with dynamic behaviors. Closest to DES with the following differences. Ant, bees. Behave independently, and a set of behavioural rules, the global behavior of the system is can discovered ABS focus on systems that contain large numbers of active objects (people, business units, animals, vehicles, and time events ordering or other kind of invididual behavior associated with them No global system behaviors (dynamics) would be defined, the modelers defines behavior at individual level, ,and the global behavior emerges as a results of many individuals, each following its own behaviors rules, living together in some environment and communicating with each other and with the environment 9/22/2018

What and How Research objective How Generate guidelines or a formalized methodology for V & V of Agent-based Simulations How NOM project as a case study Evaluate and adapt the formalized V & V techniques in industrial and system engineering for DES Identify a subset of these techniques that are more cost-effective for Agent-based Simulations 9/22/2018

NOM Agent-based Simulation Model NSF funded interdisciplinary project Understanding the evolution and heterogeneous structure of Natural Organic Matter (NOM) E-science example Chemists, biologists, ecologists, and computer scientists Agent-based stochastic model Web-based simulation model 9/22/2018

NOM What is NOM? Why study NOM? Heterogeneous mixture of molecules in terrestrial and aquatic ecosystems Why study NOM? Plays a crucial role in the evolution of soils, the transport of pollutants, and the global carbon cycle Understanding NOM helps us better understand natural ecosystems 9/22/2018

The Conceptual Model I Agents A large number of molecules Behaviors Heterogeneous properties Elemental composition Molecular weight Characteristic functional groups Behaviors Transport through soil pores (spatial mobility) Chemical reactions: first order and second order Sorption In our agent-based modeling approach. Different carbon, functional groups, mediums?? Discover emergent properties, global 9/22/2018

Stochastic Synthesis: Data Model Pseudo-Molecule Elemental Functional Structural Composition Calculated Chemical Properties and Reactivity Location Origin State 9/22/2018

The Conceptual Model II Stochastic Model Individual behaviors and interactions are stochastically determined by: Internal attributes Molecular structure State (adsorbed, desorbed, reacted, etc.) External conditions Environment (pH, light intensity, etc.) Proximity to other molecules Length of time step, Δt Space 2D Grid Structure Emergent properties Distribution of molecular properties over time 9/22/2018

Implementations There are five implementations from the conceptual model 9/22/2018

V & V of the NOM Model Examples of V & V techniques Face validity Animation Graphical representation Tracing Internal validity Historical data validation (calibration sets and test sets) Sensitivity analysis Prediction validation Comparison with other models Turing test One author identifies 75 V & V techniques. Some of them are software engineering. We talk about some of them we are going to use. Prediction is strong, avoid use historical data to calibrate and fit the model 9/22/2018

V & V of NOM Simulation Model These steps are not separated with each other. They are through the model development process Conceptual validity test “if the model believable”, just part of the confiendence Simulation verification just test correct implemnt the conceptual model, no bug Compare with the real world Data validity: make sure have right data historical data. Ensure that the data necessary for modeling building, model evaluation and testing, and conducting the model experiments to solve the problem are adequate and correct Right measurements of data Propose the research questions need to be answer, and phenomena need to be modeled. 9/22/2018 *Adapted from Sargent: “Verification and Validation of Simulation Models”

Face Validity Molecule move down, stuck. Verification For model developer debugging , also validation to provide model confidence for users. Cost effectives . Behave reasonably 9/22/2018

Internal Validity I Example of V & V using internal validity, expect the output are not dependent of seed. One type of sensitivity analysis One statement in a wrong place. It take a lot of time to debug the code to get the . Under one type of event, the random seed is unintentional reset 9/22/2018

Internal Validity II We need to investigate what cause this, more test needed. May need large sample. Problems we don’t understand. 9/22/2018

Model-to-Model Comparison I Compare the model with validated one Compare the model with non-validated one Different implementations Different programming languages Different packages Different modeling approaches Predator-Prey model Agent-based approach vs. System Dynamics approach Powerful method for ABS The next step we worked on the model to model comparison. Validate with non-validated one, help the verify if the code has bug or not Called docking Cellular potts: cellular automaton governing cell interaction, along with reaction-diffusion equation solvers to establish surrounding chemical gradients. Grid based stochastic model designed to accurately simulate cell interactions and movement 9/22/2018

Model-to-Model Comparison II Features Alpha Step No-flow Reaction Developing Group University of New Mexico, chemists University of Notre Dame, computer scientists Programming language Pascal Java (Sun JDK 1.4.2) Platforms Delphi 6, Windows Red hat Linux cluster Running mode Standalone Web based, standalone Simulation package None Swarm, Repast libraries Animation Yes Spatial representation 2D grid Second order reaction Random pick one from list Choose the nearest neighbor First order with split Add to list Find empty cell nearby 9/22/2018

Model-to-Model Comparison III Ensemble of 50 runs. These are variables that we choose most interests for users to compare their output. The major purpose of the model is to discover the evlolution of the NOM, the time series are most important 9/22/2018

Model-to-Model Comparison IV 9/22/2018

Model-to-Model Comparison V 9/22/2018

Conclusion and Future Work V & V Case Study Model-to-Model Comparison is Powerful Collect and evaluate more statistical data Compare simulation results against empirical data Tweak V & V methods Generate guidelines and methodology for V & V of agent-based simulation models 9/22/2018

Questions or Comments? 9/22/2018