1. Verification and Validation of Agent-based Scientific Simulations
Xiaorong Xiang, Ryan Kennedy, Gregory Madey
Computer Science and Engineering
University of Notre Dame
Steve Cabaniss
Department of Chemistry
University of New Mexico
9/22/2018

2. Overview Introduction A Case Study
Concepts of Verification and Validation
Research Objectives and Methods
A Case Study
Apply Verification and Validation Methods to the Case Study
Conclusion
Future Work
9/22/2018

3. Model Verification & Validation (V & V)
Verification: get model right
Validation: get right model
The cost and value influence confidence of model acceptance level
Model verification is a process determining if the implmentation of model have sufficient accuracy
Model validation is a process determing if the model represent the real world phenomena
The purpose of V & V is t o increase the model confidence. According to the relationships of cost, value and model confidence, it is critical to find most cost-effictive way for conducting the process , the verification and validation process are conducted until sufficient confidence is obtained
The purpose of V & V is to increase confidence of the model. Users’ belief of the model is high. How good and useful the model for users. V&V can be go forever, need to decide when should stop the V & V process. Get the highest model confidence with low cost. Find optimal approach for V & V of ABS. DES is the most closely approach as ABS.
It is too costly and time consuming to determine that a model is absolutely valid to be used practically. Instead, the verification and validation process are conducted until sufficient confidence is obtained
Model verification: the process of determining whether a computational model of a physical event and the code implementing the model represent the model with sufficient accuracy
Model validation: the process of determining whether a mathematical model of a physical event represents the actual event with sufficient accuracy
*Adapted from Sargent: “Verification and
Validation of Simulation Models”
9/22/2018

4. V & V for Agent-based Simulation
Agent-based modeling is a new approach
Different than Queuing Models
Entities: large number of heterogeneous active objects vs. passive objects
Space: continuous or discrete grid space vs. network of servers and queues
Interactivity: high vs. low
Active components: agents vs. queues and servers
Goal: discovery vs. design and optimization
Few literature to date address the formalized methodology for V & V of Agent-based Simulations
Agent-based modeling is only 10 years old, and recently become popular for modeling the complex system with dynamic behaviors. Closest to DES with the following differences. Ant, bees. Behave independently, and a set of behavioural rules, the global behavior of the system is can discovered
ABS focus on systems that contain large numbers of active objects (people, business units, animals, vehicles, and time events ordering or other kind of invididual behavior associated with them
No global system behaviors (dynamics) would be defined, the modelers defines behavior at individual level, ,and the global behavior emerges as a results of many individuals, each following its own behaviors rules, living together in some environment and communicating with each other and with the environment
9/22/2018

5. What and How Research objective How
Generate guidelines or a formalized methodology for V & V of Agent-based Simulations
How
NOM project as a case study
Evaluate and adapt the formalized V & V techniques in industrial and system engineering for DES
Identify a subset of these techniques that are more cost-effective for Agent-based Simulations
9/22/2018

6. NOM Agent-based Simulation Model
NSF funded interdisciplinary project
Understanding the evolution and heterogeneous structure of Natural Organic Matter (NOM)
E-science example
Chemists, biologists, ecologists, and computer scientists
Agent-based stochastic model
Web-based simulation model
9/22/2018

7. NOM What is NOM? Why study NOM?
Heterogeneous mixture of molecules in terrestrial and aquatic ecosystems
Why study NOM?
Plays a crucial role in the evolution of soils, the transport of pollutants, and the global carbon cycle
Understanding NOM helps us better understand natural ecosystems
9/22/2018

8. The Conceptual Model I Agents A large number of molecules Behaviors
Heterogeneous properties
Elemental composition
Molecular weight
Characteristic functional groups
Behaviors
Transport through soil pores (spatial mobility)
Chemical reactions: first order and second order
Sorption
In our agent-based modeling approach. Different carbon, functional groups, mediums?? Discover emergent properties, global
9/22/2018

9. Stochastic Synthesis: Data Model
Pseudo-Molecule
Elemental
Functional
Structural
Composition
Calculated
Chemical
Properties
and Reactivity
Location
Origin
State
9/22/2018

10. The Conceptual Model II
Stochastic Model
Individual behaviors and interactions are stochastically determined by:
Internal attributes
Molecular structure
State (adsorbed, desorbed, reacted, etc.)
External conditions
Environment (pH, light intensity, etc.)
Proximity to other molecules
Length of time step, Δt
Space
2D Grid Structure
Emergent properties
Distribution of molecular properties over time
9/22/2018

11. Implementations
There are five implementations from the conceptual model
9/22/2018

12. V & V of the NOM Model Examples of V & V techniques Face validity
Animation
Graphical representation
Tracing
Internal validity
Historical data validation (calibration sets and test sets)
Sensitivity analysis
Prediction validation
Comparison with other models
Turing test
One author identifies 75 V & V techniques. Some of them are software engineering. We talk about some of them we are going to use. Prediction is strong, avoid use historical data to calibrate and fit the model
9/22/2018

13. V & V of NOM Simulation Model
These steps are not separated with each other. They are through the model development process
Conceptual validity test “if the model believable”, just part of the confiendence
Simulation verification just test correct implemnt the conceptual model, no bug
Compare with the real world
Data validity: make sure have right data historical data. Ensure that the data necessary for modeling building, model evaluation and testing, and conducting the model experiments to solve the problem are adequate and correct
Right measurements of data
Propose the research questions need to be answer, and phenomena need to be modeled.
9/22/2018
*Adapted from Sargent: “Verification and Validation of Simulation Models”

14. Face Validity
Molecule move down, stuck. Verification For model developer debugging , also validation to provide model confidence for users. Cost effectives . Behave reasonably
9/22/2018

15. Internal Validity I
Example of V & V using internal validity, expect the output are not dependent of seed. One type of sensitivity analysis
One statement in a wrong place. It take a lot of time to debug the code to get the . Under one type of event, the random seed is unintentional reset
9/22/2018

16. Internal Validity II
We need to investigate what cause this, more test needed. May need large sample. Problems we don’t understand.
9/22/2018

17. Model-to-Model Comparison I
Compare the model with validated one
Compare the model with non-validated one
Different implementations
Different programming languages
Different packages
Different modeling approaches
Predator-Prey model
Agent-based approach vs. System Dynamics approach
Powerful method for ABS
The next step we worked on the model to model comparison. Validate with non-validated one, help the verify if the code has bug or not
Called docking
Cellular potts: cellular automaton governing cell interaction, along with reaction-diffusion equation solvers to establish surrounding chemical gradients. Grid based stochastic model designed to accurately simulate cell interactions and movement
9/22/2018

18. Model-to-Model Comparison II
Features
Alpha Step
No-flow Reaction
Developing Group
University of New Mexico, chemists
University of Notre Dame, computer scientists
Programming language
Pascal
Java (Sun JDK 1.4.2)
Platforms
Delphi 6, Windows
Red hat Linux cluster
Running mode
Standalone
Web based, standalone
Simulation package
None
Swarm, Repast libraries
Animation
Yes
Spatial representation
2D grid
Second order reaction
Random pick one from list
Choose the nearest neighbor
First order with split
Add to list
Find empty cell nearby
9/22/2018

19. Model-to-Model Comparison III
Ensemble of 50 runs. These are variables that we choose most interests for users to compare their output. The major purpose of the model is to discover the evlolution of the NOM, the time series are most important
9/22/2018

20. Model-to-Model Comparison IV
9/22/2018

21. Model-to-Model Comparison V
9/22/2018

22. Conclusion and Future Work
V & V Case Study
Model-to-Model Comparison is Powerful
Collect and evaluate more statistical data
Compare simulation results against empirical data
Tweak V & V methods
Generate guidelines and methodology for V & V of agent-based simulation models
9/22/2018

23. Questions or Comments?
9/22/2018