1. Ms. Lisa Jean Moya WernerAnderson, Inc. 01 May 2007 Validation Methodology for Agent-Based Simulations Workshop DoD Validation Baseline

2. Outline Validation defined General approach Issues for ABS validation

3. Outline Validation defined General approach Issues for ABS validation

4. DODI 5000.61 DoD Definitions Verification  The process of determining that a model implementation and its associated data accurately represents the developer’s conceptual description and specifications Validation  The process of determining the degree to which a model and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model Accreditation  The official certification that a model, simulation, or federation of models and simulations and its associated data are acceptable for use for a specific purpose The workshop focus is Validation

5. Utility of Validation Military analysis requires the capability to evaluate an environment dominated by non- physical effects  Cold War analysis is not sufficient  Fighting the last war is not good enough Subject matter expertise needs codification and expansion Make appropriate use of M&S  Avoid using bad M&S/analysis  Avoid throwing out good M&S/analysis

6. DoD 5000 on M&S VV&A Much attention paid to “principals” but little to “principles”  Provides DoD authoritative definitions  Little emphasis on the “how’s” Policies and procedures for M&S applications at the DoD Component level  Allows the tailoring of VV&A policies and procedures to the needs of the user  Likely to result in inconsistencies – little to no standardization of TTPs

7. Outline Validation defined General approach Issues for ABS validation

8. DMSO, VV&A Recommended Practices Guide – Validation Special Topic Validation Steps Verify M&S requirements Develop V&V plan Validate conceptual model Verify design Verify implementation Validate results Verify M&S requirements Develop V&V plan Validate conceptual model Verify design Verify implementation Validate results

9. General Process Verify M&S requirements Develop V&V plan Validate conceptual model Verify design Verify implementation Validate results Basic representation Effect of interactions Empirical Assessment Another model Mathematical Simulation Formalism Historical event Live experiment SME / Turing Statistical Metric Assessment Appropriate referents Rule set (alone & in the composition) Instantiation Interpretation Trajectory Adapted from DMSO, VV&A Recommended Practices Guide

10. Adapted from DMSO, VV&A Recommended Practices Guide – Requirements Special Topic Overlap Between Domain Areas & Requirements Use cases–e.g., scenario Representation fidelity Mission, enemy, terrain, troops, time Available (METT-T) Behaviors, tactics User Domain Use cases–e.g., scenario Representation fidelity Mission, enemy, terrain, troops, time Available (METT-T)) Behaviors, tactics Simulation Domain Application types–analysis, training, acquisition Physics–laws, forces, systems Representational requirements –Performance & behaviors of real entities Missions, doctrine, operations, rules of engagement/deployment Problem Domain M&S Requirements Real-world based Implement functions & features

11. Finding a Referent Experimental data Empirical data Experience, knowledge, and intuition of SMEs Validated mathematical models Qualitative descriptions Other simulations Combinations of the types described above Conceptual model = Content and internal representations of the M&S; includes logic and algorithms; recognizes assumptions and limitations DMSO, VV&A Recommended Practices Guide – Validation Special Topic

12. Human Behavior Model Referents SMEs Empirical observations or experimental data from actual operations Models of human behavior Models of physiological processes Models of sociological phenomena Simulations of human behavior

13. When a Referent Doesn’t Exist Assemble from known components of the system or procedure Assemble from known basic phenomena underlying the system’s behavior Build a scale model of the system or its components and perform experiments Use the referents for a similar existing system or similar situations DMSO, VV&A Recommended Practices Guide – Validation Special Topic

14. Conceptual Model Components DMSO, VV&A Recommended Practices Guide – Conceptual Model Special Topic The model should be as simple as possible, but not too simple Specifications Conceptual Model Simulation Environment Objects Data/Nouns (Inputs and Outputs) Attributes Resources Behavior states Actions/Activities/Verbs Functions & algorithms that Create/change data Create additional actions Environment Constraints Relationships Geometry Requirements

15. Conceptual Model Analysis Test/analyze component algorithms of overall model to validate each individually  Mathematical analysis  Results of component algorithms should match available data  Increases confidence that interactions of the collected algorithms (i.e., the overall model) are valid Algorithm testing  3rd party program (e.g., Excel)  Should examine a range of data Assumption testing (supplementary or alternative approach)  Determine assumptions (rarely stated) – structural, causal, and mathematical  Identify operational impacts of assumptions relative to intended application  Determine acceptability of operational impacts with Application Sponsor (Accreditation Authority) If they exist, unexpected/emergent interactions should appear in model output However, interactions between algorithms may not be addressed

16. V&V Technique Taxonomy Informal  Determine “reasonableness”  Most commonly used, subjective  Audit, review, face validation, inspection, Turing test Static  Assess accuracy of design  Automated tools available  Analyses: semantic/structural, data/control, interface, traceability Dynamic  Assess model execution  Requires model instrumentation  Tests: acceptance, fault/failure, assertion, execution, regression, predictive validation, structure, sensitivity, statistical Formal  Complex, time consuming  Induction, inference, predicate calculus, proof of correctness How much V&V depends on budgetary considerations, significance of supported decisions, and the risk of inaccuracy. DMSO, VV&A Recommended Practices Guide – V&V Techniques Special Topic

17. 7 Recommended Techniques DA-PAM 5-11, Verification, Validation, and Accreditation of Army M&S Face validation  SME review Comparison to other M&S  Legacy, non-Government, alternative formulation Functional decomposition  Validating the parts, assuming the whole Sensitivity analyses  Run boundary conditions Visualization  Output appears to match intent Turing tests  “If it walks like a duck, …” Modeling-test-model  Anticipate, experiment, refine Each technique has its drawbacks

18. Intuition vs. Data Results match intuitive expectations Dynamic technique Results  SMEs use intuition and estimates of expected behaviors and outputs  Model and system behaviors considered subjectively Best used in early stages of development Issues  Dependent on experience with the system being modeled to provide intuitive expectations  Subject to human error  Difficult to predict unexpected/emergent behaviors based on intuition/experience Results match data from past experience  Historical, exercise, other models Dynamic technique Reasonable results  Predictive validation – results provide a reasonable prediction of subsequent real- world behavior/results  Historical/exercise/model data should generate outputs similar to associated results Models should be consistent  Multiple models for the same system should produce the “same” results from the “same” data  Systematic biases will not be detected

19. Outline Validation defined General approach Issues for ABS validation

20. DMSO, VV&A Recommended Practices Guide – Human Behavioral Representation (HBR) Special Topic Moya & Tolk, Toward a Taxonomy of Agents & MAS Agent Validation Evaluate  Conceptual model design  Knowledge Base  Engine and Knowledge Base implementation  Integration with simulation environment

21. Agent System Validation Moya & Tolk, Toward a Taxonomy of Agents & MAS Effect of parameter settings and system/agent instantiations (ranges, settings, interpretations, rules)  Interactions  Overall results

22. Areas Affecting HBR Validity Interactions between multiple behaviors  Assumes that interacting nonlinear behaviors will create even more convoluted nonlinear behavior Dependencies between properties in the behavior space Sensitivities between behavior space property changes Nonlinear behavior  Errors can hide or be misinterpreted Nonlinear component behavior transitions Complex environmental interactions Stochastic behaviors  Probabilistic sensing DMSO, VV&A Recommended Practices Guide – HBR Special Topic