1. VV&A in Human, Social, Cultural Behavior (HSCB) Simulation Dr. Jimmie McEver Dr. David T. Signori Dr. Mike Smeltzer Evidence Based Research, Inc. mcever@ebrinc.com mcever@ebrinc.com +1 703.287.0374 Based on work funded by DARPA VV&A of COMPOEX Validating Large Scale Simulation of Socio-Political Phenomena (SBIR)

2. 2 What is VV&A? VV&A: Verification, validation and accreditation (of models and simulations) –Verification: The process of determining that a model implementation and its associated data accurately represent the developer's conceptual description and specifications –Validation: The process of determining the degree to which a model and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model –Accreditation: The official certification that a model, simulation, or federation of models and simulations and its associated data are acceptable for use for a specific purpose. U.S. DoD Modeling and Simulation Coordinating Office (MSCO) Glossary

3. 3 VV&A Objective and Reality The stated objective of VV&A is to minimize the risk associated with the use of a model or simulation –Ensure that avoidable errors are eliminated –Signal that a model or simulation is suited for its intended use VV&A activities are most rigorously conducted during the model development –Even then, VV&A is often poorly resourced and underemphasized Despite this, VV&A is often seen as a universal “seal of approval” that a model is accurate and can be trusted –Models and simulations taken off the shelf for later use may undergo some review to examine applicability, but seldom are extensively re-validated for the new use

4. 4 Sources of Risk in M & S Real World Proxy for or Perception of Real World Data Conceptual Model Coded Model Hard Data Soft Data CM Creation Model Creation Model Verification CM Validation Data Use Data V&V Data Verification Data V&V Model Validation Risk Model Use Risk Theory, Data & SME Validation Most PMESII values are not observable

5. 5 Complex M&S Challenges for VV&A The simulation is usually complex –Very large code size –Behavior not decomposable into independent code elements –Simulation behavior results from interactions between elements of code during run-time –Massively multi-dimensional variable space The system being simulation is complex –History only one iteration of what could have happened –Phenomena not being modeled affect system behavior –Data collection needs are massive Different circumstances may require complete revalidation –Not possible to validate over the full range of possible configurations and uses Usually, no one entity was responsible for development –Individual components, validated separately, are integrated into a larger simulation –Integration of the components is a model in itself that is often not validated at all Almost impossible for new users to pick up a model and make good judgments about appropriate use

6. 6 Zack’s Forms of Knowledge Ignorance Source: Zack, M.H., “Managing Organizational Ignorance,” Knowledge Directions, Vol. 1, Summer 1999, pp. 36-49; cited in Leedom, D., “Knowledge Representations for Military C2 Teams and Organizations”, Final Working Draft, March 2004.

7. 7 Rethinking VV&A of Complex Simulations Traditional approaches to VV&A are poorly suited to HSCB simulations (and generally to simulations of complex phenomena) Need to recognize limitations and shift emphasis of VV&A activities –Away from eliminating risk of use –Toward eliminating risk that is possible and characterizing residual risk in ways that allow users to make good decisions about Whether to use the simulation How to use the simulation, while being aware of the risks involved Need methods and tools to improve the transparency of the simulation to the user and establish the credibility of a simulation for a given use

8. 8 Characterization and Mitigation of Risk Associated with Simulation Use Risk can be mitigated by: Reducing uncertainty in the simulation Refining the model Improving the data Applying the simulation in ways consistent with degree of validity and appropriate level of risk to be assumed Notional for a given simulation and situation Potential consequences of simulation uncertainty (varies by type of use) Uncertainty within simulation Inform thinking Predictive COA selection Relative COA assessment Surprise avoidance Possible alternative futures Low Med Low Med High Low Med High Med High Med High

9. 9 Building Credibility of Social Science Models Establishing face validity of a model is a critical step in building credibility among users –Need to understand how the model behaves –Determine whether the model behaves sensibly –How to ID expertise? Users check to see if the model responds to input changes in reasonable and useful ways –Matches user judgment of what should happen –Prompts user to rethink his/her mental model in light of the simulation’s description of what’s going on Attempting to establish face validity for large complex SS models can be challenging –Identifying which variables should be tested –Understanding the cause of the model output Need Framework and Tools to Guide and Support Users

10. 10 HSCB Simulations Have Many User Communities User as a collective term refers to three different user classes in this briefing –Consumers: people and possibly processes that run simulations and interpret the results –R&D Managers: Individuals responsible for managing the development of a simulation model and ensuring that the product is meaningful and useful. –Developers: Model builders responsible for the construction of a simulation model under the guidance of R&D managers and/or Subject Matter Experts Important to reflect on all three classes

11. 11 Simulation Modelers May be: –M&S experts –Social scientists –R&D managers –Software engineers –Smart people –Others Often the models integrate components from multiple complex domains Sometimes the SMEs are involved; sometimes not Users are rarely involved; often after the fact

12. 12 Additional Complicating Factors Modelers make lots of assumptions and create very complicated black boxes that model even more complex dynamic systems Example: COMPOEX –Around 5000 simulation variables for one exercise –1000’s of equations –12,000 – 19,000 factors total M&S sometimes goes beyond the science Are there scientific techniques that can aid users (consumers, managers, modelers, SMEs) –Allow users to explore credibility of model –Inform users on whether use of model is appropriate for the user’s purpose –Facilitate users decisions on how to use the simulation

13. 13 Questions How can we extract and communicate representations of the simulation behavior? –At the right level of accuracy and detail to enable understanding of what’s going on (i.e., suitable for use) –Are cognitively manageable (how do we help people manage complex information/knowledge?) –Sensitive to/captures temporal dynamics –Help users avoid oversimplification –Capture relevant interdependence between factors Generation of phase spaces? –Interactive, computer assisted rather than fully automated Use of correlation matrices? –Computational limitations Time-phased correlation matrices?