Irregular Warfare Modeling/Data Validation Best Practices 11 th Annual MOVES Research Education Summit 12 July 2011

Purpose and Agenda 2 Purpose: To discuss the framework developed at TRAC-MTRY/NPS toward conducting validation of Methods, Models, and Tools (MMT) that inform analysis of Irregular Warfare (IW) and to overview ongoing development of a similar data validation framework. 12 July th MOVES Research and Education Summit Overview and Objectives. Validation Project Teams. Project Descriptions and Results. −IW Model Validation Best Practices. −IW Data Validation Best Practices. Summary and Way Ahead. Questions.

Overview and Objectives 12 July th MOVES Research and Education Summit TRAC-MTRY and Naval Postgraduate School. –IW Modeling Validation Best Practices Guide, FY09-FY10. -Identify and develop a best practices for verifying and validating social science theory and modeling required for tactical level IW MMT. –Validating Data that Supports IW Models, FY10-FY11. -Assess, at a minimum, the validation of IW data, to include an examination of data requirements, data sources, and data availability as well as derivation of data. Objective: To develop methodologies/best practices toward validating methods, models and tools (MMT) and associated data that support IW analysis, using the CG model for implementation of these practices.

Validation Project Teams 12 July th MOVES Research and Education Summit FY09/FY10 TRAC-MTRY/NPS. –CPT Richard F. Brown, Project Lead. –Dr. Jeffrey Appleget, Professor, OR Department. –Dr. Deborah Duong, Senior Analyst, Augustine Consulting, Inc. –Mr. Curtis Blais, Professor, MOVES Institute. –Mr. Timothy K. Perkins, ERDC. –Ms. Meredith Thompson, SEED Center. FY10/FY11 TRAC-MTRY/NPS. –MAJ Joe Vargas, Project Lead. –CPT Richard F. Brown, Project Support. –Dr. Jeffrey Appleget, Professor, OR Department. –Mr. Curtis Blais, Professor, MOVES Institute. –Dr. Michael Jaye, Professor, Defense Analysis Department. –Dr. Eric Weisel, CEO, WernerAnderson, Inc. –COL Robert Burks, Senior Military Advisor, OR Department. Each team represents efforts that focus on developing frameworks and methodologies toward validating models, validating data, and making assessments of and building documentation toward validation of CG.

Best Practices Guide Approach and Requirements Requirement 1. Provide a report that identifies validation best practices for IW MMT tactical-level representation. Requirement 2. Apply the BPG developed in Requirement 1 to both relevant MMT in order to demonstrate usability of the BPG framework. Best Practice Guide Approach. –Conducted an extensive literature review of prevalent validation literature, to include the DOD Recommended Best Practices, MIL-STD-3022, SISO General Methodology for VV&A, and others (Chapter 2). –Synthesized literature and described the challenges of modeling the IW environment and validating the models that represent the IW environment (Chapter 3). –Constructed a validation framework toward model validation. The framework includes detailed descriptions of the principal areas of the framework that must be identified to conduct proper validation (Chapter 4). –Apply the BPG, as appropriate, to relevant MMT; in particular, the CG model. 12 July th MOVES Research and Education Summit

Validation Framework Concept Map “The Golden Triangle” This effort resulted in the construction of the following model validation and development framework. The components of the “Golden Triangle” make up the crux of model validation and establish the critical basis of comparison to the real world. Validation Framework Concept Map. User Needs –The developer needs to obtain a succinct and clear statement of the problem the M&S is expected to address. Requirements –Develop specific functional or quality statements that can be directly and explicitly assessed to determine requirement. Acceptability Criteria –Develop a requirements traceability matrix relating each specified requirement with acceptability criteria applicable to the intended use. Results –The acceptability criteria identify what the model needs to do to satisfy or meet the set of respective requirements pertinent to the intended use. Data –The greater the specificity in the data requirements for a model, the greater the ability to collect the data needed to populate the model. Intended Use. –Obtain a clear, succinct statement of intended use from the user representatives. Simuland. –The simuland is the real- world system of interest, including the objects, processes, or phenomena to be simulated. Referent. –Identify the social science theory (or theories, if multiple competing theories will be represented in the model for comparison) that explains that phenomena. Conceptual Model. –Develop the conceptual model using tools and techniques that create machine-readable specifications of the data and logic of the model. Executable Model. –Design the model implementation to be as transparent as possible to permit analysis of execution paths and computed outcomes 12 July th MOVES Research and Education Summit

IW Data Validation Best Practices 12 July th MOVES Research and Education Summit WORKING OUTLINE Introduction. Literature Review. –Review validation standard and practices from DOD and related sources that add value to DoD MMT. Data Supporting Irregular Warfare Modeling. –Describe IW-type data and highlight the challenges in validating this data. IW Data Sources. –Characterize sources for IW data. Best Practices for Validating IW Data. –Explain the IW data validation framework, to include a description of methods toward validating IW data, what metadata must be included with data sets, and other significant insights. Summary and Recommendations. Appendix. –Develop IW data validation framework use-case using relevant MMT, in particular, the CG Model. Requirement: Provide a report that assesses, at a minimum, the validation of IW data, to include an examination of data requirements, data sources, and data availability as well as the derivation of data.

Summary and Way Ahead 12 July th MOVES Research and Education Summit Summary. –Multi-disciplinary team developed a BPG that provides a useful framework that a validation agent could use for validating a IW model, or that a developer could use to properly develop an IW model. –Model validation best practices incorporate concepts that developers are aware of, yet many of the concepts must be enumerated explicitly to ensure validation is possible. –“Validation” may not be a proper term with respect to data. A paradigm shift may be in order. –Any type of validation, whether model or data, must be related to a specified and/or intended use. Way Ahead. –TRADOC Analysis Center (TRAC) Irregular Warfare Modeling and Simulation Summit provided a considerable forum for discussion and actions pertaining to these issues. The validation team will work closely with the TRAC Complex Operations Data Development Activity (CODDA) to develop and vet the data validation best practices, and ensure that is codified sufficiently for use in the analytic community.

9 Comments / Discussion 12 July th MOVES Research and Education Summit

CG Model Assessment Requirements 12 July th MOVES Research and Education Summit Assess the appropriateness of the CG model representation of the Conflict Ecosystem conceptual model, and whether the model referent provided appropriate underpinning to account for human socio-cultural behavior. Assess the appropriateness of the Social Network Representation within the CG Model and the whether this representation is generalizable to conflict ecosystems in any region. Assess the appropriateness of the representation of social science theory within the CG Model, specifically tracing the implementation of the Theory of Planned Behavior. Develop risk measures for using a model or simulation for a specified application. Develop criteria for assessing the risk of using a model or simulation based on consequence, error, and validation process. Characterize an approach toward validating the CG model based on the methodologies/insights/lessons learned from the tasks developed above.