Applying Modeling and Simulation Verification, Validation and Accreditation (VV&A) Techniques to Test and Laboratory Facilities Dr. James Elele, Jeremy Smith NAWCAD BSMVV Branch James.Elele@navy.mil David Hall, Charles Pedriani SURVICE Engineering Company Dave.Hall@survice.com ASME V&V Conference 3 May 2012

Introduction Tasking to support accreditation of Test and Evaluation Facilities in support of IFF Program Develop an accreditation case for T&E facilities for operational testing Applied risk-based M&S VV&A approach to facilities Approach applied successfully to M&S for over 20 years Test case for future T&E facility accreditation efforts Successful application can support efforts to institutionalize process for T&E as well as M&S IFF = Identification Friend or Foe T&E = Test and Evaluation M&S = Model and Simulation

M&S VV&A Definitions Verification: The process of determining that a model implementation and its associated data accurately represent the developer's conceptual description and specifications. Does the model do what the originator intended, and is it relatively error free? Validation: The process of determining the degree to which a model and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model. How well do model results match real world data, in the context of your needs? Accreditation: The official certification [determination] that a model, simulation, or federation of models and simulations and its associated data are acceptable for use for a specific purpose Does the accreditation authority have adequate evidence to be confident that a model is fit for purpose? Did you build the model right? Did you build the right model? Did your customer accept it? Definitions from DODI 5000.61 dated 13 May 2003

Underlying Principles The ultimate goal of VV&A efforts is to form a foundation for making good decisions Nature and extent of information required to support accreditation decision is at the discretion of the accreditation authority and is generally based on assessment of risk Role of M&S results in decision making process Importance of decision that M&S is supporting Severity of the consequences of making incorrect decisions because M&S were wrong Probability that analysis results based upon M&S will be challenged Best of Show Good Enough “Better is the Enemy of Good Enough!”

Steps to an Accreditation Decision Analyze Intended Use Intended Use Statement Develop M&S Reqts and Accreditation Info Reqts Accreditation Plan Develop/Execute Accreditation Plan V&V Develop Accreditation Case A Review Accreditation Case Fit for Intended Use Make Accreditation Decision

How Much Credibility Is “Enough”? It Depends on Risk M&S User A Makeshift Bridge is Good Enough If You Need To Cross a Meandering Shallow Stream M&S In attempting to make an accreditation assessment, most assessors face the common question of “how much of this V&V information is necessary to demonstrate credibility for a particular application?”. The answer to this questions depends in large part on risk. If the consequences of using an inaccurate simulation are not severe, the risk is relatively low and one would need much less evidence of simulation credibility. Just as in this picture, a makeshift structure of evidence may be satisfactory for low risk applications. BUT

Evidence of Greater Credibility Greater Risks... Indicate the Need for Evidence of Greater Credibility M&S Supporting Evidence However, in those cases where inaccurate simulation predictions could have disastrous consequences, one needs much more evidence to guarantee that the simulation is credible enough to reduce the risks. For example, the simulations that are being used to prove that the ballistic missile defense system will work under all circumstances must have a lot of credibility to justify the expenditure of funds on so large and important a system. The risks associated with erroneous performance predictions and a potentially inadequate missile umbrella would be extremely high. In these cases, the User should demand a comprehensive accreditation assessment that is supported by a strong foundation of extensive V&V results. In these situations, the case supporting the accreditation needs to be fully documented and reviewed. The guiding principle is that quantity and quality of the evidence is directly related to the level of risk of the intended use. PROBLEM CREDIBLE SOLUTION

V&V: The Central Pillars of Simulation Credibility S/W Accuracy Data Accuracy Output Accuracy Simulation meets design requirements, operates as designed and is free of errors in software Simulation input data, validation data and data manipulations are appropriate and accurate Simulation outputs match the real world “well enough” to be of use in a particular problem Software (S/W) Accuracy Data Accuracy Output Accuracy Verification Validation Approaching the question of what makes a simulation credible from an abstract perspective, one can think of a simulation as the bridge. To build confidence that the bridge is adequate, one would assess the capabilities and characteristics of the bridge in light of the requirements for the bridge. Typically, one would review and evaluate some combination of design information, engineering certifications, test reports, builder qualifications, maintenance records, etc. to determine if the bridge was adequate for the type and level of traffic that was expected. In an analogous way, we can assess the credibility of a simulation for a specific application based on knowledge of its design, the accuracy of its software (verification results), the accuracy of its outputs (validation or test results), the accuracy of its input data, its usage history, etc.. Most people think of simulation credibility in terms of three types of accuracy: software accuracy, data accuracy and output accuracy. Software accuracy refers to the error-freeness of software design and coding; data accuracy refers to the accuracy and appropriateness of input data to the simulation; output accuracy refers to the degree to which simulation outputs match the real world. These aspects of simulation credibility are easily recognized as the objects of verification and validation activities. These elements give one an understanding of simulation accuracy. However, these elements, alone, do not comprise a comprehensive assessment of simulation credibility? "V" & "V" But, V&V are Just the Middle of the Bridge!

The “Other Pillars” of Simulation Credibility Capability Usability Simulation possesses all required functionality and fidelity for the problem being solved Anchors the M&S to the Problem Simulation has adequate user support to facilitate correct operation and interpretation of its outputs Ties the M&S to a Useful Solution M&S Capability Accuracy Usability M&S Requirements User Capabilities Our experience suggests that simulation accuracy (as defined in the previous slide) is a necessary but not a sufficient condition for a robust determination of simulation credibility. The software may be demonstrably free of errors, the data may be demonstrably accurate and correct, and simulation outputs may correlate well with real world data. But does the simulation represent all the real world phenomena that affect the problem you are trying to solve? Does it simulate these functions at the appropriate level of detail? Is the simulation (software, documentation and data) managed well? Is it accompanied by enough support structure to allow you to use it credibly, given the user’s level of experience with the simulation? Questions like these provide the approaches to our simulation “bridge” and provide the linkage between the bridge and the anchor points on the banks of the river that must be crossed. Any robust assessment of simulation credibility must consider not only “accuracy”, but also “capability” and “usability” in determining whether it is suitable for use in a particular application. Any accreditation assessment that is evaluating simulation credibility for a given application must consider all five of these credibility elements. Each of these five elements, capability, software accuracy, data accuracy, output accuracy, and usability, will be discussed individually in the following sections of the tutorial. Credible Solution Problem Accuracy of: Software Data Outputs Can I Be Sure I’m Not Mis-Using the M&S? Does the M&S Do What I Need It To Do?

The Essence of Accreditation M&S REQUIREMENTS IDENTIFY M&S DEFICIENCIES M&S INFORMATION Capability Accuracy Usability Data Quality M&S Documentation Design Documentation Configuration Mgt V&V Results Etc. IDENTIFY WORK-AROUNDS, USAGE CONSTRAINTS, REQUIRED IMPROVEMENTS AND RISKS Provided by the Model Developer or Model Proponent Defined by the User (Formally or Implied) ACCREDITATION DECISION Accreditation and the associated accreditation assessment have been discussed. However, it is important to understand just what accreditation entails. This diagram shows a simplified view of what accreditation really is and how it is typically done. Within the DoD, accreditation is defined as the official determination that a model is acceptable for a specific purpose. In practical terms, this definition implies that accreditation depends on a comparison of a model’s capabilities, limitations, and attributes with the simulation requirements that are generated from the specific problem for which the model will be used. To make such a judgment about model suitability, one must have a complete set of simulation requirements. These requirements are derived from the unique problem which the User is trying to solve. In addition to the simulation requirements, one must also have documented evidence of the simulation’s capabilities, limitations, and attributes. Typically, this information is found in the sources indicated. A key part of this information is the evidence of data quality. This evidence is quite frequently obtained from sources completely separated from the Model Manager. In making the assessment, one usually cannot be satisfied with a simple “no” answer if the capability, accuracy, or usability of the simulation does not meet requirements. In these cases one must analyze the impacts of any deficiencies, determine if any work-arounds exist, and evaluate these against the requirements. Such an analysis should yield a list of tasks that must be done to make the simulation acceptable. PROBLEM CONTEXT TO PROVE THE M&S IS SUITABLE FOR THE NEED REQUIRES AN OBJECTIVE COMPARISON OF M&S INFORMATION WITH M&S REQUIREMENTS WITHIN THE CONTEXT OF THE PROBLEM

How Much V&V is Enough? It Depends on Risk RISK = PROBABILITY x IMPACT Risk means something “bad” might happen because you believed an incorrect simulation result Decisions based on M&S results are at risk VV&A reduces that risk RISK HIGH LOW MODERATE “How much credibility is enough” can only be answered in the context of the application. That is, the answer to that question depends on what decisions you’re going to make using model outputs, and what the risks are if you make the wrong decision. If you have a very high risk application, such as decisions that might involve life or death situations (such as embedded flight-critical software in an aircraft), then the potential risk of a wrong decision is very high. In that case, the model must have very high credibility, which will require a high level of capability, accuracy and usability, and a high level of verification and validation activity to demonstrate that credibility. If the risk is low, then less is required to demonstrate that the model is suitable for your application. JASA has developed an Accreditation Information Requirements Guide (AIRGuide), which can be used to help assess the risks inherent in the application of models and simulations, and which guides the development of a V&V plan to mitigate those risks. The AIRGuide process is based on assessing both the impact of wrong decisions made with incorrect model outputs and the probability of a wrong decision being made. Risk is defined as the product of impact and probability. More information on the AIRGuide is available in the handouts for the tutorial. PROBABILITY IMPACT

Quantifying Risk Level Probability of Incorrect Simulation Result Level of Impact on the Program Negligible Marginal Serious Critical Catastrophic Frequent Low Moderate High Probable Occasional Remote Impossible Once the User has quantified the probabilities and impact levels, these can be combined to arrive at an overall level of risk (for each individual risk factor that is identified). This table is one of several contained in the MIL-STD. It again is an example that can be tailored to the needs of the individual application. This type of table allows the User to determine the level of risk based on a level of impact and probability of each risk factor. If there are multiple risk factors associated with a particular application, each risk factor should be analyzed individually. (For example, if a simulation is being used to to determine the footprint of the missile, one risk factor is the chance of hitting an observer on the range if the predicted footprint is too small. Another is the likelihood of wasting resources to clear too large a test area because the predicted footprint was too large.) The highest level risk, considering the risk of each factor, will determine the level of credibility that is needed in the simulation being used. RISK LEVEL VALUES ARE: Subjective Consistent with MIL-STD-882 Tailorable to each application HIGHER RISK MEANS MORE CREDIBILITY EVIDENCE IS NEEDED TO ACHIEVE ACCREDITATION

Risk Reduction Strategies 5 4 Likelihood Improved M&S Credibility 3 2 1 1 2 3 4 5 Reduced reliance on M&S results Impact Low Moderate High Risk = Likelihood x Impact

Application to Test and Laboratory Facilities This was a trial application of the M&S VV&A approach to test facilities Identification Friend-or-Foe (IFF) system testing to evaluate new IFF system performance Accreditation of test facilities required by Commander, Operational Test and Evaluation Force (COMOPTEVFOR) Facilities used for system assessment include: All-up ship radar and related system representations Simple stimulators Engineering Test Equipment (ETE) facility specific to IFF system testing

ETE Facility Generates waveform signals to stimulate a production IFF transponder in the laboratory To evaluate system requirement for resistance to signals from transmitters other than the desired transmitter Critical technical parameter “susceptibility of the IFF system to false interrogations” Metric: Probability of resistance to false signals = Pr

Initial ETE Risk Assessment M&S Characteristic Criterion Risk CAPABILITY Intended Use The specific intended use(s) of the facility, model or simulation is/are clearly stated. LOW Design The facility and analysis process (framework, algorithms, data sources, and assumptions) produces credible results. MODERATE ACCURACY Input Data For each facility, model or simulation, input data are credible and subject to review and revision. System Verification The facility, model or simulation has been formally tested or reviewed and has been demonstrated to accurately represent the specific intended use(s) and requirements. Results Validation The facility’s, model’s or simulation’s responses have been compared with known or expected behavior from the subject it represents and has been demonstrated to be sufficiently accurate for the specific intended use(s). USABILITY Configuration Management For each facility, model or simulation, modeled components are supported by a sound written Configuration Management (CM) Plan. User Community For each facility, model or simulation, the capability is designed and developed for the level of competency for its intended purpose. The capability is supported by documents such as user’s manual, technical manual, and/or reference guide.

Initial ETE Assessment and Recommendations* CAPABILITY: The intended use is clearly stated: to evaluate the probability of responding to a false signal (Pr) No formal design documents exist: recommend laboratory design be adequately documented ACCURACY: Input data are provided by actual hardware Recommend documenting a complete set of test cases and results and any previous verification activities Recommend independent subject matter expert (SME) review of laboratory results USABILITY: Recommend facility develop and implement an overall configuration management plan The test approach appears to have been successfully used over a span of many years to support a variety of identification programs for DOD and the FAA: recommend the facility provide documented results of previous uses * Incorporated into ETE Facility Accreditation Plan

Observations Application to test facility similar to M&S Differences: Biggest issue for both: getting good documentation Lack of configuration management plans Poor documentation of prior V&V results They did V&V, they just didn’t write it down Accreditation Support Package (ASP) document format works equally well for both Differences: Developing intended use statement more natural for test facilities than for M&S Use of T&E facilities seems to have more focused objectives initially than use of M&S

Accreditation Support Package ACCURACY Software Accuracy Software Verification Results Design Verification Implementation Verification Software Development and Management Environment Software Development Environment Configuration Management Software Quality Assessment Implications for M&S Use Data Accuracy Input Data Data Transformations Output Accuracy Sensitivity Analysis Benchmarking Face Validation Results Validation EXECUTIVE SUMMARY ASP OVERVIEW CAPABILITY M&S Description Functional Capabilities Development History Assumptions and Limitations Implications for M&S Use USABILITY Documentation Assessment User Support Usage History

Thoughts on Broader Application Risk-based approach seems as applicable to test facilities as to M&S Risk assessment can also help prioritize which facilities justify spending more VV&A resources Suggest standardizing and institutionalizing risk-based VV&A process for both M&S and T&E No consistent application across DOD for either Risk-based VV&A promotes cost-effective VV&A for both M&S and T&E facilities