1. Applying Modeling and Simulation Verification, Validation and Accreditation (VV&A) Techniques to Test and Laboratory Facilities
Dr. James Elele, Jeremy Smith NAWCAD BSMVV Branch
David Hall, Charles Pedriani
SURVICE Engineering Company
ASME V&V Conference
3 May 2012

2. Introduction
Tasking to support accreditation of Test and Evaluation Facilities in support of IFF Program
Develop an accreditation case for T&E facilities for operational testing
Applied risk-based M&S VV&A approach to facilities
Approach applied successfully to M&S for over 20 years
Test case for future T&E facility accreditation efforts
Successful application can support efforts to institutionalize process for T&E as well as M&S
IFF = Identification Friend or Foe
T&E = Test and Evaluation
M&S = Model and Simulation

3. M&S VV&A Definitions
Verification: The process of determining that a model implementation and its associated data accurately represent the developer's conceptual description and specifications.
Does the model do what the originator intended, and is it relatively error free?
Validation: The process of determining the degree to which a model and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model.
How well do model results match real world data, in the context of your needs?
Accreditation: The official certification [determination] that a model, simulation, or federation of models and simulations and its associated data are acceptable for use for a specific purpose
Does the accreditation authority have adequate evidence to be confident that a model is fit for purpose?
Did you build the model right?
Did you build the right model?
Did your customer accept it?
Definitions from DODI dated 13 May 2003

4. Underlying Principles
The ultimate goal of VV&A efforts is to form a foundation for making good decisions
Nature and extent of information required to support accreditation decision is at the discretion of the accreditation authority and is generally based on assessment of risk
Role of M&S results in decision making process
Importance of decision that M&S is supporting
Severity of the consequences of making incorrect decisions because M&S were wrong
Probability that analysis results based upon M&S will be challenged
Best of
Show
Good
Enough
“Better is the Enemy of Good Enough!”

5. Steps to an Accreditation Decision
Analyze Intended Use
Intended Use Statement
Develop M&S Reqts and Accreditation Info Reqts
Accreditation Plan
Develop/Execute Accreditation Plan
V&V
Develop Accreditation Case A
Review Accreditation Case
Fit for Intended Use
Make Accreditation Decision

6. How Much Credibility Is “Enough”?
It Depends on Risk
M&S User
A Makeshift Bridge is Good Enough If You Need To Cross a Meandering Shallow Stream
M&S
In attempting to make an accreditation assessment, most assessors face the common question of “how much of this V&V information is necessary to demonstrate credibility for a particular application?”. The answer to this questions depends in large part on risk. If the consequences of using an inaccurate simulation are not severe, the risk is relatively low and one would need much less evidence of simulation credibility.
Just as in this picture, a makeshift structure of evidence may be satisfactory for low risk applications.
BUT

7. Evidence of Greater Credibility
Greater Risks...
Indicate the Need for
Evidence of Greater Credibility
M&S
Supporting Evidence
However, in those cases where inaccurate simulation predictions could have disastrous consequences, one needs much more evidence to guarantee that the simulation is credible enough to reduce the risks. For example, the simulations that are being used to prove that the ballistic missile defense system will work under all circumstances must have a lot of credibility to justify the expenditure of funds on so large and important a system. The risks associated with erroneous performance predictions and a potentially inadequate missile umbrella would be extremely high. In these cases, the User should demand a comprehensive accreditation assessment that is supported by a strong foundation of extensive V&V results. In these situations, the case supporting the accreditation needs to be fully documented and reviewed.
The guiding principle is that quantity and quality of the evidence is directly related to the level of risk of the intended use.
PROBLEM
CREDIBLE
SOLUTION

8. V&V: The Central Pillars of Simulation Credibility
S/W Accuracy
Data Accuracy
Output Accuracy
Simulation meets design requirements, operates as designed and is free of errors in software
Simulation input data, validation data and data manipulations are appropriate and accurate
Simulation outputs match the real world “well enough” to be of use in a particular problem
Software (S/W) Accuracy
Data Accuracy
Output Accuracy
Verification
Validation
Approaching the question of what makes a simulation credible from an abstract perspective, one can think of a simulation as the bridge. To build confidence that the bridge is adequate, one would assess the capabilities and characteristics of the bridge in light of the requirements for the bridge. Typically, one would review and evaluate some combination of design information, engineering certifications, test reports, builder qualifications, maintenance records, etc. to determine if the bridge was adequate for the type and level of traffic that was expected. In an analogous way, we can assess the credibility of a simulation for a specific application based on knowledge of its design, the accuracy of its software (verification results), the accuracy of its outputs (validation or test results), the accuracy of its input data, its usage history, etc..
Most people think of simulation credibility in terms of three types of accuracy: software accuracy, data accuracy and output accuracy. Software accuracy refers to the error-freeness of software design and coding; data accuracy refers to the accuracy and appropriateness of input data to the simulation; output accuracy refers to the degree to which simulation outputs match the real world. These aspects of simulation credibility are easily recognized as the objects of verification and validation activities. These elements give one an understanding of simulation accuracy. However, these elements, alone, do not comprise a comprehensive assessment of simulation credibility?
"V"
&
"V"
But, V&V are Just the Middle of the Bridge!

9. The “Other Pillars” of Simulation Credibility
Capability
Usability
Simulation possesses all required functionality and fidelity for the problem being solved
Anchors the M&S to the Problem
Simulation has adequate user support to facilitate correct operation and interpretation of its outputs
Ties the M&S to a Useful Solution
M&S
Capability
Accuracy
Usability
M&S
Requirements
User
Capabilities
Our experience suggests that simulation accuracy (as defined in the previous slide) is a necessary but not a sufficient condition for a robust determination of simulation credibility. The software may be demonstrably free of errors, the data may be demonstrably accurate and correct, and simulation outputs may correlate well with real world data. But does the simulation represent all the real world phenomena that affect the problem you are trying to solve? Does it simulate these functions at the appropriate level of detail? Is the simulation (software, documentation and data) managed well? Is it accompanied by enough support structure to allow you to use it credibly, given the user’s level of experience with the simulation?
Questions like these provide the approaches to our simulation “bridge” and provide the linkage between the bridge and the anchor points on the banks of the river that must be crossed. Any robust assessment of simulation credibility must consider not only “accuracy”, but also “capability” and “usability” in determining whether it is suitable for use in a particular application. Any accreditation assessment that is evaluating simulation credibility for a given application must consider all five of these credibility elements.
Each of these five elements, capability, software accuracy, data accuracy, output accuracy, and usability, will be discussed individually in the following sections of the tutorial.
Credible
Solution
Problem
Accuracy of:
Software
Data
Outputs
Can I Be Sure I’m Not Mis-Using the M&S?
Does the M&S Do What I Need It To Do?

10. The Essence of Accreditation
M&S
REQUIREMENTS
IDENTIFY
M&S DEFICIENCIES
M&S
INFORMATION
Capability
Accuracy
Usability
Data Quality
M&S Documentation
Design Documentation
Configuration Mgt
V&V Results
Etc.
IDENTIFY
WORK-AROUNDS,
USAGE CONSTRAINTS,
REQUIRED IMPROVEMENTS
AND RISKS
Provided by the Model Developer
or Model Proponent
Defined by the User
(Formally or Implied)
ACCREDITATION
DECISION
Accreditation and the associated accreditation assessment have been discussed. However, it is important to understand just what accreditation entails. This diagram shows a simplified view of what accreditation really is and how it is typically done.
Within the DoD, accreditation is defined as the official determination that a model is acceptable for a specific purpose. In practical terms, this definition implies that accreditation depends on a comparison of a model’s capabilities, limitations, and attributes with the simulation requirements that are generated from the specific problem for which the model will be used. To make such a judgment about model suitability, one must have a complete set of simulation requirements. These requirements are derived from the unique problem which the User is trying to solve.
In addition to the simulation requirements, one must also have documented evidence of the simulation’s capabilities, limitations, and attributes. Typically, this information is found in the sources indicated. A key part of this information is the evidence of data quality. This evidence is quite frequently obtained from sources completely separated from the Model Manager.
In making the assessment, one usually cannot be satisfied with a simple “no” answer if the capability, accuracy, or usability of the simulation does not meet requirements. In these cases one must analyze the impacts of any deficiencies, determine if any work-arounds exist, and evaluate these against the requirements. Such an analysis should yield a list of tasks that must be done to make the simulation acceptable.
PROBLEM CONTEXT
TO PROVE THE M&S IS SUITABLE FOR THE NEED
REQUIRES AN OBJECTIVE COMPARISON
OF M&S INFORMATION WITH M&S REQUIREMENTS
WITHIN THE CONTEXT OF THE PROBLEM

11. How Much V&V is Enough? It Depends on Risk RISK = PROBABILITY x IMPACT
Risk means something “bad” might happen because you believed an incorrect simulation result
Decisions based on M&S results are at risk
VV&A reduces that risk
RISK
HIGH
LOW
MODERATE
“How much credibility is enough” can only be answered in the context of the application. That is, the answer to that question depends on what decisions you’re going to make using model outputs, and what the risks are if you make the wrong decision. If you have a very high risk application, such as decisions that might involve life or death situations (such as embedded flight-critical software in an aircraft), then the potential risk of a wrong decision is very high. In that case, the model must have very high credibility, which will require a high level of capability, accuracy and usability, and a high level of verification and validation activity to demonstrate that credibility. If the risk is low, then less is required to demonstrate that the model is suitable for your application.
JASA has developed an Accreditation Information Requirements Guide (AIRGuide), which can be used to help assess the risks inherent in the application of models and simulations, and which guides the development of a V&V plan to mitigate those risks. The AIRGuide process is based on assessing both the impact of wrong decisions made with incorrect model outputs and the probability of a wrong decision being made. Risk is defined as the product of impact and probability. More information on the AIRGuide is available in the handouts for the tutorial.
PROBABILITY
IMPACT

12. Quantifying Risk Level
Probability of Incorrect Simulation Result
Level of Impact on the Program
Negligible
Marginal
Serious
Critical
Catastrophic
Frequent
Low
Moderate
High
Probable
Occasional
Remote
Impossible
Once the User has quantified the probabilities and impact levels, these can be combined to arrive at an overall level of risk (for each individual risk factor that is identified). This table is one of several contained in the MIL-STD. It again is an example that can be tailored to the needs of the individual application. This type of table allows the User to determine the level of risk based on a level of impact and probability of each risk factor.
If there are multiple risk factors associated with a particular application, each risk factor should be analyzed individually. (For example, if a simulation is being used to to determine the footprint of the missile, one risk factor is the chance of hitting an observer on the range if the predicted footprint is too small. Another is the likelihood of wasting resources to clear too large a test area because the predicted footprint was too large.) The highest level risk, considering the risk of each factor, will determine the level of credibility that is needed in the simulation being used.
RISK LEVEL VALUES ARE:
Subjective
Consistent with MIL-STD-882
Tailorable to each application
HIGHER RISK MEANS MORE CREDIBILITY EVIDENCE IS NEEDED TO ACHIEVE ACCREDITATION

13. Risk Reduction Strategies5 4
Likelihood
Improved M&S Credibility 3 2 1 1 2 3 4 5
Reduced reliance on M&S results
Impact
Low
Moderate
High
Risk = Likelihood x Impact

14. Application to Test and Laboratory Facilities
This was a trial application of the M&S VV&A approach to test facilities
Identification Friend-or-Foe (IFF) system testing to evaluate new IFF system performance
Accreditation of test facilities required by Commander, Operational Test and Evaluation Force (COMOPTEVFOR)
Facilities used for system assessment include:
All-up ship radar and related system representations
Simple stimulators
Engineering Test Equipment (ETE) facility specific to IFF system testing

15. ETE Facility
Generates waveform signals to stimulate a production IFF transponder in the laboratory
To evaluate system requirement for resistance to signals from transmitters other than the desired transmitter
Critical technical parameter “susceptibility of the IFF system to false interrogations”
Metric: Probability of resistance to false signals = Pr

16. Initial ETE Risk Assessment
M&S Characteristic
Criterion
Risk
CAPABILITY
Intended Use
The specific intended use(s) of the facility, model or simulation is/are clearly stated.
LOW
Design
The facility and analysis process (framework, algorithms, data sources, and assumptions) produces credible results.
MODERATE
ACCURACY
Input Data
For each facility, model or simulation, input data are credible and subject to review and revision.
System Verification
The facility, model or simulation has been formally tested or reviewed and has been demonstrated to accurately represent the specific intended use(s) and requirements.
Results Validation
The facility’s, model’s or simulation’s responses have been compared with known or expected behavior from the subject it represents and has been demonstrated to be sufficiently accurate for the specific intended use(s).
USABILITY
Configuration Management
For each facility, model or simulation, modeled components are supported by a sound written Configuration Management (CM) Plan.
User Community
For each facility, model or simulation, the capability is designed and developed for the level of competency for its intended purpose. The capability is supported by documents such as user’s manual, technical manual, and/or reference guide.

17. Initial ETE Assessment and Recommendations*
CAPABILITY:
The intended use is clearly stated: to evaluate the probability of responding to a false signal (Pr)
No formal design documents exist: recommend laboratory design be adequately documented
ACCURACY:
Input data are provided by actual hardware
Recommend documenting a complete set of test cases and results and any previous verification activities
Recommend independent subject matter expert (SME) review of laboratory results
USABILITY:
Recommend facility develop and implement an overall configuration management plan
The test approach appears to have been successfully used over a span of many years to support a variety of identification programs for DOD and the FAA: recommend the facility provide documented results of previous uses
* Incorporated into ETE Facility Accreditation Plan

18. Observations Application to test facility similar to M&S Differences:
Biggest issue for both: getting good documentation
Lack of configuration management plans
Poor documentation of prior V&V results
They did V&V, they just didn’t write it down
Accreditation Support Package (ASP) document format works equally well for both
Differences:
Developing intended use statement more natural for test facilities than for M&S
Use of T&E facilities seems to have more focused objectives initially than use of M&S

19. Accreditation Support Package
ACCURACY
Software Accuracy
Software Verification Results
Design Verification
Implementation Verification
Software Development and Management Environment
Software Development Environment
Configuration Management
Software Quality Assessment
Implications for M&S Use
Data Accuracy
Input Data
Data Transformations
Output Accuracy
Sensitivity Analysis
Benchmarking
Face Validation
Results Validation
EXECUTIVE SUMMARY
ASP OVERVIEW
CAPABILITY
M&S Description
Functional Capabilities
Development History
Assumptions and Limitations
Implications for M&S Use
USABILITY
Documentation Assessment
User Support
Usage History

20. Thoughts on Broader Application
Risk-based approach seems as applicable to test facilities as to M&S
Risk assessment can also help prioritize which facilities justify spending more VV&A resources
Suggest standardizing and institutionalizing risk-based VV&A process for both M&S and T&E
No consistent application across DOD for either
Risk-based VV&A promotes cost-effective VV&A for both M&S and T&E facilities