T. Dawson, TASC 9/11/13 Use of a Technical Reference in NASA IV&V

Abstract Providing mission assurance for NASA systems, and other customer systems under development, requires documentation of the system under evaluation and the standards and criteria against which the system is evaluated. These two categories taken together are called the Technical Reference, and are being used at NASA IV&V to support evidence-based assurance of customer systems. This presentation describes the intent of the Technical Reference, describes typical contents and usage, and present example products in use.

Motivations for the Technical Reference 3

“IV&V’s understanding of the system needs is developed in parallel with other IV&V activities and referred to as IV&V’s Technical Reference” “… These two sources, IV&V’s Technical Reference and developer artifacts, are ultimately used in combination with the analytical processes to establish the needed evidence to determine whether or not the system’s software is going to satisfy the goals of the system.” “IV&V Technical Reference is the collection of data and knowledge regarding IV&V’s independent understanding of the system’s software. The Technical Reference serves as the basis for IV&V analysis. This information includes but is not limited to system goals and needs, software interactions amongst system design elements, normal and abnormal behaviors and conditions of the system’s software and the operational environment.” * Excerpts from the IV&V Services Contract Statement of Work 4 What is an IV&V Technical Reference? *

As “objective evidence to either confirm or deny that the software artifacts are correct and complete” “The IV&V’s Technical Reference is one source for identifying how the system’s software should behave or perform.” IV&V uses “the IV&V Technical Reference in conducting analysis and documenting the evidence needed to conclude whether the IV&V Project’s artifacts are sufficient or where there are limitations/risks/issues.” In short: as one component of providing evidence-based software assurance, a primary goal of the NASA IV&V Facility 5 Why do We Need One?

Enhances communications: as the foundation for IV&V analysis, it is natural and desirable for the Technical Reference to be used in communications of IV&V results with the developers, IV&V management, and other stakeholders The Technical Reference also captures IV&V understanding of particular aspects of the system/software that can be reused for later missions that are inheriting or reusing systems, architectures, technologies, software, etc. Part of the IVV& project documentation that lives on beyond the project 6 Additional Benefits of the Technical Reference

Characteristics of the Technical Reference 7

Is a knowledge repository that contains IV&V’s understanding of the system – This understanding is synthesized in part from developer artifacts, but does not contain project the artifacts themselves Contains specific, project-relevant criteria for performing the IV&V analyses Contains IV&V-developed and outside (not development project) standards, evaluation criteria, and references 8 Characteristics of Technical Reference Content

Documents the materials IV&V used in performing analysis and making assurance claims Has a strong tie to Evidence-Based Assurance, but does not contain the evidence itself IV&V issues and risks are not part of the Technical Reference. These items are two types of IV&V products that the Technical Reference supports producing 9 Characteristics of Technical Reference Content (cont.)

Content is elaborated and updated as the IV&V project progresses – Individual components should be elaborated or updated as appropriate based on project needs and current understanding – “Creating the Technical Reference” is not a standalone task in its own right at the beginning of the project The Technical Reference is defined for a given project, but content varies from activity to activity – A target of one analysis could form part of the basis for the Technical Reference for a follow-on activity 10 Technical Reference Over Time

IV&V’s understanding of the system as contained in IV&V-developed products: – Heritage Report – System Goals/System-Level Behaviors – IV&V Scoping Products – Context diagrams created by IV&V – Any Diagram generated to capture IV&V’s understanding of requirements/ behaviors – Use Cases generated by IV&V to capture understanding of the system/software capabilities and behaviors – Scenarios developed by IV&V to support analysis 11 Specific Technical Reference Content Examples

Project-specific criteria for performing the IV&V analyses: – List of Adverse Conditions – List of Undesired Behaviors – Required interface characteristics derived from an ICD IV&V-developed references/criteria: – Architecture analysis assessment criteria – Domain specific questions derived by IV&V Outside references/criteria – MIL-STD-1553 standard for use in evaluating the project’s 1553 flight software 12 Specific Technical Reference Content Examples (cont.)

Technical Reference in the IV&V Process 13

14 Technical Reference’s Role in Performing IV&V Analysis Other Project Artifacts Supplemental Representations Criteria IV&V Analysis Technical Reference Evidence IV&V Sources Outside Sources Target Artifacts References to project artifacts Methods

Project artifacts (synonymous with developer artifacts) consist of Target Artifacts and Other Artifacts Target artifacts are the artifacts being evaluated in a given IV&V activity and should not form part of the basis for the Technical Reference for that IV&V activity. – Target artifacts can serve as one source for Technical Reference content. One example is a flight software requirements document that is used to build a representation of the system design that is then used to confirm consistency and other aspects of the system Other project artifacts support the IV&V activity and provide one source for IV&V’s understanding of the system. The Technical Reference contains references to elements of these other artifacts as appropriate IV&V and Other Sources comprises two sets of inputs to the technical reference. IV&V sources include whites papers, prior IV&V project findings, and anything created by IV&V as a reference for performing IV&V that applies to the given project but was not generated solely for use by the given project. Other Sources include similar inputs sourced from outside IV&V, to include IEEE and other standards and technical societies, and sources of scientific, engineering, software, aerospace, subsystem and other domain knowledge 15 Components of the Technical Reference Process

Supplemental representations are created by IV&V as necessary to capture IV&V’s understanding of the system Criteria are used for the IV&V analysis. In order to determine sufficiency, some scale must be applied. This can vary dramatically from target to target and by IV&V Method being used, and these criteria form part of the Technical Reference IV&V analysis consists of the IV&V analysis activities that comprise the majority of the project team’s work. A number of considerations, including the activity goals and the IV&V Catalog of Methods, are used to select the specific IV&V activities to be performed. The purpose of the IV&V activities is to produce evidence Evidence is part of the Assurance Case approach to providing Evidence-Based Assurance and supports the Assurance Case Argument. Evidence is collected during the IV&V analysis activities 16 Components of the Technical Reference Process (cont)

The needed content should be documented in IV&V process descriptions (currently underway) A given project’s Technical Reference is defined and cataloged in the Master List – Content is activity-specific 17 Documenting the Technical Reference

Filename Location (path) Activity for which intended – FY – Activity Number – Activity Name Role in Activity Content Description Source of Content Version/Date Comments 18 Master List Fields

Impact on IV&V Analysis 19

All of the inputs are used in “traditional” NASA IV&V As such, there is very little impact on IV&V analysis with the introduction of Technical Reference concepts Any impact on the analyst or specific analysis task should primarily be a positive (beneficial) by providing reference materials and analysis criteria 20 Impact on IV&V analysis

Q: If it doesn’t change how we do things, why do we need it? A: This is a structured approach to IV&V planning and execution – Provides us a way to explicitly account for these aspects during our project/task planning process 21 Impact on IV&V analysis (cont.)

Going Forward 22

Technical Reference Evolution 23 Umbrella Concept ECM-Based Project-Profile Based IVVO Data Mgmt-Based RecentNear-TermMid-TermLong-Term Partially complete Varies by project Not fully integrated in processes Single, defined location Document- based Potential for redundant products Contained within Project Profile Allows various views of the information Mix of data elements and documents Ultimate goal for IVVO data Multi-views of same info Fully dynamic content Project Profile is a client application IVVO Data Management Concept → Implementation SWAT ODS → Data Warehouse AMF CD → Implementation Technical Reference Mode Technical Reference Characteristics Parallel Relevant Efforts Now

© 2013 TASC, Inc. 24