System Verification Plans Brian Selvy Systems Engineering Manager Kathryn Wesson Systems Engineer LSST Commissioning Plan Review January 24-26, 2017

Verification Planning’s Role in Commissioning Agenda Verification Planning’s Role in Commissioning Verification & Validation Process Overview Verification Matrix (VM) Status Verification Through the Commissioning Phases Verification Tools Role of System Verification Team in Commissioning Future Work

Verification Planning’s Role in Commissioning

Commissioning Plan Inputs Requirements for Operations Readiness System-Level Verification Plans are the key technical input to Commissioning These plans ensure each project- level specification requirement and interface requirement is mapped to an activity in Commissioning where it will be verified. System Level Verification Plans Commissioning Plan [LSE-79] Subsystem AIV Plans Scientifically Motivated Performance Characterization Technical Optimization of System Operations Emergent Technical Issues

Verification & Validation Process Overview

Verification and Validation Process LSE-160 Verification and Validation Process is the governing document for V&V on LSST Establishes a consistent, project-wide process for the development of V&V plans, compliance assessments, V&V reporting, and deliverables Defines steps in the verification process Defines requirements for developing verification plans for each project- controlled requirement

Applies to all Project-Controlled requirements: LSE-160 Applicability Applies to all Project-Controlled requirements: Specifications & Requirements Documents Interface Control Documents (ICDs) Each “shall” statement in each of these documents must be formally verified Project-Controlled Specifications Project-Controlled ICDs

V & V Process Activity

Verification Planning Requirements For each requirement (“shall statement”) a Verification Plan will be created that includes the following: Verification Owner – the subsystem team that is responsible for verification Responsible Technical Authority (RTA) – the point-of-contact assigned responsibility for the verification of the requirement from the responsible subsystem. The RTA, along with the responsible QA individual, has responsibility for overseeing all associated verification events. Verification Method(s) – Test, Analysis, Inspection, Demonstration Verification Level – Same Level, Higher Level, Lower Level Verification Requirement – A statement that defines precisely what will be done to verify the requirement. If there is any vagueness in the requirement, the Verification Requirement should clearly address the noted issues and define what precisely will be verified and any limitations. The statement should define what will be done, where it will be done, what special test equipment (SPE) is needed, and what project hardware/software is needed. Success Criteria - A statement that defines the explicit pass/fail criteria. This statement should be clear enough that an independent third party observer should be able to determine if the verification event was successful or not.

Verification Planning Requirements (Cont) Documented in a Verification Matrix for Planning (VM-P) One required for each Specification, or Requirements Document One required for each subsystem involved in an ICD

Final Verification Reporting Rqmts A final Verification Record is compiled after all requirements within a specification / ICD have been verified. A Verification Matrix for Final Verification (VM-V) serves as the final record and summary of the verification process. For each requirement, an RTA will be identified. These individuals are responsible for vouching that the requirement has been verified and generating initial responses to Non-Conformances For each requirement, summary information from the Verification Plan is included along with: Responsible Technical Authority (RTA) Verification Successful (Y/N) Verification Result Summary – a concise summary narrative explaining why the verification activities were successful or not. Verification Report – A reference to the Verification Report that contains the details of the results of the verification activities.

Verification Process Steps 4 and 5 After Verification Plans are generated, PSE uses this information, along with additional input from the subsystem teams, to develop a comprehensive system-level Verification model, : Identify Task Interdependency (Step 4) Some Verification Activities can be naturally grouped and conducted at the same time These Verification Activities are then grouped into a single Verification Event. Can result in cost and schedule savings from eliminating redundant or nearly redundant V&V activities Schedule Verification Events (Step 5) Events are scheduled, identifying predecessor/ successor relationships and other schedule constraints

Verification Matrix Status

Verification Planning Matrices Status Doc # Name Type Complete LSE-29 System Requirements System Specification 25% LSE-30 Observatory System Specifications System Specification 75% LSE-59 Camera Subsystem Requirements Subsystem Specification 50% LSE-60 Telescope and Site Subsystem Requirements LSE-61 Data Management Subsystem Requirements LSE-62 Observatory Control System Requirements LSE-64 Utilities and Services Interface Between the Camera and Telescope Mechanical ICD LSE-65 Summit Facility Interface Between the Camera and Telescope LSE-66 Guider Interface Between the Camera and Telescope Software and Hardware ICD LSE-67 Wavefront Sensor Interface Between the Camera and Telescope LSE-68 Camera Data Acquisition Interface Software and Hardware ICD LSE-69 Interface Between Camera and Data Management LSE-70 System Communication Protocol Interface Software ICD 0% LSE-80 Mechanical, Thermal and Access Interfaces between the Camera and Telescope 35% LSE-89 EPO Requirements 90% LSE-130 Support-Data Exchanges between Data Management and Camera 5% LSE-131 Interface Between Data Management and EPO LSE-209 Software Component to OCS Interface Completion Definition % Allocated Empty Matrix Posted on Docushare ready to be populated 5% Matrix populated with methods, owners 25% Matrix populated with verification statements 50% Matrix populated with success criteria 75% Requirement clarifications, modification, as a result of verification work captured for CCB submission 90% Verification matrix (VM-P) data imported into Sysarch model 100%

OSS Verification Matrix for Planning (VM-P) 342 items with an OSS-REQ-XXXX ID tag 29 items are headers, discussions, definitions – i.e. not requirements 313 Verifiable Requirements

OSS Verification Matrix – Methods and Levels Verification Activities 313 Verifiable Requirements have: 474 Verification Activities 49% have only a Primary VM 51% have a Prim & Sec VM Verification Levels: 292 of the Verification Requirements are conducted at the same (OSS) level 21 are conducted at a lower level These are mostly for requirements that are subsystem allocations Verif. Method Prim Sec Total % Total Test 100 20 120 25.3% Demo 92 28 Analysis 74 93 167 35.2% Inspection 47 67 14.1% 313 161 474 100% Verification Levels Verif. Level Number % Total Higher (LSR) 0.0% Same (OSS) 292 93.3% Lower (Subsystem) 21 6.7% Total 100%

OSS Verification Methods 313 Verification Requirements (VRs) Section Headers, Summaries, Descriptions (No Verification) # of VRs including T or D: 226 % of VRs including T or D: 72%

Verification through the Commissioning Phases

Technical Verification Activities Across Commissioning Phases Verify early, then monitor! Formal system verification begins as soon as all prerequisites for a given verification activity are met Technical Verification spans all phases of Commissioning

Categories of Verification Activities Full (F) - the requirement (or group of requirements) can be fully verified by the activity. Partial (P) - the requirement (or group of requirements) can be partially verified by the activity. Incremental (I) - the requirement has been previously fully verified but a new version of the hardware or software has been introduced and is incrementally verified to ensure previous results are not invalidated. Monitor (M) - the requirement has been previously fully verified but the key aspects of the requirement are exercised again; monitored for results that invalidate the previous verification. Full (F) - the requirement (or group of requirements) can be fully verified Partial (P) - the requirement (or group of requirements) can be partially verified. This means either some of the requirements in the group can be fully verified and others not or that portions of the scope of an individual requirement can be verified but other portions not. Requirements (or groups of requirements) that are partially verified are also mapped to subsequent verification activities to finish the coverage. Incremental (I) - the requirement has been previously fully verified but a new version of the hardware or software has been introduced and is incrementally verified to ensure previous results are not invalidated. The incremental verification activities are narrow and limited in scope, focusing primarily on the features of the unit or component that have changed from the previous version. Monitor (M) - the requirement has been previously fully verified but the key aspects of the requirement are executed again in subsequent verification activities. These requirements are noted as Monitored in the subsequent verification activities so that any results that may invalidate previous successful verifications will be noted and traced to the impacted requirements.

Mapping of Requirement Groups to Commissioning Activities All requirement groups within LSR, OSS, and ICDs have been mapped to Commissioning activities, with EPO verification occurring in parallel. This initial mapping will be iterated and revised as the bottom up planning continues to progress

Mapping of Requirement Groups to Commissioning Activities All requirement groups within LSR, OSS, and ICDs have been mapped to Commissioning activities, with EPO verification occurring in parallel. This initial mapping will be iterated and revised as the bottom up planning continues to progress

Verification Tools

Verification-Related Tools MagicDraw (Model Based Systems Engineering Tool) – Verification Plans Primavera P6 (Project Planning Tool) – Long Term Activity and Resource Plans JIRA (Issue Tracking & Agile Work Planning Tool) – Short Term, Agile Planning Kanoah Tester (JIRA Testing Add-In) – Test Planning and Execution Requirements, Interfaces & Verification Plans MagicDraw Verification Ticket <- Verification Plans Verification Events <-> Activities Verification Results Detailed Agile Work Planning Activity Schedule & Resource Planning Sprint Completion -> Step Completion (EVMS) JIRA Primavera P6 Epochs <- Activities

Verification in WMS

Verification in WMS – MagicDraw MBSE Tool Source of all project-controlled requirements (specifications & ICDs) Source of all Verification Plans Source of data for Verification Matrices

Verification in WMS – Syndeia Inter-Tool Element Transformations Auto-generates Verification Tickets in JIRA from MagicDraw VerificationPlan elements and attributes Two-way synchronization Feeds verification results back to MagicDraw to generate final verification matrices (VM-F)

Verification in WMS Map Verification Plans to Test Cases Generate Test Scripts (steps) for each Test Case Sequence Test Cases into Test Runs Execute Test Runs Address failures with issue tickets Sync results back to MagicDraw

Role of System Verification Team in Commissioning

Commissioning Organization

System Verification Team Primary Responsibilities Review and close out verification activities Monitor progress of verification against plan; recommend adjustments to plan as necessary Address verification activity failures (waivers, corrective actions, use of technical margin, etc) Convene and Conduct Commissioning Phase Reviews Composition (3 FTEs) Lead by Systems Engineering Manager Project Systems Engineering assignees Rotate in and out of Chile. 1 FTE in Chile at any one time

Verification Team’s Interaction with TCT & Systems Scientist System Verification Team (SysVT) responsible for upfront planning Generation of Verification Plans Mapping to Verification Events (Test Cases) Definition of Verification Events SysVT SysVT SysVT works with the Systems Scientist to Schedule activities SysVT + Systems Scientist Core Commissioning Team (CCT) executes the Verification Events (Test Cases) and Analyzes Results SysVT Reviews the results and makes the determination on the sufficiency of the results to close out the Verification Event CCT SysVT tracks verification progress

Addressing Verification Activity Failures Multiple possible responses to a failure Rerun if issue with procedure definition or execution Rework if reduced performance is not acceptable Examples: Software update Replacement of hardware with spares where available Updates to variable parameters …. Waiver request if reduced performance is acceptable – approved via CCB process (next slide)

Waiver & Results Approval Process for Verification Use existing CCB process (in usage now for waiver requests) Change Control Board and Project Manager involved for technical waivers; Project Science Team also involved for waivers impacting science verification

Future Work

PSE Schedule – Path to ComRR JAN 17 JUN 17 JAN 18 JUN 18 ComPDR NSF/DOE JSR LSST 2017 AH ComRR Project Events JTM Spec. VM-P Complete ICD VM-P Complete Verif. Planning Complete Internal PSE Checkpoint #1 Internal PSE Checkpoint #2 Internal PSE Checkpoint #3 Internal PSE Checkpoint #4 Internal PSE Checkpoint #5 Verification Planning Working Groups – Specifications Verification Planning Working Groups – Interfaces (ICDs) Requirements Clarifications for Verification Activity Import Verification Matrix Data into MBSE Model as Matrices Complete Project Systems Engineering Populate Verification Planning Elements in MBSE Model Populate Verification Events in JIRA -Map requirements -Assign Predecessor/Successor Relations -Identify Test Equipment Needs -Assign Actors -Sequence Steps Develop Work Management System Refine Work Management System Develop Test Scripts and Analysis Code with Interfaces to Reporting Tools Special Test Equipment List Refinement Recursive/ As Needed PSE Work Plan

PSE Schedule – Major Milestones Date Milestone V&V Entrance Criteria Products Included August 2017 LSST 2017 All Hands Complete Preliminary Verification Plan (PVP) for all system-level requirements Preliminary Verification Matrices for all System-Level requirements (method, owner, level, verification requirement, success criteria) Initial Verification Events and sequencing System-Level Compliance Status January 2018 Internal PSE Checkpoint #4 Complete Preliminary Verification Plan (PVP) for all Interface requirements Preliminary Verification Matrices for all Interface requirements (method, owner, level, verification requirement, success criteria) Interface Compliance Status Update Special Test Equipment List Summer 2018 Commissioning Readiness Review Complete Final Verification Plan for all System-Level and Interface requirements Updated material included in Verification Plan Detailed Verification Procedures Complete set of sequenced Verification Events (Test Cases) in JIRA. Finalized Special Test Equipment List Analysis and Simulation code for analyzing commissioning data Mockup of Verification Statistics and Metrics tracking dashboard

Path Forward Quarterly Internal PSE Checkpoints will help track progress and ensure completion on time for ComRR. A series of reoccurring working groups are scheduled to complete verification planning. Current focus is on specification documentation, followed by Interfaces. As matrices are completed, data will be imported into MBSE model for creation of planning elements that feed ticketing system sequenced verification events. Compliance checks for System-Level and Interface Requirements will be conducted at milestones. Collaboration with subsystems to ensure execution of verification activity is supported through special test equipment, software, and instrumentation.

End

Backup Slides

Compliance Assessment Requirements Compliance is defined as the ability of the current (any point in time) “as- designed” system to meet its associated requirements. The difference between compliance and verification is that verification is conducted on the final designed and built system, whereas compliance can be done at any earlier time and is an early step in the overall verification process. Compliance Assessments are required at each major subsystem and component design review. Required documentation: Compliance Method(s) – Analysis, Test, Demonstration, Inspection Compliance Requirement Success Criteria Compliance Status (Y/N) References to any additional documentation that further justifies the assessment, if available.

Compliance Assessment Rqmts (Cont) For any non-compliances: A risk must be documented in the Risk & Opportunity Register The risk Number should be cross-referenced in the compliance table The Risk entry should identify mitigation activities to correct the non-compliance

Verification vs. Validation Ensures that the system, its elements, and its interfaces conform to their requirements. “You built it right.” Validation: Provides objective evidence that the services provided by a system when in use in an operational environment comply with the stakeholders’ needs. “You built the right thing.”

Basis of Verification Statements of need, requirements, and constraints are written using one of three specific verbs that have a direct tie to verification: Will – A statement of fact. Will statements document something that will occur through the course of normal design practice, project process, etc. These statements do not get formally verified. Should – A goal. Should statements document a stretch goal. A should statement will be partnered with a shall statement. Should statements do not get formally verified. Shall - A requirement that gets formally verified. Shall statements document critical requirements that must be verified through inspection, demonstration, analysis, or test during the verification phase of the project to ensure objectively that the as-built design meets the requirement. As noted by these definitions, only “shall” statements are formally verified.

Verification Methods Inspection: An examination of the item against applicable documentation to confirm compliance with requirements. Inspection is used to verify properties best determined by examination and observation (e.g., paint color, weight, etc.) Analysis: Use of analytical data or simulations under defined conditions to show theoretical compliance. Analysis (including simulation) is used where verifying to realistic conditions cannot be achieved or is not cost-effective and when such means establish that the appropriate requirement, specification, or derived requirement is met by the proposed solution. Demonstration: A qualitative exhibition of functional performance, usually accomplished with no or minimal instrumentation. Demonstration (a set of verification activities with system stimuli selected by the system developer) may be used to show that system or subsystem response to stimuli is suitable. Demonstrations require the designation of a witness to observe the system response and vouch for the results. Test: An action by which the operability, supportability, or performance capability of an item is verified when subjected to controlled conditions that are real or simulated. These verifications often use special test equipment or instrumentation to obtain very accurate quantitative data for analysis. (Haskins, 127)

A Requirements to Verification Plan Example

A Requirements to Verification Plan Example

A Requirements to Verification Plan Example Derivation

A Requirements to Verification Plan Example Verification Planning & Integration

A Requirements to Verification Plan Example Verification Events