1. System Verification Plans Brian Selvy Systems Engineering Manager Kathryn Wesson Systems Engineer LSST Commissioning Plan Review January 24-26, 2017

2. Verification Planning’s Role in Commissioning
Agenda
Verification Planning’s Role in Commissioning
Verification & Validation Process Overview
Verification Matrix (VM) Status
Verification Through the Commissioning Phases
Verification Tools
Role of System Verification Team in Commissioning
Future Work

3. Verification Planning’s Role in Commissioning

4. Commissioning Plan Inputs
Requirements for Operations Readiness
System-Level Verification Plans are the key technical input to Commissioning
These plans ensure each project- level specification requirement and interface requirement is mapped to an activity in Commissioning where it will be verified.
System Level Verification Plans
Commissioning Plan
[LSE-79]
Subsystem AIV Plans
Scientifically Motivated Performance Characterization
Technical Optimization of System Operations
Emergent Technical Issues

5. Verification & Validation Process Overview

6. Verification and Validation Process
LSE-160 Verification and Validation Process is the governing document for V&V on LSST
Establishes a consistent, project-wide process for the development of V&V plans, compliance assessments, V&V reporting, and deliverables
Defines steps in the verification process
Defines requirements for developing verification plans for each project- controlled requirement

7. Applies to all Project-Controlled requirements:
LSE-160 Applicability
Applies to all Project-Controlled requirements:
Specifications & Requirements Documents
Interface Control Documents (ICDs)
Each “shall” statement in each of these documents must be formally verified
Project-Controlled Specifications
Project-Controlled ICDs

8. V & V Process Activity

9. Verification Planning Requirements
For each requirement (“shall statement”) a Verification Plan will be created that includes the following:
Verification Owner – the subsystem team that is responsible for verification
Responsible Technical Authority (RTA) – the point-of-contact assigned responsibility for the verification of the requirement from the responsible subsystem. The RTA, along with the responsible QA individual, has responsibility for overseeing all associated verification events.
Verification Method(s) – Test, Analysis, Inspection, Demonstration
Verification Level – Same Level, Higher Level, Lower Level
Verification Requirement – A statement that defines precisely what will be done to verify the requirement. If there is any vagueness in the requirement, the Verification Requirement should clearly address the noted issues and define what precisely will be verified and any limitations. The statement should define what will be done, where it will be done, what special test equipment (SPE) is needed, and what project hardware/software is needed.
Success Criteria - A statement that defines the explicit pass/fail criteria. This statement should be clear enough that an independent third party observer should be able to determine if the verification event was successful or not.

10. Verification Planning Requirements (Cont)
Documented in a Verification Matrix for Planning (VM-P)
One required for each Specification, or Requirements Document
One required for each subsystem involved in an ICD

11. Final Verification Reporting Rqmts
A final Verification Record is compiled after all requirements within a specification / ICD have been verified.
A Verification Matrix for Final Verification (VM-V) serves as the final record and summary of the verification process.
For each requirement, an RTA will be identified. These individuals are responsible for vouching that the requirement has been verified and generating initial responses to Non-Conformances
For each requirement, summary information from the Verification Plan is included along with:
Responsible Technical Authority (RTA)
Verification Successful (Y/N)
Verification Result Summary – a concise summary narrative explaining why the verification activities were successful or not.
Verification Report – A reference to the Verification Report that contains the details of the results of the verification activities.

12. Verification Process Steps 4 and 5
After Verification Plans are generated, PSE uses this information, along with additional input from the subsystem teams, to develop a comprehensive system-level Verification model, :
Identify Task Interdependency (Step 4)
Some Verification Activities can be naturally grouped and conducted at the same time
These Verification Activities are then grouped into a single Verification Event.
Can result in cost and schedule savings from eliminating redundant or nearly redundant V&V activities
Schedule Verification Events (Step 5)
Events are scheduled, identifying predecessor/ successor relationships and other schedule constraints

13. Verification Matrix Status

14. Verification Planning Matrices Status
Doc #
Name
Type
Complete
LSE-29
System Requirements
System Specification
25%
LSE-30
Observatory System Specifications
System Specification
75%
LSE-59
Camera Subsystem Requirements
Subsystem Specification
50%
LSE-60
Telescope and Site Subsystem Requirements
LSE-61
Data Management Subsystem Requirements
LSE-62
Observatory Control System Requirements
LSE-64
Utilities and Services Interface Between the Camera and Telescope
Mechanical ICD
LSE-65
Summit Facility Interface Between the Camera and Telescope
LSE-66
Guider Interface Between the Camera and Telescope
Software and Hardware ICD
LSE-67
Wavefront Sensor Interface Between the Camera and Telescope
LSE-68
Camera Data Acquisition Interface
Software and Hardware ICD
LSE-69
Interface Between Camera and Data Management
LSE-70
System Communication Protocol Interface
Software ICD 0%
LSE-80
Mechanical, Thermal and Access Interfaces between the Camera and Telescope
35%
LSE-89
EPO Requirements
90%
LSE-130
Support-Data Exchanges between Data Management and Camera 5%
LSE-131
Interface Between Data Management and EPO
LSE-209
Software Component to OCS Interface
Completion Definition
% Allocated
Empty Matrix Posted on Docushare ready to be populated 5%
Matrix populated with methods, owners
25%
Matrix populated with verification statements
50%
Matrix populated with success criteria
75%
Requirement clarifications, modification, as a result of verification work captured for CCB submission
90%
Verification matrix (VM-P) data imported into Sysarch model
100%

15. OSS Verification Matrix for Planning (VM-P)
342 items with an OSS-REQ-XXXX ID tag
29 items are headers, discussions, definitions – i.e. not requirements
313 Verifiable Requirements

16. OSS Verification Matrix – Methods and Levels
Verification Activities
313 Verifiable Requirements have:
474 Verification Activities
49% have only a Primary VM
51% have a Prim & Sec VM
Verification Levels:
292 of the Verification Requirements are conducted at the same (OSS) level
21 are conducted at a lower level
These are mostly for requirements that are subsystem allocations
Verif. Method
Prim
Sec
Total
% Total
Test
100 20
120
25.3%
Demo 92 28
Analysis 74 93
167
35.2%
Inspection 47 67
14.1%
313
161
474
100%
Verification Levels
Verif. Level
Number
% Total
Higher (LSR)
0.0%
Same (OSS)
292
93.3%
Lower (Subsystem) 21
6.7%
Total
100%

17. OSS Verification Methods
313 Verification Requirements (VRs)
Section Headers, Summaries, Descriptions
(No Verification)
# of VRs including T or D: 226
% of VRs including T or D: 72%

18. Verification through the Commissioning Phases

19. Technical Verification Activities Across Commissioning Phases
Verify early, then monitor!
Formal system verification begins as soon as all prerequisites for a given verification activity are met
Technical Verification spans all phases of Commissioning

20. Categories of Verification Activities
Full (F) - the requirement (or group of requirements) can be fully verified by the activity.
Partial (P) - the requirement (or group of requirements) can be partially verified by the activity.
Incremental (I) - the requirement has been previously fully verified but a new version of the hardware or software has been introduced and is incrementally verified to ensure previous results are not invalidated.
Monitor (M) - the requirement has been previously fully verified but the key aspects of the requirement are exercised again; monitored for results that invalidate the previous verification.
Full (F) - the requirement (or group of requirements) can be fully verified
Partial (P) - the requirement (or group of requirements) can be partially verified. This means either some of the requirements in the group can be fully verified and others not or that portions of the scope of an individual requirement can be verified but other portions not. Requirements (or groups of requirements) that are partially verified are also mapped to subsequent verification activities to finish the coverage.
Incremental (I) - the requirement has been previously fully verified but a new version of the hardware or software has been introduced and is incrementally verified to ensure previous results are not invalidated. The incremental verification activities are narrow and limited in scope, focusing primarily on the features of the unit or component that have changed from the previous version.
Monitor (M) - the requirement has been previously fully verified but the key aspects of the requirement are executed again in subsequent verification activities. These requirements are noted as Monitored in the subsequent verification activities so that any results that may invalidate previous successful verifications will be noted and traced to the impacted requirements.

21. Mapping of Requirement Groups to Commissioning Activities
All requirement groups within LSR, OSS, and ICDs have been mapped to Commissioning activities, with EPO verification occurring in parallel.
This initial mapping will be iterated and revised as the bottom up planning continues to progress

22. Mapping of Requirement Groups to Commissioning Activities
All requirement groups within LSR, OSS, and ICDs have been mapped to Commissioning activities, with EPO verification occurring in parallel.
This initial mapping will be iterated and revised as the bottom up planning continues to progress

23. Verification Tools

24. Verification-Related Tools
MagicDraw (Model Based Systems Engineering Tool) – Verification Plans
Primavera P6 (Project Planning Tool) – Long Term Activity and Resource Plans
JIRA (Issue Tracking & Agile Work Planning Tool) – Short Term, Agile Planning
Kanoah Tester (JIRA Testing Add-In) – Test Planning and Execution
Requirements, Interfaces & Verification Plans
MagicDraw
Verification Ticket <- Verification Plans
Verification Events <-> Activities
Verification
Results
Detailed Agile Work Planning
Activity Schedule & Resource Planning
Sprint Completion -> Step Completion
(EVMS)
JIRA
Primavera P6
Epochs <- Activities

25. Verification in WMS

26. Verification in WMS – MagicDraw MBSE Tool
Source of all project-controlled requirements (specifications & ICDs)
Source of all Verification Plans
Source of data for Verification Matrices

27. Verification in WMS – Syndeia Inter-Tool Element Transformations
Auto-generates Verification Tickets in JIRA from MagicDraw VerificationPlan elements and attributes
Two-way synchronization
Feeds verification results back to MagicDraw to generate final verification matrices (VM-F)

28. Verification in WMS Map Verification Plans to Test Cases
Generate Test Scripts (steps) for each Test Case
Sequence Test Cases into Test Runs
Execute Test Runs
Address failures with issue tickets
Sync results back to MagicDraw

29. Role of System Verification Team in Commissioning

30. Commissioning Organization

31. System Verification Team
Primary Responsibilities
Review and close out verification activities
Monitor progress of verification against plan; recommend adjustments to plan as necessary
Address verification activity failures (waivers, corrective actions, use of technical margin, etc)
Convene and Conduct Commissioning Phase Reviews
Composition (3 FTEs)
Lead by Systems Engineering Manager
Project Systems Engineering assignees
Rotate in and out of Chile. 1 FTE in Chile at any one time

32. Verification Team’s Interaction with TCT & Systems Scientist
System Verification Team (SysVT) responsible for upfront planning
Generation of Verification Plans
Mapping to Verification Events (Test Cases)
Definition of Verification Events
SysVT
SysVT
SysVT works with the Systems Scientist to Schedule activities
SysVT
+ Systems Scientist
Core Commissioning Team (CCT) executes the Verification Events (Test Cases) and Analyzes Results
SysVT Reviews the results and makes the determination on the sufficiency of the results to close out the Verification Event
CCT
SysVT tracks verification progress

33. Addressing Verification Activity Failures
Multiple possible responses to a failure
Rerun if issue with procedure definition or execution
Rework if reduced performance is not acceptable
Examples:
Software update
Replacement of hardware with spares where available
Updates to variable parameters ….
Waiver request if reduced performance is acceptable – approved via CCB process (next slide)

34. Waiver & Results Approval Process for Verification
Use existing CCB process (in usage now for waiver requests)
Change Control Board and Project Manager involved for technical waivers; Project Science Team also involved for waivers impacting science verification

35. Future Work

36. PSE Schedule – Path to ComRR
JAN 17
JUN 17
JAN 18
JUN 18
ComPDR
NSF/DOE JSR
LSST 2017 AH
ComRR
Project Events
JTM
Spec. VM-P Complete
ICD VM-P Complete
Verif. Planning Complete
Internal PSE
Checkpoint #1
Internal PSE
Checkpoint #2
Internal PSE
Checkpoint #3
Internal PSE
Checkpoint #4
Internal PSE
Checkpoint #5
Verification Planning Working Groups – Specifications
Verification Planning Working Groups – Interfaces (ICDs)
Requirements Clarifications for Verification Activity
Import Verification Matrix Data into MBSE Model as Matrices Complete
Project Systems Engineering
Populate Verification Planning Elements in MBSE Model
Populate Verification Events in JIRA
-Map requirements
-Assign Predecessor/Successor Relations
-Identify Test Equipment Needs
-Assign Actors
-Sequence Steps
Develop Work Management System
Refine Work Management System
Develop Test Scripts and Analysis Code with Interfaces to Reporting Tools
Special Test Equipment List Refinement
Recursive/
As Needed
PSE Work Plan

37. PSE Schedule – Major Milestones
Date
Milestone
V&V Entrance Criteria
Products Included
August 2017
LSST 2017 All Hands
Complete Preliminary Verification Plan (PVP) for all system-level requirements
Preliminary Verification Matrices for all System-Level requirements (method, owner, level, verification requirement, success criteria)
Initial Verification Events and sequencing
System-Level Compliance Status
January 2018
Internal PSE Checkpoint #4
Complete Preliminary Verification Plan (PVP) for all Interface requirements
Preliminary Verification Matrices for all Interface requirements (method, owner, level, verification requirement, success criteria)
Interface Compliance Status
Update Special Test Equipment List
Summer 2018
Commissioning Readiness Review
Complete Final Verification Plan for all System-Level and Interface requirements
Updated material included in Verification Plan
Detailed Verification Procedures
Complete set of sequenced Verification Events (Test Cases) in JIRA.
Finalized Special Test Equipment List
Analysis and Simulation code for analyzing commissioning data
Mockup of Verification Statistics and Metrics tracking dashboard

38. Path Forward
Quarterly Internal PSE Checkpoints will help track progress and ensure completion on time for ComRR.
A series of reoccurring working groups are scheduled to complete verification planning. Current focus is on specification documentation, followed by Interfaces.
As matrices are completed, data will be imported into MBSE model for creation of planning elements that feed ticketing system sequenced verification events.
Compliance checks for System-Level and Interface Requirements will be conducted at milestones.
Collaboration with subsystems to ensure execution of verification activity is supported through special test equipment, software, and instrumentation.

39. End

40. Backup Slides

41. Compliance Assessment Requirements
Compliance is defined as the ability of the current (any point in time) “as- designed” system to meet its associated requirements.
The difference between compliance and verification is that verification is conducted on the final designed and built system, whereas compliance can be done at any earlier time and is an early step in the overall verification process.
Compliance Assessments are required at each major subsystem and component design review.
Required documentation:
Compliance Method(s) – Analysis, Test, Demonstration, Inspection
Compliance Requirement
Success Criteria
Compliance Status (Y/N)
References to any additional documentation that further justifies the assessment, if available.

42. Compliance Assessment Rqmts (Cont)
For any non-compliances:
A risk must be documented in the Risk & Opportunity Register
The risk Number should be cross-referenced in the compliance table
The Risk entry should identify mitigation activities to correct the non-compliance

43. Verification vs. Validation
Ensures that the system, its elements, and its interfaces conform to their requirements.
“You built it right.”
Validation:
Provides objective evidence that the services provided by a system when in use in an operational environment comply with the stakeholders’ needs.
“You built the right thing.”

44. Basis of Verification
Statements of need, requirements, and constraints are written using one of three specific verbs that have a direct tie to verification:
Will – A statement of fact. Will statements document something that will occur through the course of normal design practice, project process, etc. These statements do not get formally verified.
Should – A goal. Should statements document a stretch goal. A should statement will be partnered with a shall statement. Should statements do not get formally verified.
Shall - A requirement that gets formally verified. Shall statements document critical requirements that must be verified through inspection, demonstration, analysis, or test during the verification phase of the project to ensure objectively that the as-built design meets the requirement.
As noted by these definitions, only “shall” statements are formally verified.

45. Verification Methods
Inspection: An examination of the item against applicable documentation to confirm compliance with requirements. Inspection is used to verify properties best determined by examination and observation (e.g., paint color, weight, etc.)
Analysis: Use of analytical data or simulations under defined conditions to show theoretical compliance. Analysis (including simulation) is used where verifying to realistic conditions cannot be achieved or is not cost-effective and when such means establish that the appropriate requirement, specification, or derived requirement is met by the proposed solution.
Demonstration: A qualitative exhibition of functional performance, usually accomplished with no or minimal instrumentation. Demonstration (a set of verification activities with system stimuli selected by the system developer) may be used to show that system or subsystem response to stimuli is suitable. Demonstrations require the designation of a witness to observe the system response and vouch for the results.
Test: An action by which the operability, supportability, or performance capability of an item is verified when subjected to controlled conditions that are real or simulated. These verifications often use special test equipment or instrumentation to obtain very accurate quantitative data for analysis. (Haskins, 127)

46. A Requirements to Verification Plan Example

47. A Requirements to Verification Plan Example

48. A Requirements to Verification Plan Example
Derivation

49. A Requirements to Verification Plan Example
Verification Planning & Integration

50. A Requirements to Verification Plan Example
Verification Events