1. Framework for Compliance, Verification, and Non-Conformance George Angeli LSST All-Hands Meeting Bremerton August 17, 2015

2. Outline 8/17/2015 LSST Systems Engineering 1 Non-conformance framework (George Angeli) – Quality assurance at project and subsystem level – Change Control Board – Acceptance Data Package – Case studies Verification and Validation on LSST (Brian Selvy) Subsystem Acceptance and Verification in Commissioning (Chuck Claver)

3. Overview 8/17/2015 LSST Systems Engineering 2 Basic procedures of Quality Assurance implemented by subsystems – Further refinement in progress (eTraveler, detailed NCR process) – Project QA should match in maturity the subsystems – Bottom-up and top-down procedures should meet in the middle Objectives – Establish a process for handling non-conformance at project level Well documented and uniform Transparent and predictable for the project community Do not interfere with current subsystem procedures – Establish a process for documenting deliverables to and by the project Uniform across all subsystems Provide a well organized archive for the lifetime of the Observatory Accessible for the project community and the operations team

4. Quality Assurance 8/17/2015 LSST Systems Engineering 3 Well established regime of inspections and tests Well defined responsibilities Traceable and detailed documentation of deliverables and their verification/certification Transparent and consistent handling of non-conformance Every subsystem has the responsibility to establish its own QA Lower level plans (safety, QA, verification, review, etc.) should conform with higher level ones Project level coordination of deliverables – To the Project by subsystems, and – By the Project to stakeholders/operators KEEP IT SIMPLE! (as much as possible …)

5. The “Linear” Way of Acceptance 8/17/2015 LSST Systems Engineering 4 All requirements are met Full compliance

6. The “Linear” Way of Acceptance 8/17/2015 LSST Systems Engineering 5 All requirements are met Full compliance PSE witnessing PSE collecting Project Systems Engineering understanding of verification and as-built component behavior facilitates system verification and commissioning

7. The “Not-So-Linear” Way 8/17/2015 LSST Systems Engineering 6 Non-Conformance ? Full compliance

8. The “Not-So-Linear” Way 8/17/2015 LSST Systems Engineering 7 Non-Conformance ? Full compliance Change Control Process Deviation note

9. Non-Conformance Resolution 8/17/2015 LSST Systems Engineering 8 Using the Change Control Process is convenient – Proper people are involved – bring understanding of all aspects of the project – Well established, exercised process – Well documented – web site captures process steps – Transparent and archived process Deviation approval process 1.Non-Conformance Report – as soon as it is recognized 2.Change Request Filed against requirements, as usual, but Can be filed for a single component only All relevant documentation posted in the DocuShare collection 3.Impact study, as usual 4.Decision, as usual CCB is an advisory body Project Manager disposes it, per the Project Execution Plan Turnaround time limited by the impact study, not by the process

10. Acceptance Data Package 8/17/2015 LSST Systems Engineering 9 Relevant requirements – Specifications and interface requirements – Non-conformance reports, change requests, deviation approvals Compliance certification – Verification Matrix – Verification and acceptance test plans and procedures – Qualification and/or acceptance test results – Calibration and certification requirements records – Material and/or process certifications Safety documentation and constraints As-built technical documentation Shipping, handling and storage plans Operations, maintenance, and repair plans and manuals

11. Case Study 1: M1M3 Acceptance 8/17/2015 LSST Systems Engineering 10 Managed by the Telescope & Site (T&S) team – Jacques Sebag, in particular All tests were done by Steward Observatory Mirror Lab (SOML) personnel – Witnessed by representatives of the T&S and Systems Engineering teams PSE was invited to all meetings with the vendor – PSE supported the T&S team with analyses and trade studies Documentation – All data received from vendor posted on an open confluence page – All documentation collected on project DocuShare M1/M3 Cast Mirror Specification M1/M3 Cast Mirror Specification M1M3 Mirror Verification Matrix M1M3 Mirror Acceptance Test Plan M1M3 Mirror final acceptance report LSST M1M3 Measurements of Prescription, Dimensions and Alignment LSST Final Acceptance Review 150410 compressed.pptx

12. Case Study 1: M1M3 Analyses 8/17/2015 LSST Systems Engineering 11 Rebuilding the “M1M3 Null Test Data Reduction Pipeline” (Document-17912) – Laying the foundation of fully understanding the most critical vendor acceptance test Potential non-conformance: crow’s feet on M3 – In support of the T&S team, PSE evaluated the impact on ultimate metrics (Document-17171) Image depth Ellipticity (single visit) – Positive outcome cleared the way to acceptance

13. Case Study 2: L1L2 Assembly 8/17/2015 LSST Systems Engineering 12 Early involvement of PSE – Invitation to all reviews (PDR, 75%, 90%, FDR) Verification/test plans Compliance status – Sharing data and analyses (FEA results, scattered light, interface volume, etc.) Potential non-conformance issues – Glass refraction index uniformity Joint optical analysis with LLNL cleared it – Surface treatment of carbon fiber Deviation request against an OSS requirement Based on PSE scattered light analysis, the request is about to be approved

14. Summary 8/17/2015 LSST Systems Engineering 13 It’s essential to involve PSE early in – Verification planning – Compliance assessments – Verification activities System level non-conformance is handled through the CCB process Acceptance Data Packages are expected to accompany every deliverable – System Data Package are built on subsystem Data Packages – Subsystem Data Packages are built on major component Data Packages; and so on …