JLab Software Assurance Program A Risk Based Approach to Software Management

Outline Software Assurance vs. Software Quality Assurance QA Order Requirements Processes that address software assurance JLab’s SW Assurance Effort Risk Based Model Assessment Method Preliminary Results Path Forward 8/18/20102

DOE 414.1C Applies to ALL Software activities Ten Criteria for Safety SW QA Program Requirements flow-down through CRD QA Order 414.1C QA Order 414.1C QA PLAN CORRECTIVE ACTION MANAGEMENT SUSPECT/COUNT ERFEIT ITEM PROCESS (SAFETY) SOFTWARE QUALITY SOFTWARE ASSURANCE PROCEDURE SOFTWARE ASSURANCE PROCEDURE GENERAL REQUIREMENTS GENERAL REQUIREMENTS 8/18/20103

JLab Software Assurance Procedure Implementation of Requirements of Quality Assurance Plan Implements a process for identifying and classifying the impact SW may have on multiple subject areas, including safety Adaptable to all software activities important to facility mission and goals Implements consistent tiered approach 8/18/20104

Software Quality Assurance  Software Assurance NASA-STD (w/Change 1) July 28, 2004 “Software assurance consists of the following disciplines: Software Quality –Software Quality Assurance –Software Quality Control –Software Quality Engineering Software Safety Software Reliability Software Verification and Validation (V&V) Independent Verification and Validation (IV&V)” NASA-GB NASA Software Safety Guidebook. Implementation guidance for high consequence software 8/18/2010 5

JLab Process SW Assurance team chartered by CIO –Representatives from across site: Scientific –Experimental –Theoretical –Computing Business Controls Cyber Security HR Safety Systems 8/18/20106

JLab SWAP - Table of Contents 1Purpose 1.1Structured Approach 2Scope 2.1Exemptions 3Responsibilities 4Software Control Procedure 4.1Software Risk Assessment Process Steps & Expectations 4.1.1Software critical to JLab safety, operations, and mission 4.1.2Software important to JLab safety, operations, and mission 4.1.3Software Risk Assessment Assumptions: 4.1.4Software Risk Assessment Tool 8/18/20107

Table of Contents - Continued 4. 2Software Assurance 4.2.1Graded Approach 4.2.2Software Assurance Program Requirements Acquirer Software Assurance Basic Requirements for Software Assurance Processes: Software Lifecycle Software Quality Competence Sustainability Configuration Management Assessment 4.3Metrics and Continuous Improvement 5.0DEFINITIONS 6.0REFERENCES 7.0REVISION SUMMARY 8/18/20108

JLab SW Assurance Procedure Clarifies scope: –Applies to all projects, programs, facilities, and activities that may impact JLab mission and goals –Applies to all software developed, or modified for use at JLab. Compliments Cyber Security Program –Reflects SW Enclaves –Applies to security software configuration items only where ineffective security software controls may directly affect operations and safety –Cyber Security Risk Assessment incorporated in to overall risk assessment 8/18/20109

JLab SW Assurance Defines SW Risk Assessment Procedure: –Identifies pre- and post-mitigation Risk –Applicable to ALL SW within scope of process Defines Requirements for Owner Organization SW –Requirements for lifecycle model –Requirements for 8/18/201010

Structured Approach Identify important JLab software configuration items and activities Identify roles and responsibilities for software control activities within the context of this procedure Perform a software risk assessment on applicable configuration items Apply a risk-based graded approach to software assurance activities Apply a value added continuous improvement process to the software assurance processes 8/18/201011

Exemptions “This procedure does not apply to unmodified general purpose computing software, unmodified enterprise software, and general purpose desk-top software managed under the IT/CIO Division. Examples include office productivity software, public web pages, and LAN/WAN networking software.” 8/18/201012

Roles and Responsibilities Defines roles, responsibilities, and authority for –COO –CIO –ESH&Q AD –Division Management –Line Management –Software Owner –Oversight committees 8/18/201013

Scope internal software development software used to collect and manage data startup and configuration scripts incorporation of open source software modified off the shelf (MOTS) software used to design, analyze, or control safety or mission essential aspects of JLab operations commercial off the shelf (COTS) software used to design, analyze, or control safety or mission essential aspects of JLab operations programs and firmware for monitoring or control, including IOCs and PLCs modifiable embedded software and firmware including PICs and PC104 type SBCs programs and development software for field programmable integrated circuits such as Field Programmable Gate Arrays Other software as defined by the JLab Chief Information Officer

SW Risk Assessment Software is scored (1-5) in each of six areas of impact –Direct Risk of Financial Loss –Direct Risk of Loss of Tangible Equipment/Property –Direct Risk of Harm to People –Direct Risk of Harm to the Environment –Direct Risk of Loss of Continuity of Operations/Organization/Mission –Direct Risk of Enforcement Action Scoring is reviewed at both the individual and cumulative level

Data Collection SW Application/Information Example Worst Credible Error/Event/IncidentOwnerType Assigned JLab Cyber Security LevelPlatform Application, information, or SW configuration item Briefly describe event and consequences of SW error Organization that owns the performance/requirements for the application Major SW application type HW/SW platform running the application Augerlost productivity (physics "farm")IT - SCIJavaLowLinux Scoring Direct Risk of Financial Loss, Includes Cost of Unplanned Labor Direct Risk of Loss of Tangible Equipment/Material Direct Risk of Harm to People Direct Risk of Harm to the Environment Direct Risk of Loss of Continuity of Operations/ Organization Mission Direct Risk of Enforcement Action Number from 0 to 5 (See Table- Definitions) Analysis Recommendations TotalPre MitigationSW QA Mitigation(s)Post Mitigation Risk Acceptance SW QA practices used for this application Risk Acceptance 5 Tolerable Testing prior to release; testing by users Acceptable

Types of Software Assessed –Lattice QCD Modeling –Experiment Data Management –MIS Accounting –MIS Procurement –Corrective Action Tracking System –Facilities Work Order System –Travel –-Timesheets –Facilities Work Order System –Accelerator Physics Model –Machine Protection –Personnel Safety System PLC Firmware PLC Program –Radiation Instrumentation –Beam Position Monitor

Risk Acceptance Criteria – Pre Mitigation

SW Assurance as a Mitigation Each Owning Organization responsible for tailoring requirements in to their own SA program Procedure provides consensus standard references for generally accepted good practice Procedure provides references for incorporation in to organization’s individual process

Lifecycle Model Requirements

Metrics Recommended for Acceptable and Tolerable Risk Required for Intolerable and Unacceptable Risk Assessments go back to central repository for analysis Allows comparison of mitigations vs. claimed effectiveness Refers to existing SW metric processes Guidance refers to CMMI process

–Pilot project complete –In process of implementing procedure lab wide –Feedback (mostly) positive –Expanding Risk Assessment data Status

Conclusions JLab is implementing a risk based software assurance process Consensus based procedure with buy-in from all SW enclaves Tools, e.g. risk assessment spreadsheet, are integrated in to the process Provides minimum requirements for SW lifecycle Incorporates resources and guidance for application Process incorporates metrics