Technologietag Baugruppentest ISO – Funktionale Sicherheit mit dem TestStand Toolkit Daniel Riedelbauch Marketing Manager CER, National Instruments Manuel Hofmann Regional Product Engineer - ECM, National Instruments

2 Escalating Complexity Space Shuttle ~500K Lines of Code Commercial Airplane ~3M Lines of Code Modern Automobile 100M Lines of Code* Up to 100 ECUs * IEEE, “This Car Runs on Code”, Charette, Robert N., February 2009

3 Certification Landscape IEC60601 Medical ISO26262 Automotive IEC61800 Power Drive Systems IEC62061 Machinery IEC61511 Process Industry DO-254 Avionics ECSS Q60-02 Spatial IEC60987 Nuclear EN50126 Railway RAMS IEC61508 generic DO-254 (DO-178B)

4 What is ISO 26262? ISO Active Systems Passive Systems ADAS By-Wire Systems Electronic Stability Control ISO addresses the needs for an automotive specific, unified, international standard that focuses on safety-critical components

5 What is an Automotive Safety Integrity Level (ASIL)? ExposureControlSeverityASIL Probability of exposure Ability to Control Severity of failure

6 Common Fears and Concerns New and confusing There are similar standards for other industries, but ISO is new, so it is unknown to many A process on top of a process Appears to introduce overhead on top of currently used development process Tracing requirements to test cases and results Not always part of the product development process and is often unclear Using qualified tools ISO specifically calls out using qualified tools, which is a new concept

7 Test and Validation Best Practices Test throughout the development process Trace test results directly to requirements Reuse test components with a common framework Solidify requirements early Create an end-to-end quality management system Use COTS qualified verification tools

8 Test and Validation Best Practices Test throughout the development process Trace test results directly to requirements Reuse test components with a common framework Solidify requirements early Create an end-to-end quality management system Use COTS qualified verification tools

9 Traditional View of the Development Process Physical Testing Design C Code Generation Prototyping HIL Validation Research and Development Testing and Validation

10 Testing Throughout the Process Physical Testing Model-in- the-Loop Testing Software-in- the-Loop Testing Lab or Field Evaluation HIL Validation Research and Development Testing and Validation

11 Test and Validation Best Practices Test throughout the development process Trace test results directly to requirements Reuse test components with a common framework Solidify requirements early Create an end-to-end quality management system Use COTS qualified verification tools

12 Requirements Coverage Requirement is an element in a specification document that specifies an ID and text Reference is an element in a covering document that references the ID of a requirement Spec Requirement ID Text Spec Requirement ID Text Coverage Reference (ID) Coverage Reference (ID)

13 Requirements Coverage with TestStand 1.Requirements defined in Word, Excel, PDF, DOORS, RequisitePro,… 2.TestStand references requirements 3.Define a project to indicate what files are involved

14 Test and Validation Best Practices Test throughout the development process Trace test results directly to requirements Reuse test components with a common framework Solidify requirements early Create an end-to-end quality management system Use COTS qualified verification tools

15 Test Components Models Analysis Reports User Interfaces Stimulus Requirements

16 Test Continuity through Component Re-use

17 Consistency and Efficiency Throughout the Process Data Logging Stimulus Profile Requirements Data Logging Stimulus Profile Data Logging Stimulus Profile Data Logging Stimulus Profile Test components automatically re-usable across all testing applications Physical Testing Design C Code Generation Prototyping HIL Validation

18 Test and Validation Best Practices Test throughout the development process Trace test results directly to requirements Reuse test components with a common framework Solidify requirements early Create an end-to-end quality management system Use COTS qualified verification tools

19 Creating More Change-Resistant Requirements Requirements based upon ASILs, which are determined by 3 factors: Severity of the failure (S) Probability of exposure to the failure (E) Controllability of preventing the failure (C) The more confidence we have in these variables, the less likely our requirements are to change.

20 Physical Testing C Code Generation Prototyping HIL Validation Design Data Logging Stimulus Profile Requirements Early Prototyping Data Logging Stimulus Profile Data Logging Stimulus Profile Data Logging Stimulus Profile Early Prototyping as an Input Into Requirements Use the same framework to prototype earlier for determining S, E, or C and defining requirements early

21 Test and Validation Best Practices Test throughout the development process Trace test results directly to requirements Reuse test components with a common framework Solidify requirements early Create an end-to-end quality management system Use COTS qualified verification tools

22 Improving Quality Management with Enhanced Collaboration and Traceability IBM Rational and National Instruments are teaming together to provide an end-to-end quality management and real-time testing solution. Traceability all the way to test and back Collaboration between ALL teams, including test Test component re-use throughout the project providing operational efficiency and accuracy Quality and test considered from the outset, not an afterthought, reducing cost of identifying and correcting defects

23 Project Management System Engineering Software Engineering Hardware Engineering Mechanical Engineering Quality/Safety Engineering Test Engineering Application Lifecycle Management System End-to-End Quality Management Solution

24 Test and Validation Best Practices Test throughout the development process Trace test results directly to requirements Reuse test components with a common framework Solidify requirements early Create an end-to-end quality management system Use COTS qualified verification tools

25 Why Qualified Tools? From ISO Clause 11.2: … In cases where a software tool can support or enable tailoring of the ISO safety lifecycle a confidence is needed that ensures the software tool effectively achieves the following goals: the risk of systematic faults in the developed product due to malfunctions of the software tool leading to erroneous outputs is minimized, and the development process is adequate with respect to compliance with ISO 26262, if activities or tasks required by ISO rely on the correct functioning of the software tool used

26 Why Qualified Tools? From ISO :2011 Clause 11.2 …The criteria used to determine if tool qualification is required include the following: the possibility that the malfunctioning software tool and its corresponding erroneous output can introduce or fail to detect errors in a safety-related item or element being developed, and the confidence in preventing or detecting such errors in its corresponding output

27 Tool Confidence Level (TCL) Tool Error Detection TD1TD2TD3 Tool Impact TI1 TCL1 TI2 TCL1TCL2TCL3 Ref Table 3 in ISO :2011 Clause

28 NI TestStand Tool Qualification Kit Cost savings of 90% or more for formal tool qualification Eliminates costly manual reviews of detailed test results Reduces cycle time for critical customer deliveries

29 NI TestStand Tool Qualification Kit Contents Tool Qualification Plan (TQP) – generic plan that describes the general aspects of the tool, and the plan for achieving qualification Tool Operational Requirements (TOR) – functional requirements for commonly used features Tool Requirements Verification Testing Procedure – procedures for verifying functional requirements, including operator instructions and all test sequence files Requirements Trace Matrix – maps requirements to associated verification test procedures Tool Qualification Accomplishment Summary (TQAS) – summary of tool qualification process and results TestStand TM Coding Standard document – provides coding standards for the qualified use of the TestStand TM test management framework Peer Review Records – records of all peer reviews for docs, requirements, tests

30 Achieving TCL2 with NI TestStand Approved by TÜV SÜD Automotive GmbH Evaluated for ‘trustworthiness’ Reviewed included safety manual “… it can be claimed that the qualification measure ‘Validation of the software tool (ISO26262:2011, Part )’ is fulfilled.”

31 Using COTS Tools for ISO Compliance Benefits of COTS tool qualification include: Increased safety and reliability Leveraging the full value of using COTS tools Significant cost savings based on increased operational efficiency Decreased time to market COTS tools can be utilized in the ISO development process

32 Test and Validation Best Practices Test throughout the development process Trace test results directly to requirements Reuse test components with a common framework Solidify requirements early Create an end-to-end quality management system Use COTS qualified verification tools